Using Sidekiq in a Microservice
Learnings applying the approaches proposed by Brandon Hilkert
Sidekiq is the most used solution nowadays in the Ruby community to handle background processing. It uses Redis to control the jobs queue and it’s being used by companies like Heroku, Netflix, and Digital Ocean.
It’s simple to install and start using it along with a Rails app. If you’ve never seen how it works, here is a common architecture example.
In this example, you have a single app that put jobs to the queue and implements the workers responsible to consume these jobs as well. The first part running on a web server and the second controlled by Sidekiq as a background process.
Isolating Sidekiq Instances in Microservices
Instead of a monolithic app, imagine now a microservice architecture. Many services should be able to enqueue jobs to an isolated worker. How can we do that?
Well, you could do it by simply putting a web API in front of your Sidekiq process and this app would be responsible for calling the SomeWorker.perform_async function (which adds a job in the queue).
That would be fine, but let’s think a little bit different.
It’s not mandatory that the same app should know how to enqueue AND consume a certain job. There is nothing stopping us to put a job directly in a Redis server that only an isolated worker is watching.
You can share the same Redis with multiple microservices (using different queues for each service) or divided into different installations. It may be important depending on how many microservices you’ll have and the volume of jobs you’re expecting.
I won’t get into the details of implementation here, but I suggest you take a look at Brandon’s blog if you are interested in.
What Are the Benefits?
Extracting functionalities to microservices have their benefits and costs. In my opinion, the most important benefit is the risk reduction because it’s easier to deal with peaks, manage and monitor your resources, and keep your software running beside unexpected situations.
The advantages of using Sidekiq as a microservice message queue is that you simply won’t need to implement any HTTP method. So it’s automatically easier to develop, deploy, monitor and scale your microservice considering that you’ll have less code, less complexity and fewer technologies in your stack.
