Comparing Background Processing Libraries: Sidekiq

Share this article

Comparing Background Processing Libraries: Sidekiq
People and door blue

This is the final part in the background processing series. You can check out the previous posts from the links above. I’ll summarize what we have covered so far.

  • The principles behind background processing (queues, processes etc.).
  • Using delayed_job, which is amazing for getting background processing rolled into your app really quickly, but, it uses ActiveRecord so it is kind of slow and you end up mixing up background and foreground code throughout your app.
  • Using Resque, which hopes to solve the problems with delayed_job by using Redis and separate worker classes.

However, we can still improve on the performance of Resque. This is where Sidekiq comes in.


If you read through the theory section in the previous articles (which you really should), you might have noticed that I talked about processes as the only concurrency option. It turns out that’s not really true and Sidekiq takes full advantage of something else: Threads.

Threads are a super lightweight concurrency measure, in order to increase performance. The headline on the Sidekiq page puts it squarely

“What if one Sidekiq process could do the work of 20 Resque or DelayedJob processes?”

Let’s check out how to use Sidekiq. If you read through the delayed_job and Resque introductions, Sidekiq is somewhat a combination of the two. Sidekiq does use Redis, so you will need to have that installed and running (check out the Resque section in the previous articles to see how to do this).

To get started, add the following line to your Gemfile:

gem 'sidekiq'

Install ‘er up:

bundle install

Fortunately, that’s about it for the setup we need. SideKiq doesn’t use the rake system directly like Resque, so we don’t have to fiddle around with “lib/tasks”. Let’s get onto writing our print worker in “app/workers/print_worker.rb”. Here it is:

class PrintWorker
  include Sidekiq::Worker

  def perform(str)
    puts str

Once again, we have to queue up a job somewhere (we’ll do it in the index controller):

class IndexController < ApplicationController
  def index

Finally, we have to get the Sidekiq process(es) fired up. Type this into your shell in the root directory of your Rails app:

bundle exec sidekiq

This starts a process that is waiting for jobs. Let’s put one in the queue.

If you go to “/index/index?to_print=sidekiqisgreat”, you should get “sidekiqisgreat” somewhere in the output of your sidekiq process (The sidekiq runner includes some other information that you can safely ignore for the sake of the example.)

Sidekiq is easy enough if you learned Resque, but it has a pretty big “problem” that we haven’t discussed yet: thread safety. Since Sidekiq uses threads, you can only uses libraries in Sidekiq if they are thread safe – anything else will likely cause a ton of problems that are difficult to track down. This really limits what you can do with Sidekiq.

Secondly, you must ensure that your code is thread safe (e.g. global variables are a no-go). Ruby (rather, the “default” ruby interpreter which is MRI) also has something called an “interpreter lock”, which means that only one thread can run at a time, so Sidekiq will work much better with alternative implementations of Ruby, such as Rubinus and JRuby.

What’s the point of going through all that hassle? Why not just use Resque? The biggest reason is performance; the difference is pretty big if you’re processing lots of jobs that would benefit with concurrency.

The Final Comparison

Over this three part series, we’ve covered the theory behind background processing and three background processing frameworks: delayed_job, Resque and Sidekiq. Each has its ups and downs.

With delayed_job: * Pros – incredibly quick and easy to get rolling – no addition to your “stack”; it can run just fine with ActiveRecord – a fantastic choice for beginners or migrating code from the foreground to the background * Cons – Runs on ActiveRecord, so it will probably run slower than something that runs on Redis – Makes it very easy to mix async and sync code in file, which, in my opinion, is a bad thing – The “.delay” calls scattered across your codebase will make it difficult to reason about six months later

With Resque: * Pros – It runs on Redis, which is fast – Great separation of background code with worker classes – It has a fantastic web dashboard – It makes it fairly easy to do background processing – It is my favorite!
* Cons – More difficult to get running with than delayed_job – Still isn’t the fastest! – Doing prioritized, time-based jobs is not as easy as delayed_job

With Sidekiq: * Pros – Pretty darn fast and workers are lightweight – You can port over code really easily from Resque – Great separation of code * Cons – More difficult than delayed_job – (this is the biggie) You must use thread-safe libraries and write thread-safe code

As I’ve mentioned, I like Resque the most out of the three. My primary reason to avoid delayed_job is the mixing of sync and async code. When considering Sidekiq, it is the requirement of thread-safe libraries that scares me off.

If your requirements are different, your choice could be different. Just take the time to understand the consequences of your choice! Make sure you don’t try to optimize too early, or you’ll spend time on something that has no relevance to the real bottleneck in your app!

Example app

I’ve built a small example application using Sidekiq. It performs the same things as the delayed_job and Resque examples, but using Sidekiq. It saves and displays page counts on an uploaded PDF synchronously and asynchronously. The most important thing to note is that I had to change about three lines of code in order to switch from Resque to Sidekiq, because the library I was using to count pages in PDFs happened to be thread safe. However, if it hadn’t been, writing a Sidekiq port would have been much more difficult since I would have had to either roll my own library or use a C extension.

The code for the example app can be found here.

Wrapping it up

As you can tell, the Ruby community has presented three unique solutions to a single problem. Each has its own benefits and drawbacks. Pick the one that you like the most and use it in your next amazing app!

If you have any questions or suggestions, do drop them in the comments below!

Frequently Asked Questions (FAQs) about Background Processing Libraries

What are the key differences between Sidekiq and other background processing libraries?

Sidekiq stands out from other background processing libraries due to its unique features and capabilities. It is built on Ruby and integrates seamlessly with Rails, making it a popular choice for Ruby on Rails developers. Sidekiq uses multithreading, which allows it to perform multiple tasks concurrently, thus improving efficiency and speed. It also has a comprehensive dashboard that provides real-time monitoring and statistics, which is not commonly found in other libraries.

How does Sidekiq handle job failures?

Sidekiq has a robust system for handling job failures. It uses a strategy called “automatic job retries”. When a job fails, Sidekiq will automatically retry it at increasing intervals, up to 25 times. If a job still fails after all these retries, it is moved to the “Dead” set where it remains for 6 months, giving developers ample time to debug and fix the issue.

What are the prerequisites for using Sidekiq?

To use Sidekiq, you need to have Ruby 2.5 or later and Redis 4.0 or later installed on your system. Sidekiq is compatible with all modern Rails applications (Rails 4.2+). It’s also recommended to have a good understanding of Ruby and Rails, as well as background processing concepts.

How does Sidekiq compare to traditional background processing methods?

Traditional background processing methods often involve creating separate processes for each job, which can be resource-intensive. Sidekiq, on the other hand, uses multithreading to perform multiple jobs within the same process, which is more efficient and scalable. Additionally, Sidekiq provides a user-friendly dashboard for monitoring job status, which is not typically available with traditional methods.

Can Sidekiq be used with other programming languages?

Sidekiq is primarily designed for use with Ruby and Ruby on Rails. However, there are third-party libraries available that allow Sidekiq to be used with other programming languages. For example, is a Crystal port of Sidekiq, and there are also libraries for integrating Sidekiq with Python and Node.js.

What is the difference between Sidekiq and Sidekiq Pro?

Sidekiq Pro is a commercial version of Sidekiq that offers additional features not available in the free version. These include batch jobs, which allow you to group related jobs together, and reliable fetch, which ensures that jobs are not lost if a worker crashes. Sidekiq Pro also comes with priority support.

How does Sidekiq handle job scheduling?

Sidekiq has built-in support for job scheduling. You can schedule jobs to run at a specific time in the future by using the perform_in or perform_at methods. Sidekiq will automatically execute the job when the specified time comes.

How can I monitor the performance of Sidekiq?

Sidekiq comes with a built-in web dashboard that provides real-time statistics about your jobs. The dashboard shows the number of processed and failed jobs, as well as information about the queues and the workers. You can also use third-party monitoring services like New Relic or Datadog to monitor Sidekiq.

Can I use Sidekiq with my existing Rails application?

Yes, Sidekiq is designed to integrate seamlessly with Rails. You can add it to your Rails application by including the Sidekiq gem in your Gemfile and configuring it according to your needs.

What are some common use cases for Sidekiq?

Sidekiq is commonly used for tasks that can be done in the background, such as sending emails, processing images, calling APIs, and more. It’s also used for tasks that need to be performed regularly, like cleaning up old data or generating reports. By moving these tasks to the background, Sidekiq allows your application to remain responsive to user requests.

Dhaivat PandyaDhaivat Pandya
View Author

I'm a developer, math enthusiast and student.

Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week
Loading form