Simple, Organized Queueing with Resque

Share this article

After having used many different queueing systems, from the venerable BackgrounDRb
, to DelayedJob, to roll-your-own solutions, I’ve settled on the excellent Resque. The now-famous blog post from GitHub’s Chris Wanstrath says it all, and the README has everything you could ever hope for in terms of getting up and running. However, Resque leaves a lot up to the imagination for exactly how to integrate it into your app. This is, of course, one of its strengths, as it allows for a high degree of flexibility. In this article, I am going to introduce one such integration strategy I’ve found particularly useful, clean, and most importantly, simple.

A Namespace of Its Own

In order to free up dependency on your database, which can cause all kinds of performance problems down the road, Resque uses Redis for its data store (note: if you’d like to read more about getting up and running with Redis, read my previous article here). Assuming you have already setup Redis in your app, you probably already have a $redis global variable to represent the connection. Resque uses the redis-namespace gem by default to avoid polluting the key space of your redis server, but I personally like to have control over important details like this. Fortunately, Resque allows this, so initializing the connection is as simple as adding the following to config/initializers/resque.rb: [gist id=”2731334″]

Enqueueing Jobs

Inspired by Delayed Job, in Resque all you need to run code in the background is provide a class or module that responds to the perform method. These objects also specify the name of the queue which processes them. The most evident solution to integrating Resque, therefore, is to have one of these objects per background task. For example, you might have one class for processing user-uploaded images, another class for sending your monthly newsletter to all users, and yet another for updating search indices. As you might imagine, the number of these workers is going to increase over time. Furthermore, Resque prioritizes queues based solely on the order in which they are specified to a worker, so you’re going to need to remember which workers act on which queues. If you have an app where new background tasks are constantly being added, removed, or re-prioritized, this is not only going to be confusing, but will also necessitate a lot of upkeep.

A Cleaner Approach

Rather than focusing on what needs to happen in the background, let’s focus on when
. That is, my approach is to let priority be the guiding factor on how to design an interface to Resque. The first step is to note that 99% of the time, background jobs are going to fall into one of three possible priorities: high, normal, and low. While it will be easy to add more priorities later, it will only happen in very rare circumstances. Having only three priorities also makes worker configuration much simpler. Every worker is always assigned to these three queues, so except for the count of workers, the command is always the same: [gist id=”2731496″] If necessary, to throw more processing power at the queue, simply spawn more workers: [gist id=”2731506″]

Enqueue Methods Instead

The next step to this approach is to make it easier to throw any instance or class method on to one of these queues. This is where the dynamic nature of Ruby is going to help out a lot. To begin at the end, we’re going to be able to enqueue anything like this: [gist id=”2731551″] In addition, it’s not going to matter if some_object is a class, module, or instance. We can change the priority of a method simply by changing between Queue::Normal, Queue::High, Queue::Low. Let’s look at how we can code this. The above interface clearly dictates that each class will need to specify an enqueue method. We also know that each class needs to specify a queue name and a perform method, so that’s a good place to start: [gist id=”2731581″] We can already see that these classes have the same interface, so let’s refactor it into a superclass: [gist id=”2731680″] The enqueue method’s only job is to supply data to the perform method, which will be called by Resque. The perform method needs to find the object by ID, invoke the enqueued method, and pass the enqueued arguments to it. Since everything passed to Resque needs to be serializable to JSON, we need to pass the class name of the object, the method, and the object’s ID as a special “meta” argument: [gist id=”2757896″] Then, in the perform method, we can make use of the constantize
Rails extension to get the class from its name, find the object, and send the method along with its arguments: [gist id=”2757909″] And with that, we are ready to enqueue any instance method. Here is an example of how the big picture might look: [gist id=”2863154″] That’s all there is to it. The only caveat is that all arguments passed to background_method are going to be serialized to JSON, and then deserialized back into Ruby. Usually this will not cause any problems, but one big difference is that all hashes with symbol keys will have string keys in background_method.

Enqueueing Class or Module Methods

Only one final step remains. We also want to enqueue class or module-level methods. That is, we don’t always need to find an instance by ID to accomplish certain background tasks. For example, we might want to send an email to every registered user. The code would look something like this: [gist id=”2863149″] This is going to require some slight modification to the enqueue and perform methods, since we need to be able to tell the difference between an enqueued class or module and an enqueued object. To do this, we need to see if the object’s class responds to :find_by_id or not. This works because if object is a class or module, its class is Class, which does not respond to :find_by_id. If object is not a model instance, we do not add the 'id' key to the meta information. As such, the perform method has only to check for the existence of this key to determine whether to invoke the method directly on the object, or to find an instance by ID first: [gist id=”2758040″] Note: this article assumes the ActiveRecord ORM. Depending on your application, you may need to modify the definition of is_model?
to more accurately specify what constitutes a model instance.

Robustness

We have a working queue interface, but there’s still plenty of room for programmer error. The enqueue method should ideally raise exceptions when the programmer attempts to enqueue a non-existent method, or tries to pass too many or too few arguments to that method. By adding a few quick checks, the number of queue-side failures can be reduced dramatically. Let’s add a method, ensure_queueable! to Queue::Base that raises an exception unless the method exists and the appropriate number of arguments have been passed. With these changes in place, the entire Queue::Base class looks like this: [gist id=”2758127″] Note: checking method arity is somewhat complex, as methods that accept a variable number of arguments return a negative number. See the Ruby documentation of Method for more information.

That’s All

With that, we have a simple and clean interface to Resque which allows us to enqueue any instance or class method with minimum effort. We can also add new priorities in a few lines of code by simply defining a new subclass of Queue::Base. Furthermore, we have provided a single entry point into the queue which uses Resque as its implementation, but should we decide to swap out Resque for another solution in the future, we should only need to make changes to Queue::Base. I hope this article has been useful. Have fun with Resque, and happy queueing!

Frequently Asked Questions (FAQs) about Simple, Organized Queueing with Resque

What is Resque and why is it important in job scheduling?

Resque is a powerful tool for creating background jobs, placing them in multiple queues, and processing them later. It is built on the robust Redis key-value store, which makes it highly reliable for storing queue data. Resque is important in job scheduling because it allows you to distribute time-consuming tasks among different workers, thereby improving the performance and user experience of your application. It also provides a dashboard for monitoring jobs, queues, and workers, which is crucial for managing and troubleshooting your background tasks.

How do I install and set up Resque in my Ruby on Rails application?

To install Resque, you need to add the gem ‘resque’ to your Gemfile and run the ‘bundle install’ command. After that, you need to set up an initializer for Resque where you specify the Redis server and the namespace for your queues. You also need to create a Rake file to load the necessary tasks for Resque. Finally, you need to start a Resque worker using the ‘rake resque:work’ command. Remember to specify the QUEUE environment variable to tell the worker which queues to work on.

How do I create a job in Resque?

Creating a job in Resque involves defining a class that responds to the ‘perform’ method. This method contains the logic of your job. You can place any code that you want to run in the background inside this method. To enqueue the job, you use the ‘Resque.enqueue’ method, passing the class of your job and any arguments that the ‘perform’ method requires.

How do I manage multiple queues in Resque?

Resque allows you to create multiple queues for different types of jobs. When enqueuing a job, you can specify the queue by setting the ‘@queue’ instance variable in your job class. You can also start workers for specific queues or for all queues. The order of the queues matters: Resque will try to process jobs from the first queue in the list before moving on to the next queue.

How do I handle failed jobs in Resque?

Resque provides a built-in mechanism for handling failed jobs. When a job fails, Resque puts it in a special ‘failed’ queue along with the exception that caused the failure. You can then inspect these failed jobs through the Resque dashboard and decide whether to retry them or remove them from the queue. You can also set up ‘failure backends’ to perform custom actions when a job fails, such as sending an email notification.

How do I monitor Resque jobs, queues, and workers?

Resque comes with a Sinatra-based web application that provides a real-time overview of your jobs, queues, and workers. You can see the number of processed and failed jobs, the current queues and their sizes, and the status of your workers. You can also view detailed information about each job, including its arguments, the exception it raised (if any), and the time it took to perform.

How do I test Resque jobs?

Testing Resque jobs involves enqueuing the job in your test and then running it with ‘Resque.run!’ to process all jobs in the queue. You can then check the results of the job. It’s also a good practice to test that the job has been enqueued with the correct arguments. For this, you can use the ‘Resque.peek’ method, which returns the next job in the queue without processing it.

How do I schedule jobs in Resque?

Resque does not support job scheduling out of the box, but you can add this functionality with the resque-scheduler gem. This gem allows you to schedule jobs at specified times or intervals, similar to cron jobs. You can define your schedule in a YAML file or directly in Ruby code.

How do I prioritize jobs in Resque?

Resque processes jobs in the order they were enqueued, but you can influence this order by using multiple queues. By starting your workers with a list of queues in a specific order, you can ensure that jobs in higher-priority queues are processed first. However, within a single queue, jobs are always processed in a first-in, first-out manner.

How do I ensure that Resque jobs are idempotent?

Ensuring that Resque jobs are idempotent, meaning they can be safely retried without unwanted side effects, is mostly a matter of good job design. You should design your jobs in such a way that running them multiple times with the same arguments will produce the same result. This often involves checking whether the job has already been performed before performing it again.

Peter BrindisiPeter Brindisi
View Author

Peter Brindisi has been a Ruby on Rails developer since the early days. About a year ago, he decided to leave his native homeland of New York for the shores of Italy to become CTO of frestyl. He thoroughly enjoys finding new and interesting ways to utilize Rails, and loves writing and playing music in his free time.

redisresque
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week