Simple Background Jobs with Sucker Punch
Running background jobs in an app is a great way to keep the UI snappy. Being it sending an email, calling some API, or any long running task, there’s always something that can be moved to the background. Sometimes, it makes sense to set up a queue and worker processes, but other times it would be great just to have the method run in background with minimal code changes. Enter Sucker Punch, a single-process asynchronous processing library. Using the wonderful celluloid framework, it lets us do asynchronous work in very few steps.
Setup
First of all, your app needs to be running on Ruby 1.9+, JRuby 1.6+ or Rubinius 2.0 (with Ruby 1.9 mode on the last 2). Then, install the gem by running
gem install sucker_punch
or add it to you Gemfile via
gem 'sucker_punch'
And that’s it! Now lets see how to use it.
Creating a Worker
Creating a worker is pretty simple. It’s just a plain ruby class that includes the SuckerPunch::Worker
module, and defines one or more instance methods. Sucker Punch suggests just one perform
method, but any name and number of methods can be defined and used.
class SomeWorker
include SuckerPunch::Worker
def perform(some_data)
# Method code. Do some work.
end
end
Calling the Worker
Before calling the worker, we need to configure our queues:
SuckerPunch.config do
queue name: :some_queue, worker: SomeWorker, workers: 5
queue name: :welcome_queue, worker: WelcomeEmailWorker, workers: 2
end
We can define as many queues as we want with as many workers as we want (preferable 2+ per queue), but each can have only one worker class per queue.
More workers equals more parallel jobs that can be performed, but be aware of running out of connections if you’re using a connecting to an external service like a database or memcached.
Then you can access the workers on the queues via
SuckerPunch::Queue[:some_queue] # or
SuckerPunch::Queue.new(:some_queue)
This will give us a Celluloid::ActorProxy
object wrapping our worker object.
We can call the perform
method (or any other method we defined) on that object directly, or use async
to return control instantly and make it run in background.
SuckerPunch::Queue[:welcome_queue].perform(1)
SuckerPunch::Queue[:welcome_queue].async.perform(1)
Just be aware that running the job async will not raise any exception if it fails. It will just fail silently.
Testing
Everything needs testing, of course!
Fortunately, testing the worker is pretty easy. You can test it as any Ruby class.
With some rspec flavor:
describe WelcomeEmailWorker
let(:user){ FactoryGirl.create :user }
let(:worker){ EmailWorker.new }
describe "#perform" do
it "delivers an email" do
expect{ worker.perform(user.id) }.to change{ UserMailer.deliveries.size }.by(1)
end
end
end
To test how it integrates with other methods, there are 2 options:
1) Test that it calls and enqueues the job. To do this you need to require 'sucker_punch/testing'
:
require 'sucker_punch/testing'
describe User do
describe "#send_welcome_email" do
it "delivers an email" do
let(:user){ FactoryGirl.create :user }
expect{
user.send_welcome_email
}.to change{ SuckerPunch::Queue.new(:email).jobs.size }.by(1)
end
end
end
2) Running jobs inline. This can be done by requiring 'sucker_punch/testing/inline'
, and jobs will always be run synchronously.
require 'sucker_punch/testing'
describe User do
let(:user){ FactoryGirl.create :user }
describe "#send_welcome_email" do
it "delivers an email" do
expect{
user.send_welcome_email
}.to change{ UserMailer.deliveries.size }.by(1)
end
end
end
Considerations
Tests Running Inside DB Transactions
There’s one thing to have in mind with tests. If you’re running each test inside a transaction, you’ll need to change that to a truncation strategy for Sucker Punch tests. Here’s an example of how to do it with DatabaseCleaner: https://gist.github.com/kitop/5248674. This happens because Sucker Punch workers always run the method in a separate thread, no matter if it is synchronous or asynchronous.
Persistence
Keep in mind that Sucker Punch runs your jobs on a separate thread, and not polling from an outside queue. That means that, if your app goes down while processing a job, it will not notify nor store that error anywhere by default, and it won’t retry it either. If you need more control over this, you can write your own wrapper. Maybe you could get some inspiration from girl_friday, or use some other solution like resque, sidekiq, or delayed_job.
Rails
If you’re working with Rails, workers usually go in the app/workers
directory. You should be careful with ActiveRecord objects and connections. Workers should receive a record id and not the full object. Preferably, workers should wrap database access related code in a ActiveRecord::Base.connection_pool.with_connection
block so it does not exhaust connections in the pool.
class WelcomeEmailWorker
include SuckerPunch::Worker
def perform(user_id)
ActiveRecord::Base.connection_pool.with_connection do
user = User.find(user_id)
UserMailer.welcome(user).deliver
end
end
end
Connections
You have to be careful not just with ActiveRecord connections, but also with any redis, memcache, or any other service that may limit connections. It’s important to also limit the number of workers based on those limits. You don’t want to have 20 workers, when there are 10 connections max!
Unicorn/Passenger
If you’re using Unicorn or Passenger as your web server, there’s one more step to ensure everything is set up well. That is to define the queues on blocks that run after the server. For unicorn (only needed if preload_app true
is set):
# config/unicorn.rb
after_fork do |server, worker|
SuckerPunch.config do
queue name: :log_queue, worker: LogWorker, workers: 10
end
end
For Passenger:
# config/initializers/sucker_punch.rb
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
SuckerPunch.config do
queue name: :log_queue, worker: LogWorker, workers: 10
end
end
end