Key Takeaways
- Shoryuken, working in conjunction with Amazon’s SQS (Simple Queue Service), is an effective solution for processing queued jobs. It is built with Amazon SQS in mind, which is cost-effective and scalable.
- Shoryuken can be set up with the Flickr API to process jobs in the background at high speed. It requires setting up a Yahoo account, creating a Rails app with a single controller action, and setting up a Flickr module to return photos of the user ID being submitted.
- To use Shoryuken, one needs to have an Amazon Web Services (AWS) account to get access to AWS Access Keys (Access Key ID and Secret Access Key). After installing and configuring the relevant gems, AWS SDK is configured with the relevant credentials.
- Shoryuken can be used with Rails, and its workers can be tested using RSpec. It handles message visibility timeout by automatically extending the visibility timeout of a message if the processing time exceeds the initial visibility timeout. It also provides several ways to handle errors.
Rails has lots of solutions for background jobs. One of these is a brilliant Gem called Sidekiq, which we have written about before on Sitepoint.
Sidekiq is great and can solve most developer needs. It is especially useful as it takes over the heavy lifting from Rails. However, it has a few shortfalls:
- If you are not a Pro user ($750 per year) and your process crashes, you will lose your jobs.
- If your job load increases, you will need a more powerful version of Redis which costs more money and resources.
- You need to host its dashboard in order to monitor what is happening with your jobs.
A few benefits of using Shoryuken:
- It is built with Amazon SQS in mind, which is incredibly cheap ($0.50 per 1 million Amazon SQS Requests).
- SQS is built to scale. You are taking advantage of Amazon’s amazing infrastructure so it’s easy to scale your workers.
- Amazon provides a simple console to watch your queues as well as configure what happens to dead messages.
- Amazon’s Ruby SDK is very flexible when it comes to creating queues. You can create as many as you want.
In this article, I am going to guide you through setting up Shoryuken for use with the Flickr API. You will see how Shoryuken processes jobs in the background at lighting speed.
To begin this tutorial, we are going to use the Flickr API to create a simple search bar where we can generate photos based on whatever id is entered.
- First up, we have to set up a Yahoo account, as this is the only way we can access the Flickr API. Once we have a Yahoo account, simply visit the Flickr Docs page
- Click the Create an App link on the Flickr docs page.
- Apply for a non-commercial key at Flickr.com.
- On the next page, you will be asked to enter some details about your project. Simply fill in the name and other bits about your project.
- You’ll receive a Key and Secret for your application. Write these down somewhere because we will need them for this tutorial.
Next, set up the Rails app with a single controller action. To set up a new Rails app, generate one like so from the command line:
rails new shoryuken_flickr
Next, set up the controller. A standard controller action with an index
action is perfect:
rails g controller Search index
Add a root route that to this action in config/routes.rb:
root 'search#index'
On the index page, set up a simple search form:
<%= form_tag("/", method: "get") do %>
<%= text_field_tag(:flickr_id) %>
<%= submit_tag("Search") %>
<% end %>
We must set up a Flickr module to return photos of the user ID being submitted:
- First, we install the flickr_fu which makes it easy to grab the data we want.
Set up the flickr.yml file with our relevant credentials. This file lives in the config folder and looks like:
key: <%= ENV["flickrkey"] %> secret: <%= ENV["flickrsecret"] %> token_cache: "token_cache.yml
- Now we create a helper method to return the photos for the index page. In app/helpers/search_helper.rb, add the following:
module SearchHelper def user_photos(user_id, photo_count = 5) flickr = Flickr.new(File.join(Rails.root, 'config','flickr.yml')) flickr.photos.search(:user_id => user_id).values_at(0..(photo_count - 1)) end end
This method returns the photos based on the provided Flickr user ID. In app/controllers/search_controller.rb, we need to put in an action to grab that data:
class SearchController < ApplicationController
def index
if params[:flickr_id]
@photos = user_photos(params[:flickr_id],10).in_groups_of(2)
@id = params[:flickr_id]
end
end
end
Now, just create a little partial to generate the photos. In app/views/search, add a photos.html.erb file with the following:
<ul>
<% @photos.each do |photo| %>
<li> <% photo.each do |p| %>
<%= link_to(image_tag(p.url(:square), :title => p.title, :border => 0, :size => '375x375'), p.url_photopage) %>
</li>
<% end %>
<% end %>
</ul>
Flickr ids are present on user profiles in the URL. An example ID is 138578671@N04
and if you submit that value in the form, a bunch of photos are returned.
Now we have a functioning application that gets new photos from Flickr. This is awesome, but it is very slow for the user and it refreshes the whole page each time.
I think this app would be better with a bit of AJAX. First, create an index.js.erb view in app/views/search and put in some simple Javascript:
$('ul').remove();
$('#flickr').append("<%= j render 'photos'%>").html_safe();
In the controller, make sure we have a respond_to
block for our code:
class SearchController < ApplicationController
def index
if params[:flickr_id]
@photos = user_photos(params[:flickr_id],10).in_groups_of(2)
@id = params[:flickr_id]
end
respond_to |format|
format.html
format.js
end
end
end
Finally, in the search form, set remote
to true
:
<br/>
<%= form_tag("/", method: "get", :remote => true) do %>
<%= text_field_tag(:flickr_id) %>
<%= submit_tag("Search") %> <% end %>
<br/>
Okay, so it’s cool, but we still have not used Shoryuken. The process is still single threaded.
Setting Up Shoryuken
If you don’t already have an Amazon Web Services (AWS) account, you’ll need to set one up. Follow
- Click on the “My Account” dropdown menu and then click “AWS Management Console”.
- Sign in and then you’ll be taken to the AWS Management Console.
- In the top right, click on your user name in the menu bar and then click on “Security Credentials”.
- Now you will be taken to a page where you can get access to your AWS Access Keys (Access Key ID and Secret Access Key)
- Click “Create New Access Key” and take down your Access Key ID and Secret Access Key. You need these to run Shoryuken and SQS.
Once we have the AWS access keys, the next step is to install and configure the relevant gems. First, install the AWS SDK with the relevant details by adding the following to your Gemfile:
gem 'aws-sdk', '~> 2'
Then, bundle install
.
We need to configure the AWS SDK with the relevant credentials. I usually create a file called aws.rb which I put in in the config/initializers folder
touch config/initializers/aws.rb
Add the following code to the file:
Aws.config.update({
region: "eu-west-1",
credentials: Aws::Credentials.new(your_access_key, your_secret_key)
})
sqs = Aws::SQS::Client.new(
region: "eu-west-1",
credentials: Aws::Credentials.new(your_access_key, your_secret_key)
)
sqs.create_queue({queue_name: 'default'})
Make sure to replace the credentials with your actual credentials.
If we go to the SQS console, we will see a new queue created if you restart your Rails server.
Finally, time to install the Shoryuken gem. In our Gemfile:
gem 'shoryuken'
Create the Shoryuken worker and some middleware. I just create a new directory under apps called workers:
mkdir app/workers
touch app/workers/flickr_worker.rb
touch app/workers/flickr_middleware.rb
touch config/shoryuken.yml
Configure our Flickr middleware:
class FlickrMiddleware
def call(worker_instance, queue, sqs_msg, body)
puts 'Before work'
yield
puts 'After work'
end
end
Set up the worker:
class MyWorker
include Shoryuken::Worker
shoryuken_options queue: QUEUE_NAME, auto_delete: true, body_parser: JSON
def perform(sqs_msg, body)
id = body.fetch('id')
flickr = Flickr.new({
key:"your_key",
secret:"your_secret",
token_cache:"token_cache.yml"
})
flickr.photos.search(:user_id => id).values_at(0..(5 - 1))
end
end
As well as configuring our config/shoryuken.yml file to the following:
aws:
access_key_id: 'AWS_KEY'
receive_message:
attribute_names:
- ApproximateReceiveCount
- SentTimestamp
region: eu-west-1
secret_access_key: 'AWS Secret Key'
concurrency: 25
delay: 0
queues:
- [default, 6]
Excellent! We nearly have everything setup and ready to go. All that’s left is to send messages to our queue. In the search controller, put in the following:
class SearchController < ApplicationController
include SearchHelper
def index
# 138578671@N04 submit this in the form
if params[:flickr_id]
FlickrWorker.perform_async("id" => params[:flickr_id])
sleep 0.1
@photos = Photo.find_by_user_id(params[:flickr_id]).photos
end
respond_to do |format|
format.html
format.js
end
end
end
Now we just submit another message. This time, you should see it appear on the SQS console. You may need to refresh the SQS console using the refresh button which is in the top right of the screen.
You should see a message in your queue but for some reason, it is not being processed. Better sort that out. Open up another terminal window and navigate to your project. When you are there, you must run the following command:
bundle exec shoryuken -R -C config/shoryuken.yml
Now you should see you worker cleaning out the queue. When you go back to your app, you will probably see an error. Remember, Shoryuken runs in the background so it cannot create instance variables for your current process. You could save the photos to a database and then poll the table when the results arrive.
rails g model Photo user_id:string photos:string
Now we just check our migrate file and make sure the correct fields are being added. Open up the migration file (in db/migrate) and make sure it looks like:
class CreatePhotos < ActiveRecord::Migration
def change
create_table :photos do |t|
t.string :user_id
t.string :photos
t.timestamps null: false
end
end
end
If everything looks OK, we need to migrate the database.
rake db:migrate
Make sure to serialize the array that is returned in our database. In app/models/photos.rb:
class Photo < ActiveRecord::Base
serialize :photos
end
Then update our table every time the worker is run. At the bottom of the SearchHelper#user_photos
method add a line to write the photos to the database:
photos = flickr.photos.search(:user_id => id).values_at(0..(5 - 1))
Photo.create(:user_id => id, :photos => [photos])
To see it working in action, add a delay to your search controller action to give your database a chance to update. In the real world, I suggest a more elegant solution using AJAX.
sleep 1.5
There you have it. You now know how to take advantage of some awesome libraries for processing queued tasks with Shoryuken. While this example is very contrived, I used it to demonstrate how to use Shoryuken with SQS. No doubt, you probably have at least one use case that would benefit from queued messages.
Frequently Asked Questions (FAQs) about Processing API Requests with Shoryuken and SQS
How do I set up Shoryuken with AWS SQS?
Setting up Shoryuken with AWS SQS involves a few steps. First, you need to install the Shoryuken gem by adding gem 'shoryuken'
to your Gemfile and running bundle install
. Next, you need to configure Shoryuken with your AWS credentials. This can be done by creating a shoryuken.yml
file in your config directory and adding your AWS access key, secret access key, and region. Finally, you need to create a worker that will process your SQS messages. This can be done by creating a new file in your workers directory and including the Shoryuken::Worker
module.
How does Shoryuken handle message visibility timeout?
Shoryuken handles message visibility timeout by automatically extending the visibility timeout of a message if the processing time exceeds the initial visibility timeout. This ensures that the message is not returned to the queue before it has been fully processed. You can also manually set the visibility timeout for a message using the change_message_visibility
method.
Can I use Shoryuken with Rails?
Yes, Shoryuken can be used with Rails. You can add Shoryuken to your Rails application by adding the Shoryuken gem to your Gemfile and running bundle install
. You can then create workers in your app/workers directory that will process your SQS messages.
How do I test my Shoryuken workers?
You can test your Shoryuken workers using RSpec. You can create a spec file for your worker and use the perform
method to simulate processing a message. You can then make assertions about the expected outcome of processing the message.
How do I handle errors in Shoryuken?
Shoryuken provides several ways to handle errors. If an exception is raised during the processing of a message, Shoryuken will automatically retry the message until the maximum number of retries is reached. You can also define a shoryuken_fail
method in your worker to handle exceptions in a custom way.
How do I configure Shoryuken to use a specific AWS region?
You can configure Shoryuken to use a specific AWS region by setting the aws
option in your shoryuken.yml
file. The aws
option should be a hash that includes your access key, secret access key, and region.
Can I process SQS messages in batches with Shoryuken?
Yes, Shoryuken supports batch processing of SQS messages. You can enable batch processing by including the Shoryuken::Worker::Batch
module in your worker and defining a perform
method that takes a batch of messages as its argument.
How do I deploy Shoryuken to production?
You can deploy Shoryuken to production by running it as a daemon. You can use the shoryuken
command with the -d
option to run Shoryuken in the background. You can also use a process manager like Upstart or Systemd to manage your Shoryuken processes.
How do I monitor Shoryuken?
You can monitor Shoryuken using a variety of tools. AWS CloudWatch can be used to monitor your SQS queues and Shoryuken provides a shoryuken
command that can be used to monitor your workers. You can also use third-party services like New Relic or AppSignal to monitor your Shoryuken processes.
How do I scale Shoryuken workers?
You can scale Shoryuken workers by running multiple Shoryuken processes. Each Shoryuken process will run a specified number of threads, and each thread will process one message at a time. You can increase the number of Shoryuken processes or threads to increase your processing capacity.
William is a self-taught Software Developer who went from a job he hates to a job he now loves. As well as writing about himself in the 3rd person, he breaks down exactly what you need to do if you want to get your first job as a software developer on his blog and newsletter.