Writing a Feed Aggregator with Sinatra

Share this article

Who doesn’t like sunsets? Of all the sites you’ve seen about sunset you feel that there isn’t one that does it right. You decide to make the best damn sunset site on the web. There are a bunch of sunset photos. Could we use public images to populate this new awesome site? We’ll create a site while the lawyers figure out that last question.

Are the Kids Still Talking About APIs?

One might think “I bet Flickr has some great sunset photos.” Indeed they do. We’ll pick on this group. At the bottom of the page you’ll notice an RSS feed. Could we read that feed and display the images and give credit to the proper people? Yes, yes we can. Let’s use our friends Sinatra for our site and rack/test for testing. Install Sinatra and rack/test
gem install sinatra
gem install rack-test
We will create a feed_aggregator and a test folder inside of it. While we are at it, create a blank main and test file too.
$ mkdir feed_aggregator 
$ mkdir feed_aggregator/test 
$ touch feed_aggregator/test/feed_aggregator_test.rb
Maybe our first test is make sure the app starts up. In the test file let’s add
require '../main'  
require 'test/unit'  
require 'rack/test'  
 
ENV['RACK_ENV'] = 'test'
 
class FeedAggregatorTest < Test::Unit::TestCase
  include Rack::Test::Methods
 
  def app
    Sinatra::Application
  end
 
  def test_it_says_feed_aggregator
    get '/'
    assert last_response.ok?
    assert_equal 'Feed Aggregator', last_response.body
  end
 
end
Run the test.
test$ ruby feed_aggregator_test.rb 
<internal:lib/rubygems/custom_require>:29:in `require': no such file to load -- ../main (LoadError)
	from <internal:lib/rubygems/custom_require>:29:in `require'
	from feed_aggregator_test.rb:1:in `<main>'
Yeah! We failed the test. It says there is no main file. Go ahead and make make that file.
feed_aggregator $ touch main.rb
test$ ruby feed_aggregator_test.rb 
Loaded suite feed_aggregator_test
Started
E
Finished in 0.001054 seconds.
 
  1) Error:
test_it_says_feed_aggregator(FeedAggregatorTest):
NameError: uninitialized constant FeedAggregatorTest::Sinatra
    feed_aggregator_test.rb:11:in `app'
    /Users/johnivanoff/.rvm/gems/ruby-1.9.2-p180@feed_aggregator/gems/rack-test-0.6.2/lib/rack/test/methods.rb:31:in `build_rack_mock_session'
    /Users/johnivanoff/.rvm/gems/ruby-1.9.2-p180@feed_aggregator/gems/rack-test-0.6.2/lib/rack/test/methods.rb:27:in `rack_mock_session'
    /Users/johnivanoff/.rvm/gems/ruby-1.9.2-p180@feed_aggregator/gems/rack-test-0.6.2/lib/rack/test/methods.rb:42:in `build_rack_test_session'
    /Users/johnivanoff/.rvm/gems/ruby-1.9.2-p180@feed_aggregator/gems/rack-test-0.6.2/lib/rack/test/methods.rb:38:in `rack_test_session'
    /Users/johnivanoff/.rvm/gems/ruby-1.9.2-p180@feed_aggregator/gems/rack-test-0.6.2/lib/rack/test/methods.rb:46:in `current_session'
    feed_aggregator_test.rb:15:in `test_it_says_feed_aggregator'
 
1 tests, 0 assertions, 0 failures, 1 errors, 0 skips
Let’s add code for an app that just says “Feed Aggregator.”
require 'sinatra'
 
get '/' do
  "Feed Aggregator"
end
Do you think the test will pass now? Let’s find out.
test$ ruby feed_aggregator_test.rb 
Loaded suite feed_aggregator_test
Started
.
Finished in 0.063198 seconds.
 
1 tests, 2 assertions, 0 failures, 0 errors, 0 skips
The test does pass. Now let’s go get that feed. Since we are developing by tests, it would be a great idea to have a fixture of the RSS feed. Go ahead and make a directory and blank xml file for them.
test$ mkdir fixtures 
test$ touch fixtures/feed.xml
Excellent. Let’s copy the source of the feed into the feed.xml file. If we test against the live feed, entries could change and make it very frustrating while testing. Who needs that?
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"
	    xmlns:media="http://search.yahoo.com/mrss/"
	    xmlns:dc="http://purl.org/dc/elements/1.1/"
	    xmlns:creativeCommons="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html"
	    	    xmlns:flickr="urn:flickr:user" >
	<channel>
 
 
		<title>Flickr's Best Sunsets Pool</title>
		<link>http://www.flickr.com/groups/flickrsbestsunsets/pool/</link>
... Content elided ...
(Note: The entire XML file can be found in this gist. WOW! Where do we start? Why don’t we look for links so that a person can click on the see the original photo on Flickr. If you look at the XML you see that each picture is in an item section. The link tag will take us to the photo’s page. Based on that, we need to look into the feed, find the items, and find the link in that item. That sounds like an awesome test. Write it up.
def test_find_the_link
  feed = File.read('fixtures/feed.xml')
  items = parse feed
  item = items.first
  link = 'http://www.flickr.com/photos/mattcaustin/8205498382/in/pool-1373979@N22'
  assert_equal item[:link], link
end
Where did I get the text for the link variable? I copied the link from the first item in the fixture. You will soon see how it is used. Like I stated before, we will load and parse the fixture. The code will look in the first item to find the link node, take its text, compare it to our link variable. It should match. Let’s run the test. I hope it fails.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

E.

Finished tests in 0.025705s, 77.8059 tests/s, 77.8059 assertions/s.

  1) Error:
test_find_the_link(FeedAggregatorTest):
NoMethodError: undefined method `parse' for #<FeedAggregatorTest:0x007ff7412ac390>
    feed_aggregator_test.rb:24:in `test_find_the_link'

2 tests, 2 assertions, 0 failures, 1 errors, 0 skips
No method for ‘parse’ Go ahead and create that method in the main.rb file.
require 'sinatra'

def parse  
end

get '/' do
  "Feed Aggregator"
end
Rerun test.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

E.

Finished tests in 0.251661s, 7.9472 tests/s, 7.9472 assertions/s.

  1) Error:
test_find_the_link(FeedAggregatorTest):
ArgumentError: wrong number of arguments (1 for 0)
    /Users/john/Dropbox/feed_aggregator/main.rb:4:in `parse'
    feed_aggregator_test.rb:23:in `test_find_the_link'

2 tests, 2 assertions, 0 failures, 1 errors, 0 skips
We didn’t pass in any arguments. That’s fine since we only wanted to solve the last error. What do you do to make this pass? Back in the main.rb file
require 'sinatra'

def parse feed
end

get '/' do
  "Feed Aggregator"
end
Do you think that will get rid of the arguments error? Rerun the test and see.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

E.

Finished tests in 0.027235s, 73.4349 tests/s, 73.4349 assertions/s.

  1) Error:
test_find_the_link(FeedAggregatorTest):
NoMethodError: undefined method `first' for nil:NilClass
    feed_aggregator_test.rb:24:in `test_find_the_link'

2 tests, 2 assertions, 0 failures, 1 errors, 0 skips
Indeed it did. Now we need a ‘first’ method. In order to get to the first we will need to search through the XML in order to get the first link. Now what’s a way to parse the XML to find that link? I like (Nokogiri)[http://nokogiri.org]. Do you have that gem installed? Let’s check. You’re output might look different than mine.
test$  gem list --local -d noko

*** LOCAL GEMS ***
Apparently I don’t. If you don’t, go ahead and install it.
test$ sudo gem install nokogiri
Other that seeing in the terminal that it successfully installed, how could you check to see if it’s installed?
test$  gem list --local -d noko

*** LOCAL GEMS ***

nokogiri (1.5.5)
    Authors: Aaron Patterson, Mike Dalessio, Yoko Harada, Tim Elliott
    Rubyforge: http://rubyforge.org/projects/nokogiri
    Homepage: http://nokogiri.org
    Installed at: /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator

    Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser
Where we’re we? Parsing the XML document. You need to load the feed into Nokogiri. Then you can go through each item and get the link and store them in a hash. Don’t forget to return the items.
require 'sinatra'
require 'nokogiri'

def parse feed
  doc = Nokogiri::XML feed
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item
  end
end

get '/' do
  "Feed Aggregator"
end
Do you have a good feeling about this? Go ahead and run the test. Did you remember to include nokogiri in main.rb?
test$ ruby feed_aggregator_test.rb
Loaded suite feed_aggregator_test
Started
..
Finished in 0.112374 seconds.

2 tests, 3 assertions, 0 failures, 0 errors, 0 skips
Snoopy Dance Can we say that? We need to get the thumbnail next. Where is that in the fixture? Did you find it? It’s an attribute in the tag. Go ahead and write a test for that. It’s pretty close to the first one.
def test_find_the_thumbnail_image
  feed = File.read('fixtures/feed.xml')
  items = parse feed
  item = items.first
  thumbnail = 'http://farm9.staticflickr.com/8488/8205498382_4e5ed09a62_s.jpg'
  assert_equal item[:thumbnail], thumbnail
end
Run the test.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

.F..

Finished tests in 0.053503s, 74.7622 tests/s, 93.4527 assertions/s.

  1) Failure:
test_find_the_thumbnail_image(FeedAggregatorTest) [feed_aggregator_test.rb:42]:
<nil> expected but was
<"http://farm9.staticflickr.com/8488/8205498382_4e5ed09a62_s.jpg">.

4 tests, 5 assertions, 1 failures, 0 errors, 0 skips
That was expected. Let’s think this through. We need the value of an attribute of the media:thumbnail node. How about this?
require 'sinatra'
require 'nokogiri'

def parse feed
  doc = Nokogiri::XML feed
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item[:thumbnail] = doc_item.at('media:thumbnail').attr('url').value
    item
  end
end

get '/' do
  "Feed Aggregator"
end
That makes sense since the url we need is an attribute of that node. Go ahead and try it.
test john$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

EE.

Finished tests in 0.031776s, 94.4109 tests/s, 62.9406 assertions/s.

  1) Error:
test_find_the_link(FeedAggregatorTest):
NoMethodError: undefined method `attr' for nil:NilClass
    /Users/john/Dropbox/feed_aggregator/main.rb:10:in `block in parse'
    /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:239:in `block in each'
    /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:238:in `upto'
    /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:238:in `each'
    /Users/john/Dropbox/feed_aggregator/main.rb:7:in `map'
    /Users/john/Dropbox/feed_aggregator/main.rb:7:in `parse'
    feed_aggregator_test.rb:23:in `test_find_the_link'

  2) Error:
test_find_the_thumbnail_image(FeedAggregatorTest):
NoMethodError: undefined method `attr' for nil:NilClass
    /Users/john/Dropbox/feed_aggregator/main.rb:10:in `block in parse'
    /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:239:in `block in each'
    /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:238:in `upto'
    /Users/john/.rvm/gems/ruby-1.9.3-p194@feed_aggregator/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:238:in `each'
    /Users/john/Dropbox/feed_aggregator/main.rb:7:in `map'
    /Users/john/Dropbox/feed_aggregator/main.rb:7:in `parse'
    feed_aggregator_test.rb:31:in `test_find_the_thumbnail_image'

3 tests, 2 assertions, 0 failures, 2 errors, 0 skips
Well, that failed. undefined method ‘attr’ for nil:NilClass
It’s not finding the <media:thumbnail> node. If you look at the top of the RSS feed you can see that the media namespace is used. More about namespaces. Turns out, with Nokogiri, you can just uses the pipe symbol to indicate a namespace search. go ahead and swap out the colon and replace it with a pipe in the thumbnail line.
require 'sinatra'
require 'nokogiri'

def parse feed
  doc = Nokogiri::XML feed
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item[:thumbnail] = doc_item.at('media|thumbnail').attr('url')
    item
  end
end

get '/' do
  "Feed Aggregator"
end
Give it try. rerun the tests
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

...

Finished tests in 0.045266s, 66.2749 tests/s, 88.3665 assertions/s.

3 tests, 4 assertions, 0 failures, 0 errors, 0 skips
Awesome. We should probably use the title of the picture too. Go ahead and write the test for that. I’ll wait. Finished? Here’s what I did.
def test_find_the_title
  feed = File.read('fixtures/feed.xml')
  items = parse feed
  item = items.first
  title = 'An Evening at Shell Beach'
  assert_equal item[:title], title
end
Again, we are using the title from the first item of our fixture. Run test.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

.F.

Finished tests in 0.078211s, 38.3578 tests/s, 51.1437 assertions/s.

  1) Failure:
test_find_the_title(FeedAggregatorTest) [feed_aggregator_test.rb:34]:
<nil> expected but was
<"An Evening at Shell Beach">.

3 tests, 4 assertions, 1 failures, 0 errors, 0 skips
We need to look for the title. How would you add this to the parse method?
require 'sinatra'
require 'nokogiri'

def parse feed
  doc = Nokogiri::XML feed
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item[:thumbnail] = doc_item.at('media|thumbnail').attr('url')
    item[:title] = doc_item.at('title').text
    item
  end
end

get '/' do
  "Feed Aggregator"
end
Run the test.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

....

Finished tests in 0.062725s, 63.7704 tests/s, 79.7130 assertions/s.

4 tests, 5 assertions, 0 failures, 0 errors, 0 skips
Awesome, but let’s see something on a web page. We want results in the browser. To keep things simple I’ll use erb for making the web page.
require 'sinatra'
require 'nokogiri'


def parse feed
  doc = Nokogiri::XML feed
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item[:thumbnail] = doc_item.at('media|thumbnail').attr('url')
    item[:title] = doc_item.at('title').text
    item
  end
end

get '/' do
  erb :index
end

__END__

@@index
<!DOCTYPE html>
<html>
  <head>
  <meta charset="UTF-8">
  <meta name="viewport" content="user-scalable=yes, width=device-width" />
<title>Lovely Sunsets</title> 
</head>
<body>
  <h1>Feed Aggregator</h1>
</body>
</html>
Since we made some changes we should rerun the tests.
test$ ruby feed_aggregator_test.rb 
Run options: 

# Running tests:

...F

Finished tests in 0.251257s, 15.9200 tests/s, 19.8999 assertions/s.

  1) Failure:
test_it_says_feed_aggregator(FeedAggregatorTest) [feed_aggregator_test.rb:18]:
<"Feed Aggregator"> expected but was
<"<!DOCTYPE html>n<html>n  <head>n  <meta charset="UTF-8">n  <meta name="viewport" content="user-scalable=yes, width=device-width" />n<title>Lovely Sunsets</title> n</head>n<body>n  <h1>Feed Aggregator</h1>n</body>n</html>n">.

4 tests, 5 assertions, 1 failures, 0 errors, 0 skips
Oops. Go ahead and fix that.
def test_it_says_feed_aggregator
  get '/'
  assert last_response.ok?
  assert_match 'Feed Aggregator', last_response.body
end
We are making sure that ‘Feed Aggregator’ is within the page. Now that tests are passing, let’s move on. You re-ran the test right? We’ll add the feed url and then parse out our info.
require 'sinatra'
require 'nokogiri'

feed = File.read('test/fixtures/feed.xml')

def parse feed
  doc = Nokogiri::XML feed
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item[:thumbnail] = doc_item.at('media|thumbnail').attr('url')
    item[:title] = doc_item.at('title').text
    item
  end
end

get '/' do
  @pictures = parse feed
  erb :index
end

__END__

@@index
<!DOCTYPE html>
<html>
  <head>
  <meta charset="UTF-8">
  <meta name="viewport" content="user-scalable=yes, width=device-width" />
<title>Lovely Sunsets</title> 
</head>
<body>
  <h1>Feed Aggregator</h1>
  <dl>
    <% @pictures.each do |picture| %>
      <dt><a href="<%= picture[:link] %>"><%= picture[:title] %></a></dt>
      <dd><img src="<%= picture[:thumbnail] %>" /></dd>
    <% end %>
  </dl>
</body>
</html>
You might notice I am referring to the test fixture instead of RSS feed. Again I don’t want to constantly request from their server while developing. Go ahead and start the server and check out the lovely work in your browser. Everything is looking good. Let’s use real data. How would you wire this up to pull the feed from Flickr? Yep, let’s add open-uri to the main.rb file and then have nokogiri open the file.
require 'sinatra'
require 'nokogiri'
require 'open-uri'

feed = 'http://api.flickr.com/services/feeds/groups_pool.gne?id=1373979@N22&lang=en-us&format=rss_200'

def parse feed
  doc = Nokogiri::XML(open(feed))
  doc.search('item').map do |doc_item|
    item = {}
    item[:link] = doc_item.at('link').text
    item[:thumbnail] = doc_item.at('media|thumbnail').attr('url')
    item[:title] = doc_item.at('title').text
    item
  end
end

get '/' do
  @pictures = parse feed
  erb :index
end

__END__

@@index
<!DOCTYPE html>
<html>
  <head>
  <meta charset="UTF-8">
  <meta name="viewport" content="user-scalable=yes, width=device-width" />
<title>Lovely Sunsets</title> 
</head>
<body>
  <h1>Lovely Sunsets</h1>
  <dl>
    <% @pictures.each do |picture| %>
      <dt><a href="<%= picture[:link] %>"><%= picture[:title] %></a></dt>
      <dd><img src="<%= picture[:thumbnail] %>" /></dd>
    <% end %>
  </dl>
</body>
</html>
Fire it up and view it in the browser http://127.0.0.1:4567/ Sweet. Now you can add error handling, maybe some caching, or multiple feeds. If you would like to see an article on one of these let us know. Cheers.

Frequently Asked Questions about Writing a Feed Aggregator with Sinatra

How do I handle errors in Sinatra?

Error handling in Sinatra is quite straightforward. You can use the error method to define custom error pages. For example, you can define a custom 404 error page like this:

not_found do
'This is nowhere to be found.'
end

You can also define custom error handlers for specific exceptions. For example:

error MyCustomException do
'So sorry, something went wrong.'
end

How do I display error messages in Sinatra?

Displaying error messages in Sinatra can be done using the flash hash. The flash hash is a part of the session hash, which means it is cleared out after every request. This makes it perfect for displaying error messages. Here’s an example:

post '/login' do
user = User.find_by(email: params[:email])
if user && user.authenticate(params[:password])
session[:user_id] = user.id
redirect '/success'
else
flash[:error] = 'Invalid email or password.'
redirect '/login'
end
end

How do I report handled errors in Sinatra?

Handled errors in Sinatra can be reported using an error reporting tool like Bugsnag. Bugsnag provides a Sinatra middleware that automatically captures and reports exceptions in your Sinatra applications. Here’s how you can use it:

require 'bugsnag'

Bugsnag.configure do |config|
config.api_key = 'your-api-key-here'
end

use Bugsnag::Rack
run Sinatra::Application

How do I use Sinatra basics for handling errors?

Sinatra provides several methods for handling errors. The error method allows you to define custom error handlers. The halt method allows you to immediately stop a request and return a specific response. The pass method allows you to forward a request to the next matching route. Here’s an example:

get '/divide' do
begin
params[:a].to_i / params[:b].to_i
rescue ZeroDivisionError
halt 400, 'divided by 0'
end
end

How do I use the ‘sinatra/flash’ gem for displaying error messages?

The ‘sinatra/flash’ gem provides a way to store information for the next request. This is useful for displaying error messages. Here’s how you can use it:

require 'sinatra/flash'

enable :sessions

get '/login' do
flash[:notice] = 'You must be logged in to access this page.'
redirect '/login'
end

How do I use the ‘sinatra/reloader’ gem for error handling?

The ‘sinatra/reloader’ gem allows you to automatically reload your Sinatra application whenever a file is modified. This is useful for error handling because it allows you to see the effects of your changes immediately. Here’s how you can use it:

require 'sinatra/reloader'

configure :development do
register Sinatra::Reloader
end

How do I use the ‘sinatra/namespace’ gem for error handling?

The ‘sinatra/namespace’ gem allows you to organize your routes into namespaces. This is useful for error handling because it allows you to define error handlers for specific namespaces. Here’s how you can use it:

require 'sinatra/namespace'

register Sinatra::Namespace

namespace '/api' do
error do
'API error occurred.'
end
end

How do I use the ‘sinatra/respond_with’ gem for error handling?

The ‘sinatra/respond_with’ gem allows you to automatically choose the best format for your response based on the request’s Accept header. This is useful for error handling because it allows you to return error messages in the format that the client can best handle. Here’s how you can use it:

require 'sinatra/respond_with'

register Sinatra::RespondWith

error do
respond_with :error
end

How do I use the ‘sinatra/cookies’ gem for error handling?

The ‘sinatra/cookies’ gem provides a way to store information in cookies. This is useful for error handling because it allows you to store error messages in cookies. Here’s how you can use it:

require 'sinatra/cookies'

helpers Sinatra::Cookies

get '/login' do
cookies[:error] = 'Invalid email or password.'
redirect '/login'
end

How do I use the ‘sinatra/base’ gem for error handling?

The ‘sinatra/base’ gem provides a way to create modular Sinatra applications. This is useful for error handling because it allows you to define error handlers in a modular way. Here’s how you can use it:

require 'sinatra/base'

class MyApp < Sinatra::Base
error do
'An error occurred.'
end
end

John IvanoffJohn Ivanoff
View Author

John is a Dallas-based front-end/back-end web developer with 15+ years experience. His professional growth has come from big corporate day jobs and weekend freelance. His is enjoys working the Ruby the most these days and has even added pain to the process by developing Rails for Windows! He’s had many years of enjoyment with Cold Fusion and has strong background in web standards.

Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week