Propono: Pub/Sub for Ruby

Building a Service Orientated Architecture

The concept of the “monorail” is long established in the Ruby world – a large Rails application that tries to do everything but quickly becomes cumbersome and overbearing. Developers who have found themselves in this situation are now looking for a better way to work.

One way to solve the problem is to split a Rails app into various pieces that work together in a Service Orientated Architecture (SOA). By leaving the web-request and data-layer functionality to the Rails application and moving the more complex, less CRUD code into separate services, it is much easier to separate concerns and choose the right tools for the right jobs. In some cases the best tool for many of these jobs might be a different language or system (e.g. Erlang for distributed systems, Java for times when you need specific libraries), but I have found that Ruby is generally a good default language to use if nothing else is jumping out as the obvious choice.

Last year, my team at Meducation decided to split our “monorail” into a service-oriented architecture. Over the last 12 months, we’ve been breaking it apart and now have 21 different applications that work together. Each service has very separate concerns and knowledge.

We effectively treat the Single Responsibility Principle as true for applications, not just classes. Creating and integrating new applications into our ecosystem is fast and cheap – in fact our record for creating and deploying a fully functional, integrated application is under an hour. The key to achieving this has been building robust tools that we trust, which allow us to focus entirely on the business logic of our application, not on how it all pieces together. This article is on the first of these tools – Propono.

Propono (the Latin verb for publish) is a pub/sub Ruby Gem built on top of Amazon Web Services (AWS). It offers the following benefits:

• It’s exceptionally simple to setup and use.
• Everything is handled natively by AWS, which means:
  • Automatic scaleability.
  • Exceptional reliability.
  • Almost zero cost – We publish hundreds of thousands of messages a month and spend under $5.
  • No third party security risks other than Amazon.

What is Pub/Sub?

We described Propono as a pub/sub Ruby Gem. Let’s quickly break down what that means.

Pub/Sub (publish/subscribe) is a very simple concept. You have a number of applications that want to send messages (publishers) and a number of applications that want to receive messages (subscribers). Often the same application will publish some messages and subscribe to others. Messages are published to topics, and subscribers listen to the topics they care about. See…simple.

Image Source: docs.oracle.com

For example, at Meducation we have a topic called “user” that many applications subscribe to. When a new user signs up, the website publish a JSON message to that topic that in effect says “A user has signed up and now has an id of #239372.” The subscribed applications receive that message and then do things like update our search indexes, generate friend suggestions and send a welcome email.

With Propono, as long as you have an AWS account, then both publishing this message in the website and subscribing in the applications are one liners. Let’s dive into a basic example, and then take a look at how Propono works with AWS under the hood.

Propono 101

To get started with Propono you need an AWS account and an IAM user that has permissions for SNS and SQS. If you’re just trying Propono, you can probably experiment with your normal IAM user. However, in production I’d recommend setting up a specific user with a policy that allows access to SQS and SNS. e.g.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1401034357000", "Effect": "Allow", "Action": [ "sqs:*", "sns:*" ], "Resource": [ "*" ] } ] }

We’re going to create two applications that can talk to each other – one publisher and one subscriber. To get started, create a directory called learning_propono and add a Gemfile that looks like this:

source 'https://rubygems.org'
gem 'propono'

Run bundle and you should get a Gemfile.lock.

We’re now ready to create our publisher and subscriber. Let’s create a file for each of them: publisher.rb and subscriber.rb. Both need to require and configure Propono with your IAM credentials. Open them both in your favorite editor and populate them with the following:

require 'propono'
Propono.config do |config|
  config.access_key       = "Your-IAM-Access-Key"
  config.secret_key       = "Your-IAM-Secret-Key"
  config.queue_region     = "eu-west-1"
  config.application_name = "Either-publisher-or-subscriber"
end

That’s all you need to configure Propono. In our apps, we tend to put the values in a propono.yml file that we inject during deployment (via Chef) and read at runtime.

In subscriber.rb, listen for messages on the ‘user’ topic. That’s as simple as doing the following:

Propono.listen_to_queue(:user) do |message|
  p "Message Received"
  p message
end

Spin up a console and run bundle exec ruby subscriber.rb. Your subscriber will sit and wait for messages on the user queue. If you get an error at this point, check your IAM credentials.

Leave that running, and head back to publisher.rb in your editor to publish a message about a new user to the user topic. Below the Propono config, put the following:

message = {entity: 'user', action: 'created', id: 123123} # Any string, hash, or array is valid here.
Propono.publish(:user, message, async: false)

By default, messages are published in separate threads. As we want to make sure this thread completes before the main thread finishes and the application dies, we call async: false to publish in the same thread.

With the subscriber still running, in a different console, run bundle exec ruby publisher.rb. Watch your subscriber window and within a second or two you should see the message printed out.

You now have all the power of pub/sub at your fingertips. You can publish messages from any application, and subscribe to them from any other. Simple.

What’s Actually Going On?

Under the surface, Propono utilizes Amazon’s Simple Notification Service (SNS) and Simple Queue Service (SQS) to store and disseminate messages. I’ll give a quick introduction to both services then explain how we utilize them with Propono.

Simple Notification Service (SNS)

On the publishing side we have SNS. Amazon’s describes SNS as “a fast, flexible, fully managed push messaging service… To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple availability zones… It costs $1.00 to send one million mobile push notifications…. With SNS you can publish a message once, and deliver it one or more times.”

To translate that marketing speak, SNS is a very cheap, robust service that takes messages and delivers them to one or more recipients. SNS has a multitude of possible uses, but in the Propono world, it provides the topics to which we publish messages. When we publish a message, we are sending it to SNS, which then looks for any other services that care about it, and passes the message on.

Simple Queue Service (SQS)

At the other end are our subscribers who use SQS to receive messages. Amazon describes SQS as “a fast, reliable, scalable, fully managed message queuing service. You can use SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available.”

In a nutshell, SQS queues receive messages and keep them there until a service takes them off. The cost is again minimal. Under the hood in Propono, your application is subscribing to an SQS queue and pulling off messages as they arrive. If your application goes down, the messages will queue up until it comes back up again.

A Message Lifecycle

Let’s explore the journey of a message starting from the line:

Propono.publish(:user, {foo: 'bar'})

The first thing that happens when this code is run is that Propono creates the user topic on SNS. After executing that code, if you look at your SNS console, you will see the user topic. At this stage, it will have no subscriptions, so it will, in effect, ignore any messages that are sent to it.

Now, let’s look at the other side

Propono.listen_to_queue(:user) do |message|
  # ...
end

The code checks for the existence of the user topic on SNS and creates it if it doesn’t exist. However, it also looks for the existence of a queue for these messages for this application on the SQS side. When you configured the subscriber, you chose an application name. That application name now gets concatenated with the topic name, to give us the queue name.

For example, if you called your application subscriber, executing the above code will create a queue called subscriber-user. Take a look at your SQS console and you’ll see that the topic has been created:

Head back to your SNS console and click on the user topic, you’ll notice that a new subscription has been created between the SNS topic and SQS queue. Now any messages published to the user topic will be sent to the subscriber-user topic and stored until the subscriber reads them off by using listen_to_queue.

All that is setup for free with security policies and mappings handled transparently by Propono. In fact, lots of other stuff happens too. You have probably noticed that there are error queues, corrupt queues, slow topics, and lots of other bits and pieces that have been created. I’ll be covering all that, and lots of the other ways in which you can use Propono in the future.

I hope this has been a useful introduction to Propono and that you’ll give it a try. We use it heavily in production and there are quite a few other companies now doing the same. We’d love to hear your feedback and really love your contributions. Please feel free to get involved the Propono Github Repository or in the comments section below. Thanks for reading!

Frequently Asked Questions (FAQs) about Propono PubSub in Ruby