Propono: Pub/Sub for Ruby

Share this article

Building a Service Orientated Architecture

The concept of the “monorail” is long established in the Ruby world – a large Rails application that tries to do everything but quickly becomes cumbersome and overbearing. Developers who have found themselves in this situation are now looking for a better way to work.

One way to solve the problem is to split a Rails app into various pieces that work together in a Service Orientated Architecture (SOA). By leaving the web-request and data-layer functionality to the Rails application and moving the more complex, less CRUD code into separate services, it is much easier to separate concerns and choose the right tools for the right jobs. In some cases the best tool for many of these jobs might be a different language or system (e.g. Erlang for distributed systems, Java for times when you need specific libraries), but I have found that Ruby is generally a good default language to use if nothing else is jumping out as the obvious choice.

Last year, my team at Meducation decided to split our “monorail” into a service-oriented architecture. Over the last 12 months, we’ve been breaking it apart and now have 21 different applications that work together. Each service has very separate concerns and knowledge.

We effectively treat the Single Responsibility Principle as true for applications, not just classes. Creating and integrating new applications into our ecosystem is fast and cheap – in fact our record for creating and deploying a fully functional, integrated application is under an hour. The key to achieving this has been building robust tools that we trust, which allow us to focus entirely on the business logic of our application, not on how it all pieces together. This article is on the first of these tools – Propono.

Propono (the Latin verb for publish) is a pub/sub Ruby Gem built on top of Amazon Web Services (AWS). It offers the following benefits:

  • It’s exceptionally simple to setup and use.
  • Everything is handled natively by AWS, which means:
    • Automatic scaleability.
    • Exceptional reliability.
    • Almost zero cost – We publish hundreds of thousands of messages a month and spend under $5.
    • No third party security risks other than Amazon.

What is Pub/Sub?

We described Propono as a pub/sub Ruby Gem. Let’s quickly break down what that means.

Pub/Sub (publish/subscribe) is a very simple concept. You have a number of applications that want to send messages (publishers) and a number of applications that want to receive messages (subscribers). Often the same application will publish some messages and subscribe to others. Messages are published to topics, and subscribers listen to the topics they care about. See…simple.

Image Source: docs.oracle.com

For example, at Meducation we have a topic called “user” that many applications subscribe to. When a new user signs up, the website publish a JSON message to that topic that in effect says “A user has signed up and now has an id of #239372.” The subscribed applications receive that message and then do things like update our search indexes, generate friend suggestions and send a welcome email.

With Propono, as long as you have an AWS account, then both publishing this message in the website and subscribing in the applications are one liners. Let’s dive into a basic example, and then take a look at how Propono works with AWS under the hood.

Propono 101

To get started with Propono you need an AWS account and an IAM user that has permissions for SNS and SQS. If you’re just trying Propono, you can probably experiment with your normal IAM user. However, in production I’d recommend setting up a specific user with a policy that allows access to SQS and SNS. e.g.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1401034357000", "Effect": "Allow", "Action": [ "sqs:*", "sns:*" ], "Resource": [ "*" ] } ] }

We’re going to create two applications that can talk to each other – one publisher and one subscriber. To get started, create a directory called learning_propono and add a Gemfile that looks like this:

source 'https://rubygems.org'
gem 'propono'

Run bundle and you should get a Gemfile.lock.

We’re now ready to create our publisher and subscriber. Let’s create a file for each of them: publisher.rb and subscriber.rb. Both need to require and configure Propono with your IAM credentials. Open them both in your favorite editor and populate them with the following:

require 'propono'
Propono.config do |config|
  config.access_key       = "Your-IAM-Access-Key"
  config.secret_key       = "Your-IAM-Secret-Key"
  config.queue_region     = "eu-west-1"
  config.application_name = "Either-publisher-or-subscriber"
end

That’s all you need to configure Propono. In our apps, we tend to put the values in a propono.yml file that we inject during deployment (via Chef) and read at runtime.

In subscriber.rb, listen for messages on the ‘user’ topic. That’s as simple as doing the following:

Propono.listen_to_queue(:user) do |message|
  p "Message Received"
  p message
end

Spin up a console and run bundle exec ruby subscriber.rb. Your subscriber will sit and wait for messages on the user queue. If you get an error at this point, check your IAM credentials.

Leave that running, and head back to publisher.rb in your editor to publish a message about a new user to the user topic. Below the Propono config, put the following:

message = {entity: 'user', action: 'created', id: 123123} # Any string, hash, or array is valid here.
Propono.publish(:user, message, async: false)

By default, messages are published in separate threads. As we want to make sure this thread completes before the main thread finishes and the application dies, we call async: false to publish in the same thread.

With the subscriber still running, in a different console, run bundle exec ruby publisher.rb. Watch your subscriber window and within a second or two you should see the message printed out.

You now have all the power of pub/sub at your fingertips. You can publish messages from any application, and subscribe to them from any other. Simple.

What’s Actually Going On?

Under the surface, Propono utilizes Amazon’s Simple Notification Service (SNS) and Simple Queue Service (SQS) to store and disseminate messages. I’ll give a quick introduction to both services then explain how we utilize them with Propono.

Simple Notification Service (SNS)

On the publishing side we have SNS. Amazon’s describes SNS as “a fast, flexible, fully managed push messaging service… To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple availability zones… It costs $1.00 to send one million mobile push notifications…. With SNS you can publish a message once, and deliver it one or more times.”

To translate that marketing speak, SNS is a very cheap, robust service that takes messages and delivers them to one or more recipients. SNS has a multitude of possible uses, but in the Propono world, it provides the topics to which we publish messages. When we publish a message, we are sending it to SNS, which then looks for any other services that care about it, and passes the message on.

Simple Queue Service (SQS)

At the other end are our subscribers who use SQS to receive messages. Amazon describes SQS as “a fast, reliable, scalable, fully managed message queuing service. You can use SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available.”

In a nutshell, SQS queues receive messages and keep them there until a service takes them off. The cost is again minimal. Under the hood in Propono, your application is subscribing to an SQS queue and pulling off messages as they arrive. If your application goes down, the messages will queue up until it comes back up again.

A Message Lifecycle

Let’s explore the journey of a message starting from the line:

Propono.publish(:user, {foo: 'bar'})

The first thing that happens when this code is run is that Propono creates the user topic on SNS. After executing that code, if you look at your SNS console, you will see the user topic. At this stage, it will have no subscriptions, so it will, in effect, ignore any messages that are sent to it.

Now, let’s look at the other side

Propono.listen_to_queue(:user) do |message|
  # ...
end

The code checks for the existence of the user topic on SNS and creates it if it doesn’t exist. However, it also looks for the existence of a queue for these messages for this application on the SQS side. When you configured the subscriber, you chose an application name. That application name now gets concatenated with the topic name, to give us the queue name.

For example, if you called your application subscriber, executing the above code will create a queue called subscriber-user. Take a look at your SQS console and you’ll see that the topic has been created:

Head back to your SNS console and click on the user topic, you’ll notice that a new subscription has been created between the SNS topic and SQS queue. Now any messages published to the user topic will be sent to the subscriber-user topic and stored until the subscriber reads them off by using listen_to_queue.

All that is setup for free with security policies and mappings handled transparently by Propono. In fact, lots of other stuff happens too. You have probably noticed that there are error queues, corrupt queues, slow topics, and lots of other bits and pieces that have been created. I’ll be covering all that, and lots of the other ways in which you can use Propono in the future.

I hope this has been a useful introduction to Propono and that you’ll give it a try. We use it heavily in production and there are quite a few other companies now doing the same. We’d love to hear your feedback and really love your contributions. Please feel free to get involved the Propono Github Repository or in the comments section below. Thanks for reading!

Frequently Asked Questions (FAQs) about Propono PubSub in Ruby

What is Propono PubSub in Ruby?

Propono PubSub is a Ruby-based library that allows for easy implementation of the publish-subscribe pattern in your applications. It uses Amazon Web Services (AWS) Simple Notification Service (SNS) and Simple Queue Service (SQS) to handle the messaging infrastructure. This allows for efficient, scalable, and reliable message distribution between different parts of your application.

How does Propono PubSub compare to DRb in Ruby?

While both Propono PubSub and DRb (Distributed Ruby) allow for communication between different parts of your application, they do so in different ways. DRb uses a client-server model, where one part of your application acts as a server and other parts act as clients. Propono PubSub, on the other hand, uses a publish-subscribe model, where messages are published to a topic and then delivered to all subscribers of that topic. This makes Propono PubSub more scalable and better suited for distributed systems.

How do I install and set up Propono PubSub in Ruby?

To install Propono PubSub, you need to add the gem to your Gemfile and run the bundle install command. Setting it up involves configuring it with your AWS credentials and setting up your topics and queues in AWS SNS and SQS. Detailed instructions can be found in the Propono PubSub documentation.

Can I use Propono PubSub with other programming languages?

Propono PubSub is a Ruby-based library, so it’s primarily designed to be used with Ruby. However, since it uses AWS SNS and SQS for messaging, it’s possible to interact with the messages from other programming languages that have AWS SDKs.

What are the benefits of using Propono PubSub?

Propono PubSub provides a simple and easy-to-use interface for implementing the publish-subscribe pattern in your applications. It handles all the complexities of setting up and managing the messaging infrastructure, allowing you to focus on your application logic. It’s also highly scalable and reliable, thanks to the underlying AWS services.

Are there any limitations or drawbacks to using Propono PubSub?

One potential drawback of Propono PubSub is that it relies on AWS services, so you need to have an AWS account and be comfortable with AWS pricing and usage limits. Also, while it’s highly scalable, there may be some latency in message delivery due to the distributed nature of AWS services.

How does Propono PubSub handle message delivery and reliability?

Propono PubSub uses AWS SQS to ensure reliable message delivery. Messages are stored in a queue until they are successfully processed by the subscriber. If a message fails to be processed, it remains in the queue and can be retried.

Can I use Propono PubSub for real-time applications?

While Propono PubSub is highly reliable and scalable, it may not be suitable for real-time applications due to the potential latency in message delivery. For real-time applications, you might want to consider other solutions like WebSockets or real-time APIs.

How secure is Propono PubSub?

Propono PubSub uses AWS services for messaging, which provide a high level of security. Messages are encrypted in transit and at rest, and you can control access to your topics and queues using AWS Identity and Access Management (IAM).

Where can I find more resources and support for Propono PubSub?

You can find more information and support for Propono PubSub in the official documentation and on the GitHub page. There’s also a community of users and contributors who can provide help and advice.

Jeremy WalkerJeremy Walker
View Author

Jeremy Walker has been using Rails since 2005. He is the CTO of Meducation - an educational social-network for medics - and runs his own software consultancy. He is a maintainer of various open source projects, including Propono, Larva and Inqusitio. You can follow Jeremy on Github and Twitter, and read more about him at his website.

GlennG
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week
Loading form