Introduction to Messaging Systems for Rubyists

What is Messaging

As the Ruby community matures and our applications grow, we seek new ways to manage complexity, reduce coupling, and improve scalability. Though REST services are a good solution for a broad spectrum of problems, the temporal coupling is getting harder to manage when the traffic and the number of moving parts is growing. If processing of every request requires several remote calls it’s almost impossible to guarantee quick response time.

A well-known solution to this problem that has gained some popularity in the Ruby community recently is messaging. The core idea behind all message based systems and patterns is very simple:

  • You don’t call other applications through REST or SOAP synchronously.
  • You send messages to a message broker. The message broker will deliver messages asynchronously to other applications (or just workers inside your application).

As a result your application can respond without being blocked by external resources.

Message Broker

Why do we need a message broker? Using a broker and not sending messages directly to other applications brings a few very significant advantages. You don’t need to manage storing and delivering messages. A message broker does it. It guarantees delivery, durability and can provide some additional services such as filtering, logging, failover etc. But the main advantage of using a message broker is keeping different parts of your application (or different services) independent from each other:

  • It provides temporal decoupling, so you can redeploy all your services independently. If one of your services is unavailable or blocked (for instance a third-party web service went down) your application can continue working and respond quickly. A message broker keeps all messages, and, as soon as the problem is resolved, they will be processed.
  • It provides structural decoupling. The broker does not do it per se, however, having a message broker as a part of a messaging infrastructure makes it is much easier to add application independent message transformations.

A message broker is an additional level of indirection that is really useful. It minimizes the amount of information two parts of your system should know about each other. For instance, a sender may not be aware that all its messages are processed by many clients, that some transformations are performed before its messages are delivered to clients etc. The only thing the sender should worry about is the broker that recieves the messages.

The AMQP Model

Several mature messaging systems have been developed over the years, such as MSMQ, ActiveMQ, OpenMQ, ZeroMQ, RabbitMQ etc. In my view, the systems based on AMQP standard are the best fit for Rubyists. One of which – RabbitMQ – I’m going to use in this article to demonstrate the very basic scenarios of using messaging.

But before I get down to business, I’d like to cover some very basics of the AMQP model:

  • Message Broker. A message broker is a system taking incoming messages from one application and delivering them to another. RabbitMQ server is a message broker.
  • Producer. It is a system sending messages to a message broker.
  • Consumer. It is a system receiving messages from a message broker.
  • Exchange. It is an entry point for all incoming messages. Producers send messages to exchanges.
  • Queue. It is an entity where all messages are stored. Consumers read messages from queues.
  • Binding. A relationship between an exchange and a queue.

The basic workflow:

  • Some configuration code in your application defines an exchange E and a queue Q.
  • It also binds E and Q. So every message sent to E will be stored in Q.
  • Your application (Producer) sends messages to an exchange E.
  • RabbitMQ routes all incoming messages from E to Q. All of them are stored in Q.
  • Another application (Consumer) reads messages from Q.

As you can see all interactions between Producer and Consumer are happening through the message broker. They don’t interact with each other directly. That leads to loose coupling.

Bunny

There are a huge variety of ruby gems that can be used for message passing. My current choice is Bunny, as I find it being the simplest one. It does not require any knowledge of Event Machine or asynchronous programming.

Examples

Enough theoretical background, let’s take a look at how it’s done in practice.

The first example I’d like to show is the 1-exchange-1-queue case:

Notes

  • All interactions with RabbitMQ are happening through an instance of Bunny (a message broker).
  • There are two versions of AMQP specification that are in use right now: 08 and 09. I’m using 0.9 in this example.
  • Creating queues and exchanges is idempotent. So if a queue with the given name has been already created Bunny will just return it without creating a new one.
  • Reading a message from a queue does not return the payload that you have published. It returns an object containing the payload and all corresponding headers. For now you can just ignore the headers and read the payload.
  • We are using a “Direct” exchange in this example. There are several types of exchanges that AMQP supports. Describing all of them is out of the scope of this article.
  • The pop method returns a special :empty_queue message when there are no messages in the queue.

The queue that has been created in the previous example is stored in memory. So you will lose all unprocessed messages if you restart RabbitMQ. When information that you send is not critical, using transient queues is the best choice (mostly performance-wise). When it’s not an option you can always configure RabbitMQ flush your queue to the disk.

Notes

  • Only persistent messages sent to a durable queue are stored on disk.

Another scenario that you may see quite regularly is the 1-producer-n-consumers case. There are several consumers reading messages from the same queue. Every message, as in the previous examples, will be processed only once.

Notes

  • In real life every consumer is a separate process. So if one of them goes down the system will continue processing messages.

The last example is the Publish/Subscribe case. There are a producer and N consumers and all the consumers will receive all sent messages. There are many ways to implement this scenario in AMQP. One method is to create a queue per consumer and bind them to the same exchange. RabbitMQ when receiving a message will deliver it to all the queues.

You Can Do More

The four examples is just to get you started. There are a lot of options that you can consider when you are adding messaging to your application such as:

  • Implementing clients using “pop” or “subscribe”.
  • Different exchange types such as: fanout or topic.
  • Using transactions

Read More About Messaging

Though the idea of messaging is simple, the number of patterns that you can implement on top of it is enormous. The best book on the topic is “Enterprise Integration Patterns” by G. Hohpe and B. Woolf. I highly recommend reading this book if you are thinking about adding messaging to your application.

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • Wes Shaddix

    I like the article, thanks for sharing. I currently develop/maintain a large scale messaging system for service provisioning in .Net. Based on this article, I’m curious how you would be able to abstract the exchange/queue binding knowledge from the AMQP clients. I have many different producers and I don’t want each of them to have to know about the queue bindings, only the exchange. The way I’m thinking about this type of system is that the AMQP exchange serves as your “public api”, so I don’t want clients to have to know or care about the internals. Just want them to send a message to an endpoint and forget about the rest.

    Any insight that you can offer?

    • http://victorsavkin.com Vic Savkin

      Thanks for your comment Wes. Answering your question:

      1. You can always configure all your exchanges and bindings outside of your applications (for instance, using rabbitmq management plugin).
      2. You can keep your configuration under version control as a ruby script and run it when it’s needed. This script will create all exchanges and bindings. As creating exchanges and bindings is idempotent you can run the script as many times as you want (for example, as a part of your deployment).
      3. You can always have this configuration inside your application as a separate initialization step. Each time you redeploy it will run the configuration.

      Regardless of the chosen option your producers should never know about bindings between exchanges and queues. This is the main point of having a message broker.

      Cheers,
      Victor

  • http://tropo.com Jason Goecke

    Have you had a look at DCell (https://github.com/tarcieri/dcell) built on Zeromq and part of the Celluloid (https://github.com/tarcieri/celluloid) family?

    • http://victorsavkin.com Vic Savkin

      Thanks for your comment. I read about DCell and I think that the actor model is a way to go. However, I haven’t tried DCell yet.

      The project looks really interesting and I really want to try it.

      Cheers,
      Victor