Introduction to Messaging Systems for Rubyists

What is Messaging

As the Ruby community matures and our applications grow, we seek new ways to manage complexity, reduce coupling, and improve scalability. Though REST services are a good solution for a broad spectrum of problems, the temporal coupling is getting harder to manage when the traffic and the number of moving parts is growing. If processing of every request requires several remote calls it’s almost impossible to guarantee quick response time.

A well-known solution to this problem that has gained some popularity in the Ruby community recently is messaging. The core idea behind all message based systems and patterns is very simple:

• You don’t call other applications through REST or SOAP synchronously.
• You send messages to a message broker. The message broker will deliver messages asynchronously to other applications (or just workers inside your application).

As a result your application can respond without being blocked by external resources.

Message Broker

Why do we need a message broker? Using a broker and not sending messages directly to other applications brings a few very significant advantages. You don’t need to manage storing and delivering messages. A message broker does it. It guarantees delivery, durability and can provide some additional services such as filtering, logging, failover etc. But the main advantage of using a message broker is keeping different parts of your application (or different services) independent from each other:

• It provides temporal decoupling, so you can redeploy all your services independently. If one of your services is unavailable or blocked (for instance a third-party web service went down) your application can continue working and respond quickly. A message broker keeps all messages, and, as soon as the problem is resolved, they will be processed.
• It provides structural decoupling. The broker does not do it per se, however, having a message broker as a part of a messaging infrastructure makes it is much easier to add application independent message transformations.

A message broker is an additional level of indirection that is really useful. It minimizes the amount of information two parts of your system should know about each other. For instance, a sender may not be aware that all its messages are processed by many clients, that some transformations are performed before its messages are delivered to clients etc. The only thing the sender should worry about is the broker that recieves the messages.

The AMQP Model

Several mature messaging systems have been developed over the years, such as MSMQ, ActiveMQ, OpenMQ, ZeroMQ, RabbitMQ etc. In my view, the systems based on AMQP standard are the best fit for Rubyists. One of which – RabbitMQ – I’m going to use in this article to demonstrate the very basic scenarios of using messaging.

But before I get down to business, I’d like to cover some very basics of the AMQP model:

• Message Broker. A message broker is a system taking incoming messages from one application and delivering them to another. RabbitMQ server is a message broker.
• Producer. It is a system sending messages to a message broker.
• Consumer. It is a system receiving messages from a message broker.
• Exchange. It is an entry point for all incoming messages. Producers send messages to exchanges.
• Queue. It is an entity where all messages are stored. Consumers read messages from queues.
• Binding. A relationship between an exchange and a queue.

The basic workflow:

• Some configuration code in your application defines an exchange E and a queue Q.
• It also binds E and Q. So every message sent to E will be stored in Q.
• Your application (Producer) sends messages to an exchange E.
• RabbitMQ routes all incoming messages from E to Q. All of them are stored in Q.
• Another application (Consumer) reads messages from Q.

As you can see all interactions between Producer and Consumer are happening through the message broker. They don’t interact with each other directly. That leads to loose coupling.

Bunny

There are a huge variety of ruby gems that can be used for message passing. My current choice is Bunny, as I find it being the simplest one. It does not require any knowledge of Event Machine or asynchronous programming.

Examples

Enough theoretical background, let’s take a look at how it’s done in practice.

The first example I’d like to show is the 1-exchange-1-queue case:

[gist id=”1812170″]

Notes

• All interactions with RabbitMQ are happening through an instance of Bunny (a message broker).
• There are two versions of AMQP specification that are in use right now: 08 and 09. I’m using 0.9 in this example.
• Creating queues and exchanges is idempotent. So if a queue with the given name has been already created Bunny will just return it without creating a new one.
• Reading a message from a queue does not return the payload that you have published. It returns an object containing the payload and all corresponding headers. For now you can just ignore the headers and read the payload.
• We are using a “Direct” exchange in this example. There are several types of exchanges that AMQP supports. Describing all of them is out of the scope of this article.
• The pop method returns a special :empty_queue message when there are no messages in the queue.

The queue that has been created in the previous example is stored in memory. So you will lose all unprocessed messages if you restart RabbitMQ. When information that you send is not critical, using transient queues is the best choice (mostly performance-wise). When it’s not an option you can always configure RabbitMQ flush your queue to the disk.

[gist id=”1812186″]

Notes

• Only persistent messages sent to a durable queue are stored on disk.

Another scenario that you may see quite regularly is the 1-producer-n-consumers case. There are several consumers reading messages from the same queue. Every message, as in the previous examples, will be processed only once.

[gist id=”1812209″]

Notes

• In real life every consumer is a separate process. So if one of them goes down the system will continue processing messages.

The last example is the Publish/Subscribe case. There are a producer and N consumers and all the consumers will receive all sent messages. There are many ways to implement this scenario in AMQP. One method is to create a queue per consumer and bind them to the same exchange. RabbitMQ when receiving a message will deliver it to all the queues.

[gist id=”1812216″]

You Can Do More

The four examples is just to get you started. There are a lot of options that you can consider when you are adding messaging to your application such as:

• Implementing clients using “pop” or “subscribe”.
• Different exchange types such as: fanout or topic.
• Using transactions

Read More About Messaging

Though the idea of messaging is simple, the number of patterns that you can implement on top of it is enormous. The best book on the topic is “Enterprise Integration Patterns” by G. Hohpe and B. Woolf. I highly recommend reading this book if you are thinking about adding messaging to your application.