Rails Disco: Get Down with Event Sourcing

Aside: I feel like you should play “You Should Be Dancing” by the Bee Gees right now. Once it’s going in the background, read on…

Here are some questions for you:

• What did your data look like last year? Last week?
• How many different client applications are using your single data store? How much data transformation are they doing to use the data?
• Have you ever changed your model, resulting in a tedious data migration to add the new attributes to the existing data?
• Have you ever wanted to change technologies around your data (data store, for example), only to say “No” because the migration was too hard?

I recently had to answer these questions, and my answers were terrible. So, I asked myself another question: What can I do to have better answers to these (and other) questions in the future?

After some research, the answer that popped up was “Event Sourcing”.

Event Sourcing 101

What is Event Sourcing? The short answer is, storing the events that occurred to get our data/application to it’s current state. Here is what Martin Fowler has to say:

The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself.

That linked article is worth the read. The “big brain” in the Event Sourcing space is Greg Young and he has several fantastic resources around Event Sourcing, including:

• Presentation: “CQRS and Event Sourcing” (Code on the Beach)
• Video: “Polyglot Data”
• Article: “Why Use Event Sourcing?”
• Software: EventStore is probably the biggest piece of software that is dedicated to Event Sourcing. Unfortunately, there’s no Ruby wrapper for it, just Java and C#. Regardless, its docs are definitely worth a read.

A quick Google of “Greg Young Event Sourcing” will bring even more resources.

Mr. Young’s presentation above mentions CQRS, which stands for Command Query Responsibilty Segregation. The fundamental concept behind CQRS is separating the things that change your data (commands) from the things that ask for your data (queries). The separation can be different classes or entirely different applications. Again, we’ll turn to Mr. Fowler:

[CQRS] At its heart is a simple notion that you can use a different model to update information than the model you use to read information. By separate models we most commonly mean different object models, probably running in different logical processes, perhaps on separate hardware.

Event Sourcing and CQRS are peas and carrots, chocolate and peanut butter. I strongly recommend that you read up on these two concepts a bit before continuing this article. It’ll make rails-disco easier to digest.

I’ll run through a quick scenario to make things a bit clearer and introduce a couple of concepts. Imagine an application that builds sports teams with players. The domain consists of a Team model and a Player model.

Using Event Sourcing, when the user creates a team, the result is a CreatedTeamEvent with the team data. When a player is added to a team, a CreatedPlayerEvent is created. The event store looks like this:

CreatedTeamEvent {"name": "Cowboys"}
CreatedTeamEvent {"name": "Redskins"}
CreatedPlayerEvent {"name": "Troy Aikman", "number": "8", "position": "QB", "team": "Cowboys"}
DeletedTeamEvent {"team": "Redskins", "reason": "Because they are awful"}

The question is, what does the application show to the user when all teams are requested? It could replay the events everytime teams are requested. However, it’s not scalable. One of the tenets of Event Sourcing is NEVER deleting data. Ever. Let that sink in. You can’t delete events, because the events MUST be replayable to ensure you end up with the correct application state. So, replaying events for each query might work for a bit, but as the number of events grows, the query becomes a bottleneck. What do you do?

In Event Sourcing speak, the answer is: Projections. A “projection” is a rolling snapshot of the data. In essense, a projection consumes the events and writes the data to the specification of a particular client. For a Rails app, we might create a Team model and create a team for each CreatedTeamEvent in the event store. Our projection, basically, subscribes to the events, like an RSS/Atom feed. It’s important to understand that the projection is responsible for managing its own subscription, again like an RSS/Atom client.

Projections can be added to answer many common questions/scenarios:

• Want to switch to a new data store for your Rails app? Create a projection that consumes the events and writes the data to the new store. Then, start from event 0 and go through all events.
• Want a separate data store for reports that uses a more de-normalized data structure? Done.
• Want to know what players were on the team last summer? Done.
• Need to make drastic changes to your data model? Do it, change the projection to handle the new model, and run the events starting from event 0.

The possibilities really are far out, outta sight, DY-NO-MITE!

Rails Disco

Now that you understand the basics of Event Sourcing, how can a bell-bottom wearing Rubyist get it on? The answer is the rails-disco gem from the funky cats at Hicknhack Software.

Rails Disco is “A distributed party with commands, events and projections”. It is made up of 3 gems:

(Image provided by Hicknhack. They boogie down in Germany.)

• Active Domain – “A framework that allows to write a domain server, that processes commands and creates events.”
• Active Event – Contains events, commands, and validations for the events that go into the event store.
• Active Projection – The projections to consume events and create the Rails database models.

Rails Disco presumes a certain infrastructure to get funky:

• A distributed Ruby (DRb) server is used to communicate commands to the domain.
• A RabbitMQ exchange is used to publish events from the domain to the projection. As such, you must have RabbitMQ installed and running.
• A streaming web server (like Puma or Thin) must be used. Event source changes are communicated to the client using streaming. Disco will add Puma to your Gemfile, by default.

Rails Disco comes with a disco executable that allows a Rails developer to generate a “disco scaffold” (BTW, I LOVE putting “disco” in front of other words. Favorite. Gem name. Ever.) As such, it also adds some generators called by the disco command that I’ll use it a bit to get the app setup.

High Level Flow

(Image provided by Hicknhack. Nutzer mean “user”. Disco Nutzer.)

The standard flow of data through a Rails Disco app is as follows:

1. User issues a request that changes data, like “create team”.
2. The controller issues a Domain command for the change.
3. The Domain command passes itself to the DRb server.
4. The DRb server matches the command with a command processor that stores the event in the event domain.
5. The command processor publishes the event to the Event Server.
6. The Event Server drops a message on the RabbitMQ events exchange.
7. The application Projection server receives the message from RabbitMQ.
8. The projection server creates the domain model (a team, in this case)
9. User issues a request to see the new data, like “get teams”
10. Standard Rails controller-model-view sequence occurs to render the data to the client.

Demo App

The demo application today is a team builder, as previously described. The app allows the creation of teams and the addition of players to those teams. I’m using Ruby 2.1.2, Rails 4.1.6, and Disco 0.5.3.

Install the rails-disco gem (gem install rails-disco) and type disco new team-builder to get the app setup. The output will look very similar to what rails new spits out. The main differences are some extra folders (domain, app/commands, app/projections), a new executable (disco), a new controller (event_source_controller), along with some other minor additions.

Change into the team-builder app directory. Before we start making sweet disco, let’s make some changes. First, I am going to use PostgreSQL, so add the ‘pg’ gem, bundle, and change the database configuration to use it.

Rails Disco has its own configuration file. This file holds the information for the domain event data store, the DRb server, and the RabbitMQ connection and exchange.

Notice that I am using different database names for our domain (events) and our projection (standard Rails), so I will have distinct databases for the domain events and the projections. This simulates what a “real” install would look like a bit more accurately.

With all the configuration in place, it’s time to use the disco command to create our scaffold. Rails disco provides a command that mirrors the rails scaffold command:

disco g scaffold team name:string

This results in a fair bit of output, so I’ll just highlight the items that Disco adds.

invoke  model
    invoke    projection
    create      app/projections/team_projection.rb
    create      test/projections/team_projection_test.rb
    ....
    invoke  command
    create    app/commands/create_team_command.rb
    create    app/events/created_team_event.rb
    invoke    command_processor
    create      domain/command_processors/domain/team_processor.rb
    insert      domain/command_processors/domain/team_processor.rb
    insert    app/projections/team_projection.rb
    create    app/commands/update_team_command.rb
    create    app/events/updated_team_event.rb
    invoke    command_processor
      skip      domain/command_processors/domain/team_processor.rb
    insert      domain/command_processors/domain/team_processor.rb
    insert    app/projections/team_projection.rb
    create    app/commands/delete_team_command.rb
    create    app/events/deleted_team_event.rb
    invoke    command_processor
      skip      domain/command_processors/domain/team_processor.rb
    insert      domain/command_processors/domain/team_processor.rb
    insert    app/projections/team_projection.rb
    ...
   prepend    app/views/teams/index.html.erb
   prepend    app/views/teams/show.html.erb

The Disco generator (man, I LOVE typing that) adds a projection, along with a command-event-processor sequence for each CRUD action. At the end, it also touches the view, prepending and event_source helper.

The standard controller actions are all there, but they don’t call the usual ActiveRecord methods.

Controller Changes

Remember from our flow above, the controllers run a Domain command. The command is validated, just like a model might be, and then sent to the DRb server for processing.

class TeamsController < ApplicationController
  include EventSource

  def index
    @teams = Team.all
  end

  def show
    @team = Team.find(id_param)
  end

  def new
    @team = CreateTeamCommand.new
  end

  def edit
    @team = UpdateTeamCommand.new Team.find(id_param).attributes
  end

  def create
    @team = CreateTeamCommand.new team_params
    if store_event_id Domain.run_command(@team)
      redirect_to @team, notice: 'Team was successfully created.'
    else
      render action: 'new'
    end
  end

  def update
    @team = UpdateTeamCommand.new team_params.merge(id: id_param)
    if store_event_id Domain.run_command(@team)
      redirect_to @team, notice: 'Team was successfully updated.'
    else
      render action: 'edit'
    end
  end

  def destroy
    delete_team = DeleteTeamCommand.new(id: id_param)
    if store_event_id Domain.run_command(delete_team)
      redirect_to teams_url, notice: 'Team was successfully destroyed.'
    else
      redirect_to team_url(id: id_param), alert: 'Team could not be deleted.'
    end
  end

  private

  def team_params
    params.require(:team).permit(:name)
  end

  def id_param
    params.require(:id).to_i
  end
end

The first noticable change is the inclusion of an EventSource module, which comes from the app/concerns directory and just adds some methods to store the event ID in the session. We are creating models, so we can’t redirect to a new model when it’s created. Rails Disco does some nifty things using the event id to get the eventual model ID.

The find and show actions are unchanged. The command actions (create, update, destroy) are where the funk gets funky. Looking at the create action, it creates a CreateTeamCommand and passes it to Domain. What’s the Command do?

Commands

The commands are pretty simple. A command is responsible for validation and serving data to forms.

class CreateTeamCommand
  include ActiveModel::Model
  include ActiveEvent::Command
  form_name 'Team'
  attributes :name

  validates :name, presence: true
end

As you can see, the command simple holds the attributes and validations for the object. Standard ActiveModel validation methods can be used, which is nice.

If a command is valid, it is passed to the Domain via DRb and ends up being processed by an ActiveDomain::CommandProcessor.

Command Processors

Command processors are responsible for creating our domain events from the commands. The command processors are found in domain/command_processors/domain. In there, you’ll find team_processor, which looks like:

module Domain
  class TeamProcessor
    include ActiveDomain::CommandProcessor
    process DeleteTeamCommand do |command|
      if command.valid?
        event DeletedTeamEvent.new command.to_hash
      end
    end

    process UpdateTeamCommand do |command|
      if command.valid?
        event UpdatedTeamEvent.new command.to_hash
      end
    end

    process CreateTeamCommand do |command|
      if command.valid?
        id = ActiveDomain::UniqueCommandIdRepository.new_for command.class.name
        event CreatedTeamEvent.new command.to_hash.merge(id: id)
      end
    end

  end
end

Very straightforward, the command processor creates the matching event for the command. These events will be stored in our domain_events table and published to the RabbitMQ exchange.

Projections

The projection server, which is started with the Rails application, is listening for events on the RabbitMQ events exchange. For each event, the projection server will loop through the registered projections and invoke the event. It’s worth noting that not all projections will perform changes for all events. You might have several projections in the app, each handling a subset of the events.

The app/projections/team_projection.rb file looks like:

class TeamProjection
  include ActiveProjection::ProjectionType

  def deleted_team_event(event)
    Team.find(event.id).destroy!
  end

  def updated_team_event(event)
    Team.find(event.id).update! event.values
  end

  def created_team_event(event)
    Team.create! event.to_hash
  end
end

If you remember, the projection handles the event, transforming it into something our app can read. For a Rails app, that is simply our ActiveRecord calls.

That’s basically it. If you fire up the add and go to /teams/new you can add a new team and watch the logs. You’ll see messages like:

2014-10-20 20:27:15 -0400: [Domain Server][DEBUG]: Published CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-20 20:27:15 -0400: [Projection Server][DEBUG]: Received CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-20 20:27:15 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedTeamEvent[1]

Players Gonna Play

The application we have built thus far is pretty vanilla. I want you to focus more on the concepts than the implemenation, so I am not making it too complex. However, I’d like to quickly run through the steps to add the Player Projection so you can see how associations can be handled and what happens when you have multiple projections.

Players have a name, position, number, and must be on a team:

disco g scaffold player name:string position:string number:integer references:team

You already know what that is going to do, but we need to tweak some things to make sure a player is always in the context of a team.

First, change the config/routes.rb by moving resources :players so that it’s under teams:

resources :teams do
  resources :players
end

On to the PlayersController, we need to grab the team before each action is run.

class PlayersController < ApplicationController
  include EventSource
  before_action team

  ...existing code...

  def create
    # Change the command to include the team_id
    @player = CreatePlayerCommand.new player_params.merge(team_param)
    ...the rest is the same...
  end

  ...existing code...

  def team_param
    params.require(:team_id).to_i
  end

  def team
    @team = Team.find(params[:team_id])
  end
end

We added the before_action and changed the create method to pass in the team id to the CreatePlayerCommand.

What about that command? It looks the same, except it has a references attribute from the generator. I am going to change that to team_id:

class CreatePlayerCommand
  include ActiveModel::Model
  include ActiveEvent::Command
  form_name 'Player'
  attributes :name, :position, :number, :team_id
end

Do the same thing for CreatePlayerEvent:

class CreatedPlayerEvent
  include ActiveEvent::EventType
  attributes :id, :name, :position, :number, :team_id
  def values
    attributes_except :id
  end
end

The last Rails Disco related change is on the PlayerProjection. Since players are added to a team, the team has to be found before we can create the player:

class PlayerProjection
  include ActiveProjection::ProjectionType

  def deleted_player_event(event)
    Player.find(event.id).destroy!
  end

  def updated_player_event(event)
    Player.find(event.id).update! event.values
  end

  def created_player_event(event)
    attrs = event.to_hash
    team_id = attrs.delete(:team_id)
    return if team_id.nil? # Or raise
    team = Team.find(team_id)
    return if team.nil? # Or raise
    team.players.create! event.to_hash
  end
end

We’ve added a model that belongs to another model, so there are the standard Rails & ActiveRecord changes to be made:

app/models/team.rb

class Team < ActiveRecord::Base
  self.table_name = 'teams'
  has_many :players
end

app/models/players.rb

class Player < ActiveRecord::Base
  self.table_name = 'players'
  belongs_to :team
end

The player views (in app/views/players) need to be changed to reflect the team. Basically, anywhere you see players_path change it to team_players_path(@team). You probably want to add a ‘New Player’ link to the teams/show.html.erb. These changes are just Rails stuff, and I’ll presume you can work out the rest. If not, look at the Github repository to see what I did.

You can see this all in action if you go to a team (you added one already right?) New Player page (/teams/1/player/new, assuming the team ID is 1), then fill out and submit the form. Here’s what I see in the logs:

Started POST "/teams/1/players" for 127.0.0.1 at 2014-10-27 19:44:10 -0400
Processing by PlayersController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"kA23feXUmm6P3paIRO8IAeBt/YAmiQrmhjZZKaiwOhU=", "player"=>{"name"=>"Troy Aikman", "position"=>"QB", "number"=>"8", "team_id"=>"1"}, "commit"=>"Create Player", "team_id"=>"2"}
  Team Load (0.3ms)  SELECT  "teams".* FROM "teams"  WHERE "teams"."id" = $1 LIMIT 1  [["id", 1]]
2014-10-27 19:44:10 -0400: [Domain Server][DEBUG]: Published CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
2014-10-27 19:44:10 -0400: [Projection Server][DEBUG]: Received CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
Redirected to http://localhost:3000/teams/1/players
Completed 302 Found in 54ms (ActiveRecord: 0.3ms)


Started GET "/teams/1/players" for 127.0.0.1 at 2014-10-27 19:44:10 -0400
Processing by PlayersController#index as HTML
  Parameters: {"team_id"=>"2"}
  Team Load (0.3ms)  SELECT  "teams".* FROM "teams"  WHERE "teams"."id" = $1 LIMIT 1  [["id", 2]]
  Player Load (0.2ms)  SELECT "players".* FROM "players"
  Rendered players/index.html.erb within layouts/application (1.5ms)
Completed 200 OK in 26ms (Views: 24.9ms | ActiveRecord: 0.5ms)
2014-10-27 19:44:10 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedPlayerEvent[4]
2014-10-27 19:44:10 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedPlayerEvent[4]

You can see the event getting published, received, and processed. Notice that both the TeamProjection and the PlayerProjection get the event, but only the PlayerProjection does anything with it.

Domain Events

Now that we have a couple of teams and players, let’s take a look at our Domain data. Here are the entries in the domain_events table:

ID	Event	Data
1	CreatedTeamEvent	{“name”:”Cowboys”,”id”:1}
2	CreatedTeamEvent	{“name”:”Redskins”,”id”:2}
3	CreatedPlayerEvent

Back in our Rails data, there are 2 entries in the projections table:

ID	class_name	last_id	solid
1	TeamProjection	3	true
2	PlayerProjection	3	true

As you can see, each projection is responsible for managing the events that it has processed.

Rails Disco has a couple of other tables, but the data above is the most important for the basic concept.

Replaying the Projection

One of the main points of Event Sourcing is to enable the ability to play/replay projections as new requirements and circumstances arise. We can see this with a quick-and-dirty example in our demo.

Fire up a Rails console and delete all teams and players (Team.delete_all, Player.delete_all). At this point, we have to tell the projection to replay all events by resetting it’s last_id value to the id of the start event. So, if your first event in the domain_events table is 1, then open up a SQL Prompt (I use pgAdmin3, FWIW) and type:

update projections set last_id=0

A quicker way to do this is to simply rake db:drop and rake db:migrate.

Now, when you restart the server, you see all the events being replayed in the logs:

2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: received resend request with id 1
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: Republished CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: Republished CreatedTeamEvent with {"name":"Redskins","id":2}
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: Republished CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: Received CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedTeamEvent[1]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedTeamEvent[1]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: Received CreatedTeamEvent with {"name":"Redskins","id":2}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedTeamEvent[2]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedTeamEvent[2]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: Received CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedPlayerEvent[3]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedPlayerEvent[3]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: All replayed events received

And just like Disco, you’re data is BACK! (OK, maybe Disco isn’t back, but I can dream, right?)

The source for all this funky disco magic is on Github

CompletedPostOnRailsDiscoEvent

As is often the case with these kinds of posts, this example is a bit contrived. I had Andreas Reischuck from Hicknhack Software review this post and he added the following:

One minor thing I would lik to add. The CRUD parts from the generators will only show how Rails Disco works. The shiny parts comes into play, if you start to add semantic domain events. For your demo I can think of [events like] PlayerChangedTeam (with the player, teams, and money involved), PlayerBecomesCoach, and other domain specific events.

Andreas raises a good point. My demo was very CRUDish, but most domains have events where domain logic is more complicated. Event Sourcing is a godsend is such cases.

There’s your whirlwind tour of Event Sourcing and Rails Disco. There’s more to what Rails Disco does, but I didn’t want to get too far out. I really think Event Sourcing is a concept you should have in your 17-inch afro of skills. Rails Disco really brings the funk and the beat of Event Sourcing to Ruby, so get on down and check it out. I gotta split, you cool cats!

Struts off to “You Should Be Dancing Sourcing, Yeah”

I want to sincerely thank Andreas at HicknHack Software for reviewing the article.