Rails Disco: Get Down with Event Sourcing
Aside: I feel like you should play “You Should Be Dancing” by the Bee Gees right now. Once it’s going in the background, read on…
Here are some questions for you:
- What did your data look like last year? Last week?
- How many different client applications are using your single data store? How much data transformation are they doing to use the data?
- Have you ever changed your model, resulting in a tedious data migration to add the new attributes to the existing data?
- Have you ever wanted to change technologies around your data (data store, for example), only to say “No” because the migration was too hard?
I recently had to answer these questions, and my answers were terrible. So, I asked myself another question: What can I do to have better answers to these (and other) questions in the future?
After some research, the answer that popped up was “Event Sourcing”.
Event Sourcing 101
What is Event Sourcing? The short answer is, storing the events that occurred to get our data/application to it’s current state. Here is what Martin Fowler has to say:
The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself.
That linked article is worth the read. The “big brain” in the Event Sourcing space is Greg Young and he has several fantastic resources around Event Sourcing, including:
- Presentation: “CQRS and Event Sourcing” (Code on the Beach)
- Video: “Polyglot Data”
- Article: “Why Use Event Sourcing?”
- Software: EventStore is probably the biggest piece of software that is dedicated to Event Sourcing. Unfortunately, there’s no Ruby wrapper for it, just Java and C#. Regardless, its docs are definitely worth a read.
A quick Google of “Greg Young Event Sourcing” will bring even more resources.
Mr. Young’s presentation above mentions CQRS, which stands for Command Query Responsibilty Segregation. The fundamental concept behind CQRS is separating the things that change your data (commands) from the things that ask for your data (queries). The separation can be different classes or entirely different applications. Again, we’ll turn to Mr. Fowler:
[CQRS] At its heart is a simple notion that you can use a different model to update information than the model you use to read information. By separate models we most commonly mean different object models, probably running in different logical processes, perhaps on separate hardware.
Event Sourcing and CQRS are peas and carrots, chocolate and peanut butter. I strongly recommend that you read up on these two concepts a bit before continuing this article. It’ll make rails-disco easier to digest.
I’ll run through a quick scenario to make things a bit clearer and introduce a couple of concepts. Imagine an application that builds sports teams with players. The domain consists of a Team model and a Player model.
Using Event Sourcing, when the user creates a team, the result is a CreatedTeamEvent
with the team data. When a player is added to a team, a CreatedPlayerEvent
is created. The event store looks like this:
CreatedTeamEvent {"name": "Cowboys"}
CreatedTeamEvent {"name": "Redskins"}
CreatedPlayerEvent {"name": "Troy Aikman", "number": "8", "position": "QB", "team": "Cowboys"}
DeletedTeamEvent {"team": "Redskins", "reason": "Because they are awful"}
The question is, what does the application show to the user when all teams are requested? It could replay the events everytime teams are requested. However, it’s not scalable. One of the tenets of Event Sourcing is NEVER deleting data. Ever. Let that sink in. You can’t delete events, because the events MUST be replayable to ensure you end up with the correct application state. So, replaying events for each query might work for a bit, but as the number of events grows, the query becomes a bottleneck. What do you do?
In Event Sourcing speak, the answer is: Projections. A “projection” is a rolling snapshot of the data. In essense, a projection consumes the events and writes the data to the specification of a particular client. For a Rails app, we might create a Team
model and create a team for each CreatedTeamEvent
in the event store. Our projection, basically, subscribes to the events, like an RSS/Atom feed. It’s important to understand that the projection is responsible for managing its own subscription, again like an RSS/Atom client.
Projections can be added to answer many common questions/scenarios:
- Want to switch to a new data store for your Rails app? Create a projection that consumes the events and writes the data to the new store. Then, start from event 0 and go through all events.
- Want a separate data store for reports that uses a more de-normalized data structure? Done.
- Want to know what players were on the team last summer? Done.
- Need to make drastic changes to your data model? Do it, change the projection to handle the new model, and run the events starting from event 0.
The possibilities really are far out, outta sight, DY-NO-MITE!
Rails Disco
Now that you understand the basics of Event Sourcing, how can a bell-bottom wearing Rubyist get it on? The answer is the rails-disco gem from the funky cats at Hicknhack Software.
Rails Disco is “A distributed party with commands, events and projections”. It is made up of 3 gems:
(Image provided by Hicknhack. They boogie down in Germany.)
- Active Domain – “A framework that allows to write a domain server, that processes commands and creates events.”
- Active Event – Contains events, commands, and validations for the events that go into the event store.
- Active Projection – The projections to consume events and create the Rails database models.
Rails Disco presumes a certain infrastructure to get funky:
- A distributed Ruby (DRb) server is used to communicate commands to the domain.
- A RabbitMQ exchange is used to publish events from the domain to the projection. As such, you must have RabbitMQ installed and running.
- A streaming web server (like Puma or Thin) must be used. Event source changes are communicated to the client using streaming. Disco will add Puma to your Gemfile, by default.
Rails Disco comes with a disco
executable that allows a Rails developer to generate a “disco scaffold” (BTW, I LOVE putting “disco” in front of other words. Favorite. Gem name. Ever.) As such, it also adds some generators called by the disco
command that I’ll use it a bit to get the app setup.
High Level Flow
(Image provided by Hicknhack. Nutzer mean “user”. Disco Nutzer.)
The standard flow of data through a Rails Disco app is as follows:
- User issues a request that changes data, like “create team”.
- The controller issues a Domain command for the change.
- The Domain command passes itself to the DRb server.
- The DRb server matches the command with a command processor that stores the event in the event domain.
- The command processor publishes the event to the Event Server.
- The Event Server drops a message on the RabbitMQ events exchange.
- The application Projection server receives the message from RabbitMQ.
- The projection server creates the domain model (a team, in this case)
- User issues a request to see the new data, like “get teams”
- Standard Rails controller-model-view sequence occurs to render the data to the client.
Demo App
The demo application today is a team builder, as previously described. The app allows the creation of teams and the addition of players to those teams. I’m using Ruby 2.1.2, Rails 4.1.6, and Disco 0.5.3.
Install the rails-disco gem (gem install rails-disco
) and type disco new team-builder
to get the app setup. The output will look very similar to what rails new
spits out. The main differences are some extra folders (domain
, app/commands
, app/projections
), a new executable (disco
), a new controller (event_source_controller
), along with some other minor additions.
Change into the team-builder app directory. Before we start making sweet disco, let’s make some changes. First, I am going to use PostgreSQL, so add the ‘pg’ gem, bundle
, and change the database configuration to use it.
Rails Disco has its own configuration file. This file holds the information for the domain event data store, the DRb server, and the RabbitMQ connection and exchange.
Notice that I am using different database names for our domain (events) and our projection (standard Rails), so I will have distinct databases for the domain events and the projections. This simulates what a “real” install would look like a bit more accurately.
With all the configuration in place, it’s time to use the disco
command to create our scaffold. Rails disco provides a command that mirrors the rails scaffold
command:
disco g scaffold team name:string
This results in a fair bit of output, so I’ll just highlight the items that Disco adds.
invoke model
invoke projection
create app/projections/team_projection.rb
create test/projections/team_projection_test.rb
....
invoke command
create app/commands/create_team_command.rb
create app/events/created_team_event.rb
invoke command_processor
create domain/command_processors/domain/team_processor.rb
insert domain/command_processors/domain/team_processor.rb
insert app/projections/team_projection.rb
create app/commands/update_team_command.rb
create app/events/updated_team_event.rb
invoke command_processor
skip domain/command_processors/domain/team_processor.rb
insert domain/command_processors/domain/team_processor.rb
insert app/projections/team_projection.rb
create app/commands/delete_team_command.rb
create app/events/deleted_team_event.rb
invoke command_processor
skip domain/command_processors/domain/team_processor.rb
insert domain/command_processors/domain/team_processor.rb
insert app/projections/team_projection.rb
...
prepend app/views/teams/index.html.erb
prepend app/views/teams/show.html.erb
The Disco generator (man, I LOVE typing that) adds a projection, along with a command-event-processor sequence for each CRUD action. At the end, it also touches the view, prepending and event_source
helper.
The standard controller actions are all there, but they don’t call the usual ActiveRecord methods.
Controller Changes
Remember from our flow above, the controllers run a Domain command. The command is validated, just like a model might be, and then sent to the DRb server for processing.
class TeamsController < ApplicationController
include EventSource
def index
@teams = Team.all
end
def show
@team = Team.find(id_param)
end
def new
@team = CreateTeamCommand.new
end
def edit
@team = UpdateTeamCommand.new Team.find(id_param).attributes
end
def create
@team = CreateTeamCommand.new team_params
if store_event_id Domain.run_command(@team)
redirect_to @team, notice: 'Team was successfully created.'
else
render action: 'new'
end
end
def update
@team = UpdateTeamCommand.new team_params.merge(id: id_param)
if store_event_id Domain.run_command(@team)
redirect_to @team, notice: 'Team was successfully updated.'
else
render action: 'edit'
end
end
def destroy
delete_team = DeleteTeamCommand.new(id: id_param)
if store_event_id Domain.run_command(delete_team)
redirect_to teams_url, notice: 'Team was successfully destroyed.'
else
redirect_to team_url(id: id_param), alert: 'Team could not be deleted.'
end
end
private
def team_params
params.require(:team).permit(:name)
end
def id_param
params.require(:id).to_i
end
end
The first noticable change is the inclusion of an EventSource
module, which comes from the app/concerns directory and just adds some methods to store the event ID in the session. We are creating models, so we can’t redirect to a new model when it’s created. Rails Disco does some nifty things using the event id to get the eventual model ID.
The find
and show
actions are unchanged. The command actions (create
, update
, destroy
) are where the funk gets funky. Looking at the create
action, it creates a CreateTeamCommand
and passes it to Domain
. What’s the Command do?
Commands
The commands are pretty simple. A command is responsible for validation and serving data to forms.
class CreateTeamCommand
include ActiveModel::Model
include ActiveEvent::Command
form_name 'Team'
attributes :name
validates :name, presence: true
end
As you can see, the command simple holds the attributes and validations for the object. Standard ActiveModel validation methods can be used, which is nice.
If a command is valid, it is passed to the Domain
via DRb and ends up being processed by an ActiveDomain::CommandProcessor
.
Command Processors
Command processors are responsible for creating our domain events from the commands. The command processors are found in domain/command_processors/domain. In there, you’ll find team_processor, which looks like:
module Domain
class TeamProcessor
include ActiveDomain::CommandProcessor
process DeleteTeamCommand do |command|
if command.valid?
event DeletedTeamEvent.new command.to_hash
end
end
process UpdateTeamCommand do |command|
if command.valid?
event UpdatedTeamEvent.new command.to_hash
end
end
process CreateTeamCommand do |command|
if command.valid?
id = ActiveDomain::UniqueCommandIdRepository.new_for command.class.name
event CreatedTeamEvent.new command.to_hash.merge(id: id)
end
end
end
end
Very straightforward, the command processor creates the matching event for the command. These events will be stored in our domain_events
table and published to the RabbitMQ exchange.
Projections
The projection server, which is started with the Rails application, is listening for events on the RabbitMQ events exchange. For each event, the projection server will loop through the registered projections and invoke the event. It’s worth noting that not all projections will perform changes for all events. You might have several projections in the app, each handling a subset of the events.
The app/projections/team_projection.rb file looks like:
class TeamProjection
include ActiveProjection::ProjectionType
def deleted_team_event(event)
Team.find(event.id).destroy!
end
def updated_team_event(event)
Team.find(event.id).update! event.values
end
def created_team_event(event)
Team.create! event.to_hash
end
end
If you remember, the projection handles the event, transforming it into something our app can read. For a Rails app, that is simply our ActiveRecord calls.
That’s basically it. If you fire up the add and go to /teams/new
you can add a new team and watch the logs. You’ll see messages like:
2014-10-20 20:27:15 -0400: [Domain Server][DEBUG]: Published CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-20 20:27:15 -0400: [Projection Server][DEBUG]: Received CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-20 20:27:15 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedTeamEvent[1]
Players Gonna Play
The application we have built thus far is pretty vanilla. I want you to focus more on the concepts than the implemenation, so I am not making it too complex. However, I’d like to quickly run through the steps to add the Player Projection so you can see how associations can be handled and what happens when you have multiple projections.
Players have a name, position, number, and must be on a team:
disco g scaffold player name:string position:string number:integer references:team
You already know what that is going to do, but we need to tweak some things to make sure a player is always in the context of a team.
First, change the config/routes.rb by moving resources :players
so that it’s under teams:
resources :teams do
resources :players
end
On to the PlayersController
, we need to grab the team before each action is run.
class PlayersController < ApplicationController
include EventSource
before_action team
...existing code...
def create
# Change the command to include the team_id
@player = CreatePlayerCommand.new player_params.merge(team_param)
...the rest is the same...
end
...existing code...
def team_param
params.require(:team_id).to_i
end
def team
@team = Team.find(params[:team_id])
end
end
We added the before_action
and changed the create
method to pass in the team id to the CreatePlayerCommand
.
What about that command? It looks the same, except it has a references
attribute from the generator. I am going to change that to team_id
:
class CreatePlayerCommand
include ActiveModel::Model
include ActiveEvent::Command
form_name 'Player'
attributes :name, :position, :number, :team_id
end
Do the same thing for CreatePlayerEvent
:
class CreatedPlayerEvent
include ActiveEvent::EventType
attributes :id, :name, :position, :number, :team_id
def values
attributes_except :id
end
end
The last Rails Disco related change is on the PlayerProjection
. Since players are added to a team, the team has to be found before we can create the player:
class PlayerProjection
include ActiveProjection::ProjectionType
def deleted_player_event(event)
Player.find(event.id).destroy!
end
def updated_player_event(event)
Player.find(event.id).update! event.values
end
def created_player_event(event)
attrs = event.to_hash
team_id = attrs.delete(:team_id)
return if team_id.nil? # Or raise
team = Team.find(team_id)
return if team.nil? # Or raise
team.players.create! event.to_hash
end
end
We’ve added a model that belongs to another model, so there are the standard Rails & ActiveRecord changes to be made:
app/models/team.rb
class Team < ActiveRecord::Base
self.table_name = 'teams'
has_many :players
end
app/models/players.rb
class Player < ActiveRecord::Base
self.table_name = 'players'
belongs_to :team
end
The player views (in app/views/players) need to be changed to reflect the team. Basically, anywhere you see players_path
change it to team_players_path(@team)
. You probably want to add a ‘New Player’ link to the teams/show.html.erb
. These changes are just Rails stuff, and I’ll presume you can work out the rest. If not, look at the Github repository to see what I did.
You can see this all in action if you go to a team (you added one already right?) New Player page (/teams/1/player/new
, assuming the team ID is 1), then fill out and submit the form. Here’s what I see in the logs:
Started POST "/teams/1/players" for 127.0.0.1 at 2014-10-27 19:44:10 -0400
Processing by PlayersController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"kA23feXUmm6P3paIRO8IAeBt/YAmiQrmhjZZKaiwOhU=", "player"=>{"name"=>"Troy Aikman", "position"=>"QB", "number"=>"8", "team_id"=>"1"}, "commit"=>"Create Player", "team_id"=>"2"}
Team Load (0.3ms) SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT 1 [["id", 1]]
2014-10-27 19:44:10 -0400: [Domain Server][DEBUG]: Published CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
2014-10-27 19:44:10 -0400: [Projection Server][DEBUG]: Received CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
Redirected to http://localhost:3000/teams/1/players
Completed 302 Found in 54ms (ActiveRecord: 0.3ms)
Started GET "/teams/1/players" for 127.0.0.1 at 2014-10-27 19:44:10 -0400
Processing by PlayersController#index as HTML
Parameters: {"team_id"=>"2"}
Team Load (0.3ms) SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT 1 [["id", 2]]
Player Load (0.2ms) SELECT "players".* FROM "players"
Rendered players/index.html.erb within layouts/application (1.5ms)
Completed 200 OK in 26ms (Views: 24.9ms | ActiveRecord: 0.5ms)
2014-10-27 19:44:10 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedPlayerEvent[4]
2014-10-27 19:44:10 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedPlayerEvent[4]
You can see the event getting published, received, and processed. Notice that both the TeamProjection
and the PlayerProjection
get the event, but only the PlayerProjection
does anything with it.
Domain Events
Now that we have a couple of teams and players, let’s take a look at our Domain data. Here are the entries in the domain_events
table:
ID | Event | Data |
---|---|---|
1 | CreatedTeamEvent | {“name”:”Cowboys”,”id”:1} |
2 | CreatedTeamEvent | {“name”:”Redskins”,”id”:2} |
3 | CreatedPlayerEvent |
Back in our Rails data, there are 2 entries in the projections
table:
ID | class_name | last_id | solid |
---|---|---|---|
1 | TeamProjection | 3 | true |
2 | PlayerProjection | 3 | true |
As you can see, each projection is responsible for managing the events that it has processed.
Rails Disco has a couple of other tables, but the data above is the most important for the basic concept.
Replaying the Projection
One of the main points of Event Sourcing is to enable the ability to play/replay projections as new requirements and circumstances arise. We can see this with a quick-and-dirty example in our demo.
Fire up a Rails console and delete all teams and players (Team.delete_all
, Player.delete_all
). At this point, we have to tell the projection to replay all events by resetting it’s last_id
value to the id of the start event. So, if your first event in the domain_events
table is 1, then open up a SQL Prompt (I use pgAdmin3, FWIW) and type:
update projections set last_id=0
A quicker way to do this is to simply
rake db:drop
andrake db:migrate
.
Now, when you restart the server, you see all the events being replayed in the logs:
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: received resend request with id 1
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: Republished CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: Republished CreatedTeamEvent with {"name":"Redskins","id":2}
2014-10-28 05:18:29 -0400: [Domain Server][DEBUG]: Republished CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: Received CreatedTeamEvent with {"name":"Cowboys","id":1}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedTeamEvent[1]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedTeamEvent[1]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: Received CreatedTeamEvent with {"name":"Redskins","id":2}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedTeamEvent[2]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedTeamEvent[2]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: Received CreatedPlayerEvent with {"name":"Troy Aikman","position":"QB","number":"8","team_id":1,"id":1}
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [PlayerProjection]: successfully processed CreatedPlayerEvent[3]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: [TeamProjection]: successfully processed CreatedPlayerEvent[3]
2014-10-28 05:18:29 -0400: [Projection Server][DEBUG]: All replayed events received
And just like Disco, you’re data is BACK! (OK, maybe Disco isn’t back, but I can dream, right?)
The source for all this funky disco magic is on Github
CompletedPostOnRailsDiscoEvent
As is often the case with these kinds of posts, this example is a bit contrived. I had Andreas Reischuck from Hicknhack Software review this post and he added the following:
One minor thing I would lik to add. The CRUD parts from the generators will only show how Rails Disco works. The shiny parts comes into play, if you start to add semantic domain events. For your demo I can think of [events like] PlayerChangedTeam (with the player, teams, and money involved), PlayerBecomesCoach, and other domain specific events.
Andreas raises a good point. My demo was very CRUDish, but most domains have events where domain logic is more complicated. Event Sourcing is a godsend is such cases.
There’s your whirlwind tour of Event Sourcing and Rails Disco. There’s more to what Rails Disco does, but I didn’t want to get too far out. I really think Event Sourcing is a concept you should have in your 17-inch afro of skills. Rails Disco really brings the funk and the beat of Event Sourcing to Ruby, so get on down and check it out. I gotta split, you cool cats!
Struts off to “You Should Be Dancing Sourcing, Yeah”
I want to sincerely thank Andreas at HicknHack Software for reviewing the article.