Activity Feeds Based on Redis

Today, activity feeds are an essential part of almost all web apps. SaaS based apps, social apps, and the like all need a user timeline. These activity feeds and user timelines can get easily out of hand if not managed properly, especially with a large set of users logging in regularly.
Redis, in this case can be a good choice to attain ease of use, speed of operation, scalability, and the option of expiring data if that is not required after a specific time period.

In my approach, the default database for the application is MySQL, and I am running Redis within my application separately for activity feeds. This architecture decision is debatable, because it does increase an effort of managing 2 separate data stores. However, it does provide a significant boost to the performance of the application.

Let’s quickly run throgh setting up Redis in our application. Add the redis gem to our Gemfile and run bundle.

gem 'redis', :git => 'https://github.com/redis/redis-rb.git'

bundle install

Then in our config/initializers, create a file called redis.rb and initialize the connection to the Redis server:

$redis = Redis.new(:host => 'localhost', :port => 6379)

Let’s load up the rails console and see if the redis connection works:

$ rails c
Loading development environment (Rails 4.0.0.rc1)
1.9.3-p327 :001 > $redis
 => #<Redis client v3.0.4 for redis://localhost:6379/0>
1.9.3-p327 :002 >

Works great!

Here is the context of our application: The app allows people to create and share meals with others. Now, a system timeline would provide activity about creation and updating of meals from the people whom they’re following.

We will use the redis timeline gem for creating our Redis-based timeline. Later in this article, we will deconstruct that gem’s functions. The gem uses an after_create callback to serve as events for tracking a particular activity. This also provides us with necessary ids.

The current version of gem supports Rails 3.2 (due to the dependency on ActiveSupport and ActiveModel 3.2 for callbacks.) However, I just forked it and updated the gemspec dependency to work with Rails 4.0.0.rc1 here. You can utilize my fork in your Gemfile as shown:

gem 'redis_timeline', :git => 'https://github.com/saurabhbhatia/redis-timeline.git'</p>

<p>bundle install

The redis_timeline gem requires us to first setup an actor. It’s typically wired to our User class because the actor is reponsible for an action and User performs all the actions in an app.

class User < ActiveRecord::Base
  include Timeline::Actor
  has_many :meals
end

After setting up the actor, we need to setup the class to track. In our case, we need to track the Meal class create and edit methods. Let’s see how to do that.

We first setup an include Timeline::Track method to setup tracking on the particular model. You can now define the method you want to ‘track’ and the callback you want it to track ‘on’.

class Meal < ActiveRecord::Base
 include Timeline::Track
 belongs_to :user

  track :new_meal,
    on: :create,
    actor: :user

  track :edit_meal,
    on: :update,
    actor: :user
end

Let’s load the rails console and see if this works.

rails c
Loading development environment (Rails 4.0.0.rc1)
1.9.3-p327 :001 > meal = Meal.create(:title => 'Nachos', :description => 'Cheesy & Corny' , :user_id => 1)
 => #<Meal id: 5, title: "Nachos", description: "Cheesy & Corny", created_at: "2013-06-23 09:13:30", updated_at: "2013-06-23 09:13:30", user_id: 1>

In order to make a call on timeline, we need to get the actor (in our case the user) and call timeline.

1.9.3-p327 :002 > user = User.find(1)
1.9.3-p327 :003 > user.timeline
 => [#<Timeline::Activity actor=#<Timeline::Activity class="User" display_name="#<User:0x000000049b2900>" id=1> created_at="2013-06-23T17:13:31+08:00" object=#<Timeline::Activity class="Meal" display_name="#<Meal:0x000000051b50f8>" id=5> target=nil verb="new_meal">]

The timeline object contains the class name, meal name, user object and a verb. The verb basically denotes the action being tracked.

This still has something missing. We need to push our updates to a user’s followers. Our follow mechanism is also built with Redis and looks something like this. For convenience’s sake I have used the article here to create this mechanism and defined it in my user model.

def follow!(user)
  $redis.multi do
    $redis.sadd(self.redis_key(:following), user.id)
    $redis.sadd(user.redis_key(:followers), self.id)
  end
end

def unfollow!(user)
  $redis.multi do
    $redis.srem(self.redis_key(:following), user.id)
    $redis.srem(user.redis_key(:followers), self.id)
  end
end

def followers
  user_ids = $redis.smembers(self.redis_key(:followers))
  User.where(:id => user_ids)
end

def following
  user_ids = $redis.smembers(self.redis_key(:following))
  User.where(:id => user_ids)
end

In order to send a user’s update to a follower, we would use the `followers` key and configure it as shown in the following snippet. The gem includes a param called `followers` where we can send our followers key stored in the Redis database.

track :new_post,
  on: :create,
  actor: :user,
  followers: :followers

track :edit_post,
  on: :update,
  actor: :user,
  followers: :followers

In order to display your feed, you can make a call on the `current_user` method and call the feed in your controller. (Note: This presumes you are using something like [Devise](https://github.com/plataformatec/devise) which provides a `current_user` helper.)

def index
 @feed = current_user.timeline
end

Under the Hood

The gem uses a “fan-out-on-write” kind of model for making the call to the Redis store. It implements this using lpush. The feed is stored as a list of hashes. It first creates a hash of all the items to be tracked and ids using setters. Then sends them to a list using lpush. You can check this out in more detail in the track class docs.

def redis_add(list, activity_item)
  Timeline.redis.lpush list, Timeline.encode(activity_item)
end

In order to read and write the list, it uses a JSON encoder and decoder. This is because redis treats all its objects as strings, and hashes need to be serialized as strings before they can be stored. This is defined in the helper class.

Also defined in the helper class is the method to read the list using lrange.

def get_list(options={})
  Timeline.redis.lrange options[:list_name], options[:start], options[:end]
end

Some Tricks

Other things that can be done to optimize the lists :

1. Trim the list: One advantge of using list as the datastructure is you can trim it. In case, for example, you want a generalized timeline of the system and want to display last 100 objects, you can simply trim the list to that.

   LTRIM <user timeline list key> 0 99

2. Expire the list: In case you feel that maintinging a permanent store of system timeline is too much overhead, you can expire it within a given time interval. The time is defined in seconds.

   EXPIRE <user timeline list key> 3600

Conclusion

Redis is an effective way to develop and manage activity feeds and isolate them from the system. This can help in scaling the main app as it would keep some unnecessary writes out of the main database. Now, go create those activity feeds!