Sigma: Rankings Made Right!

In case you missed it, I’ve been deepely involved with rankings in the past months. Look here, here and here for examples. (it’s in portuguese, sorry about that…)

Rankings may not seem so important or interesting, but if we take a closer look at applications around us, we can see how it is being used in many different ways.

As I said in a previous article:

In fact, the majorities of rankings implementations are wrong, and can even make your application less attractive. It’s difficult or downright impossible to reach the top positions, you may never find a real challenge against other users

This happens because the rankings implementations we use measure the history and not the skill of the users. I have presented the solution that Microsoft came up with to this problem for the XBox:

True Skill is a Bayesian ranking algorithm developed by Microsoft Research and used in the Xbox matchmaking system built to address some perceived flaws in the Elo rating system.

TrueSkill is a great solution. The problem is that it’s patented, but the concepts behind it can be used to ennhance our actual implementations. This is where Sigma comes in.

Sigma

Sigma is an implementation of the TrueSkill concepts, written in Ruby. The existing gem targets Ruby on Rails applications that use Active Record.

The thought behind sigma is that the concept of rating is split into two different variables: The user’s skill and the trust that the system has in it. This makes it possible to handle unexpected results in a fairer way and measure the real skill faster than most ranking implementations.

Example

Here is a scenario where there are 4 players that have already played many matches against each other. Imagine how hard would be for a new player get a top position. This is actually one of the tests Sigma users:

100.times {
  user_1.won (user_1.rating - user_2.rating)
  user_2.lost (user_2.rating - user_1.rating)

  user_2.won (user_2.rating - user_3.rating)
  user_3.lost (user_3.rating - user_2.rating)

  user_3.won (user_3.rating - user_4.rating)
  user_4.lost (user_4.rating - user_3.rating)
}

Once all the users have played 100 matches, we will have a consolidated ranking.
A new player appears, and will have to prove his skill to get a competitive position.

2.times {
  user_5.won(user_5.rating - user_2.rating)
  user_2.lost(user_2.rating - user_5.rating)
}

2.times {
  user_5.won(user_5.rating - user_1.rating)
  user_1.lost(user_1.rating - user_5.rating)
}

After only 4 matches against the top players, our new user reaches the top of the rankings.

expect(User.ranking.first).to eq user_5

It illustrates one of the benefits of working with rankings using this new concept.

Usage

Setup

Sigma is really simple to use, and even more simple to setup. All you have to do is add it to your Gemfile:

gem 'sigma'

Run the sigma generator and answer just two friendly questions:

$ rails g sigma
What is your resource model? (eg. user) user
What will be the scale? (default is 50) 100

It will generate some migrations, add some new attributes to your resource model, and a create a new initializer called sigma.rb.

The new attributes are:

• skill
• doubt
• wins
• losses
• draws
• expectations

Methods

Sigma also adds some methods to the model to make access to the ranking information simple. These methods are:

• rating
  • Returns the resource rating based on skill x doubt.
• position
  • Returns the position of a resource in the ranking.
• matches
  • Return an integer with the number of wins, losses and draws.
• won (difficulty)
  • Register a win for a resource and calculate it’s rating and expectations.
• lost (difficulty)
  • Register a loss for a resource and calculate it’s rating and expectations.

Screencast

The easiest way to understand sigma is to see it in action. I have created a screencast that shows, from scratch, how to setup and use it. It’s my first screencast, and it is most legible if you view it full screen and in HD. Also, I am not a native English speaker, so I did the best I could… :)

[youtube eFNW61P6hcE]

Let me know if you have any questions or comments about Sigma.