Did Rails Sink Twitter?

Twitter is arguably the most heavily used Ruby on Rails application in the world. Almost since its inception, Twitter has fostered a wildly passionate cult following. Also from the beginning, Twitter has suffered from chronic outages under that load.

In the past month, record downtime has prompted fresh outcry within its ever-growing user base. This, along with increased attention from mainstream media has forced Twitter to publicly acknowledge these issues, and to reveal a few details about the source of these problems.

Though there has been much speculation about the source of Twitter’s performance issues, the closest we have had to a real peek under the hood has been this Twitter Developer Blog post:

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practices that are more appropriate to a content management system.

But wait, what exactly do they mean by “technologies and practices”? Could they mean Ruby on Rails?

Twitter and the Curse of the Framework

Ruby on Rails is a Model-View-Controller (MVC) framework. To build an application in Rails, you start by defining a collection of objects that model the things your application will do, then you write controllers that juggle your model in response to user requests, and finally you write the views that present your application to its users.

As separate as these three elements are, the performance of an app can depend largely on how they work together. If your model isn’t optimized for the way your controllers will use it, you could find yourself facing some surprising performance issues. This is the kind of problem that I expect Twitter is faced with.

Let’s look at a simplified example of what Twitter’s problem might look like on the inside.

Twitter is a site where users post short messages called Tweets, and follow the Tweets of other users. When the original developers behind the site first sat down to define their model, the simplest thing to do would have been to have a database table for all the Tweets, another table for all the users, and a third table to keep track of who is following whom.

This approach closely resembles the model you might build for a content management system (CMS) like a blog: each post has an author, so you can easily get a list of all the posts (i.e. Twitter’s public timeline), or just the posts in a given category (e.g. my tweets).

What this fails to take into account is the most common type of request that Twitter receives: show me the most recent tweets from just the users that I’m following. With a CMS-style model, this common request will generate a lot of work for the application, as it has to filter the timeline of Tweets for each individual user.

Twitter as a CMS

If the original developers of Twitter had realised just how many requests of this type it would be fielding, Twitter’s model would probably have been designed very differently. As mentioned in the blog post quoted above, Twitter really is a messaging system, and its model should reflect that.

Here’s how the model of a messaging system might look: whenever a user posts a Tweet, the model looks up which users are following the author and places a copy of the Tweet into those users’ private timelines. Tweets are queued up for their recipients when they’re posted, shifting the processing burden away from the more common “show me my tweets!” request.

Twitter as a messaging system

Now, let me be clear: both models could be built with Ruby on Rails. The CMS-style model is just the direction that the framework will lead you in if you don’t stop and think about the characteristics of your application. Any good framework will handle for you the things that most web applications have in common, so you can focus on the unique aspects of your app. The curse of the framework is that it can be very tempting to let the framework take the lead in handling those unique aspects too.

The fact is, Twitter’s developers were in a hurry, and took every opportunity to let the framework do the thinking for them. Just about every general-purpose web framework out there would have led them down the same road. “The fault, dear Brutus, is not in our stars, but in ourselves.” The message here is not that Rails is a bad framework—just that no framework can do the important thinking about how an app should work. That’s what developers are for.

Like any framework, Rails has strengths and weaknesses. Those who love Rails believe that its strengths—the virtual elimination of the scutwork of building web apps using a fresh, expressive programming language—far outweigh weaknesses like scaling being a non-trivial undertaking. Truth is, if you’re building a service intended for mass adoption and you don’t design it to support that, no framework in the world is going to keep your app from collapsing in on itself.

SitePoint’s new book, Simply Rails 2 is now available. Get it, and build your own Twitter killer. :-)

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • http://www.mikehealy.com.au cranial-bore

    Being a control freak I’ve often not liked the concept of convenience methods that “automatically” do stuff with your database. I’ve been suspicious of their ability to write the best performing queries when the framework developers are only dealing with abstracts and not the concrete for a given app.
    There’s always a trade off for convenience.

  • http://panesofglass.org/ aranwe

    @craial-bore: I don’t know that database abstraction is all that bad, so long as it’s designed well. (Though I love SQL and don’t know why so many people hate on it.)

    @Kevin: Great post, and great assessment on what might be going on at Twitter. On your point about where Rails “leads” a developer in terms of architecture, I don’t know that such a statement is fair. I think the mountains of books and projects centering around content management lead a developer to that design, not the framework. As you mentioned, the messaging-centric model could have been built just as easily. I doubt many of us would have thought about it, either, especially if we weren’t sure of exactly how the system would be used.

    The real test for Rails will be an application, appropriately designed, that can scale correctly. Will Rails ever get the chance again is another question. I imagine anyone wanting to build something large will go with Python, PHP, .NET, Java, or one of the other tried-and-true platforms. Why risk it?

  • php_penguin

    I don’t usually read the posts categorised as RoR, but as I use CodeIgniter (A _brilliant_ php mvc framework) this was quite useful.

    The core of the post was useful for any MVC developers, not just RoR. Thanks for that!

  • phpimpact

    No, it wasn’t Rails, it was Ruby and the system architect that design Twitter. Rails is just a framework, and can be use in the frontend without any problems. But in the backend, you don’t use Ruby. And there’s where the system architect got it wrong. They tried to do with Ruby what everyone else is doing with Java. eBay and Linkedin, for example, are much more complex sites, and they are both using Java in the backend, and they both scale.

    Was is Rails in the frontend? No. Was it Ruby in the backend? Yes.

  • phpimpact

    Kev, where it says “Twitter was built with technologies”, it’s not referring to Rails or the framework, it’s referring to Ruby. But hey, you need to sell some books right :)

  • Cunac

    When you say copy of message I hope you mean just pointer (key in db) to the message otherwise you are creating a huge I/O problem for your database and here come another outage :-)

    Are they hiring help ? ;-)

  • Cunac

    just checked on the site, job section and nothing there :-( , ah well

  • Chris

    I think there is a fundamental problem with frameworks that you touch on Kevin. Rails as a framework ignores some tried and true optimization and powerful abilities of properly indexing. I’m an RoR developer (PHP and .NET also), and I love not writing plumbing. However, I have foresaken some of the easier db access methods to use optimized views with criteria in the proper order to use indexes.

    The real problem I see with frameworks is that they foster the new found belief that SQL is dirty and should be avoided whenever possible. As far as Java as a backend, I’ve seen enough Java apps brought to their knees by poor design. Pick your language/framework, but there is no replacement for understanding how you can wrestle performance out of a database and application.

  • Leon

    Even the second architecture doesn’t work too well for Twitter. When the top user, Robert Scoble, writes a tweet, it places the tweet into all of his 25,000 followers’ private timelines? It seems this architecture solves the processing power on the app server, but not on the db backend side.

    The problem is really the system’s extreme usage of broadcasting.

  • http://aldoblog.com/ Alderete

    @ phpimpact: You can write a highly scalable web application in Rails/Ruby. You can write a crappy, unscalable web application in Java. It’s not about the language, it’s about the developers and the architecture.

    For example, at RailsConf this year, the lead developer for YellowPages.com described how he and a team of four other developers rewrote their Java-based version of the site into Rails in three months (after more than a year of research and selection of platform, etc.). The new site uses approximately the same number of servers but handles twice the traffic that the old Java version did. (48 million requests per day.) At launch about a year ago, even before tuning, the Rails version exceeded every performance requirement they had. (The only one I remember was a sub-4-second page load time.) They did this by being smart, by architecting a series of services instead of one giant intertwined application, and by making sure that things that were slow did not block things that were fast.

    Ruby, as a language, is currently not particularly fast. Certainly, in micro-benchmarks, you can “prove” that Java is faster. (And for most basic operations, it is.) But micro-benchmarks just tell you how fast small things are. Architecture and smart algorithms in your own application can mitigate or trump a slow interpreter, every time.

    And be careful of making definitive statements about the performance of the official Ruby interpreter. It’s slow-ish today. It won’t be next year. Arguments and opinions that depend on the status quo will be entirely moot.

  • Anonymous

    in my previous experience, (about 9 months ago) with a high volume app, we found our testing discovered slowness due almost entirely to database architecture, not choice of server-side programming language…. this speaks a bit to the architecture idea above, but not completely… The whole Java vs Ruby vs Whatever debate seems to be like people racing cars with great engines but hopelessly bad transmissions arguing about the engines….

  • phpimpact

    @Alderete

    In 2005 I told a friend to learn Ruby, I told him that it was fun, that it was a nice language to learn. He bought all the books and learnt the language. Today, he can’t get a job, he is unemployed. So, what should I tell him the next time I see him? To wait for next year? To buy more books? To have faith? It’s been 3 years! If I can go back in time, I would definitely tell him to learn Java, there are plenty of jobs, and the syntax is similar to C++, C#, Javascript and PHP, in case he wants to expand his toolbox or change jobs.

  • Jeff Sonstein

    Nicely written article, my thanks to the author.

    IMHO Marshall McLuhan was right, and about coding too. The work-tools I use are expressive media, and they can and do have an effect upon the structure (and thus functionality) of what I build.

    Just my $0.02-worth.

    jeffs

  • rubyjobs

    @phpimpact

    Not sure what to tell you about your poor friend who only knows one language and is now unemployed because he knows ruby. Perhaps it’s location? His ability to write a resume? Doesn’t know the right people?

    I am not sure what the problem is but I am still a student in college and I learned ruby. I am constantly being offered new jobs and opportunities to use my ruby on rails knowledge to the point of having to turn things down.

    I guess the experience is different for everyone. Where I live (Colorado), everyone of the students here who knows any language competently has been able to find a job and I see tons of job postings online for ruby jobs in nearly any location.

  • Ryan Walker

    @phpimpact

    Many, many people are making a living doing Ruby on Rails now — don’t blame Ruby for your buddy’s unemployment. And if PHP is better, why the hell would you recommend Ruby to him? Blame you, blame him, but it’s ridiculous to blame Ruby!

  • SQL guy

    What this fails to take into account is the most common type of request that Twitter receives: show me the most recent tweets from just the users that I’m following. With a CMS-style model, this common request will generate a lot of work for the application, as it has to filter the timeline of Tweets for each individual user.

    How will this generate more work? Pretty much every application known to man works this way. What’s wrong with:

    SELECT * FROM tweets INNER JOIN following ON tweets.posterID = following.buddyID WHERE following.userid = 'xxxx'

    Or am I missing something?

  • madpilot

    The crux of the issue is that Twitter shouldn’t have been built as a traditional web site – the data model is the weak point. They could have built it in PHP, Java, .NET or assembly and they would come across the same issues.

    Most websites have few writes and many reads. Twitter have MANY of both, so optimising for one or the other isn’t going to help, and when you have people like @scoble who have ~20k followers and followees, every time he hits the database it is going to get pounded. And don’t forget that no two twitter pages are the same data, so extensive caching is out the window.

    At the end of the day, Ruby on Rails is a WEBSITE framework. A website framework that does one thing REALLY well. You wouldn’t buy a framework for a kit home and try to build a skyscraper would you?

    So I don’t know why everyone is so suprised when it doesn’t perform well in a configuration that it wasn’t designed for.

  • Jake

    @phpimpact

    There may not be as many Ruby jobs available as there is for Java, but I continue to see demand for Ruby developers where I live.

    However, if your friend really knows Ruby, then Java, C# or PHP should be very easy for him to pick up because Ruby is a more sophisticated language. Ruby has more complicated features like blocks, modules, open classes, etc., but both Ruby and Java have classes and objects and much of the OO principles are the same.

    The syntax is more verbose in Java and PHP, but Ruby syntax isn’t that different from languages with C based syntaxes, especially in comparison with languages like Lisp.

    Having gone from Java and PHP to Ruby, I think that it would have been much easier to have started with Ruby and then learned Java, C# or PHP if I had to. So if he can’t get a Ruby job where he is, I think your friend is in a prime position to quickly pick up another language. You did the right thing recommending Ruby.

  • rmassart

    @maspilot.

    “At the end of the day, Ruby on Rails is a WEBSITE framework. ”

    If that’s the case then why does the RoR homepage say this:

    Rails is a full-stack framework for developing database-backed web applications according to the Model-View-Control pattern.

    You don’t need RoR to build websites, there are lots of frameworks out there which will let you do that just as easily. RoR comes into its own in developing new web apps from the ground up in a short time.

    As for Twitter. They’ve had performance issues for months (years?) now and they have obviously not been solved. Why has it taken them so long to fix this? Еither their developers are slacking or the framework is the wrong one. I doubt very much it is the former.

    If you look at their jobs section you will see they are looking for expert C++ and Java systems engineers. RoR’s most heavily used web app is looking for C++ and Java developers. I think that answers phpimpacts question as to where all the Ruby jobs are.

  • Anonymous

    Еither their developers are slacking or the framework is the wrong one. I doubt very much it is the former.

    Totally agree. But as I alluded to, you would be hard pressed to find a RDMS-based solution that will be suitable. Drop the Model part of MVC and replace it with a messaging system, and you would be using the right tool for the job.

    If you look at their jobs section you will see they are looking for expert C++ and Java systems engineers. RoR’s most heavily used web app is looking for C++ and Java developers. I think that answers phpimpacts question as to where all the Ruby jobs are.

    I do, however totally disagree with this statement. It doesn’t answer anything. There is little need for C++ programmers in the web industry, in the same way that there is little need for PHP programmers in game industry. They aren’t very likely to completely replace Rails with a C++ application are they? They will get the C++/Java guys to write a more suitable data layer, and I would be pretty surprised if they completely got rid of Rails for the interface.

    In regards to “lack of Rails jobs” there is probably one a week begin posted on the Rails Oceania mailing list, and I’m currently working on three rails jobs, so they exist. But @phpimpacts point is looking at the problem the wrong way – you rarely have clients come to you and say “I want my site written in x”. It’s up to the developer to decided what the job will be written in. The reason there are more job vacancies for PHP developers is because a majority (in Australia at least) of design houses have been working in it for years. They probably already have CMSs written in it, and are using existing code. If you were a web design company, you aren’t just going to throw away your existing code base because something new comes out.

    At the end of the day, Rails is still a young framework and Ruby is still a young language (from the point of view of enterprise-level applications). But it’s getting a hell of a lot of work done to it, with projects like passenger, Rubinius, IronRuby, JRuby and others. PHP sucked in the early days too.

  • Anonymous

    @Anonymous

    > The reason there are more job vacancies for PHP developers is because a majority (in
    > Australia at least) of design houses have been working in it for years.

    That’s absurd. I’m a PHP, Python and Ruby programmer and for all our front-end development we use PHP. Honestly, do you really think we are going to believe that people use PHP because it’s an old habit? Please.

  • phpimpact

    @Anonymous

    > The reason there are more job vacancies for PHP developers is because a majority (in
    > Australia at least) of design houses have been working in it for years.

    That’s absurd. I’m a PHP, Python and Ruby programmer and for all our front-end development we use PHP. Honestly, do you really think we are going to believe that people use PHP because it’s an old habit? Please.

  • Chris S

    Good comments. The problem isn’t with the framework (nor has Twitter ever said that), it’s with their database architecture (they *have* said that). The nice thing about Rails is that it lets you take care of all your database stuff fairly easily. The bad thing about Rails is that it makes it easier to write bad database architectures. A friend of mine recently did a series of blog posts about this, you all should check it out: http://www.drewblas.com

    Nice article.

  • Anonymous

    “That’s absurd. I’m a PHP, Python and Ruby programmer and for all our front-end development we use PHP. Honestly, do you really think we are going to believe that people use PHP because it’s an old habit? Please.”

    I am not the anon user before, but if you use PHP then it is your own fault.

    PHP can do nothing that either python or ruby cant. People only are lazy and opted to use php, which is thus the only “strength” php has – the www.

    And php is such an ugly language that over the next few years, either python or ruby – or both – will leave PHP behind. If php does not change a great many things.

  • Zamous

    I think you are complete idiot to write this entire article on pure speculation of how you think twitter was developed! Twitter has some other profound performance factors, not including the fact that they were essentially running off only one database. Why don’t you take the time to learn some real issue of Rails–they are quite a few.

  • Jonas B.

    Sorry for disturbing your archiecture modelling high, but an IRC server fits this description pretty well. They handle (tens of?) thousands of channels/users and cluster pretty well (“networks”). Don’t let the fact that the HTML is missing from the messages confuse you.

    I doubt it was time to market that let to the architecture you describe above, since there is plenty of robust scalable messaging code already out there which would be a much easier start than coding yet another. I think it’s more to do with the developers being oop wank^H^H^H^Hbelievers desperately pushing square pegs through round holes.

  • Simon Willison

    What’s wrong with:

    SELECT * FROM tweets INNER JOIN following ON tweets.posterID = following.buddyID WHERE following.userid = ‘xxxx’

    It doesn’t scale. Anyone who has built a high scale application will tell you that joins on huge tables absolutely murder performance.

  • Anonymous

    Even Tim Bray which is a strong advocate for Rails and displays quite a bit of disgust at PHP and strongly recommends PHP developers to move to Rails, has widely been quoted as saying that PHP scales better:
    http://www.tbray.org/ongoing/When/200x/2006/11/10/Comparing-Frameworks

    The qualities that PHP apps lack, can be fixed with best development practices. However, the weaknesses other languages have, tend to be more intrinsic to the language.

  • Anonymously

    A lot of Rails developers are switching to Spring and Zend Framework. Twitter gave a very bad reputation to Rails, and also, Ruby doesn’t offer any carrier opportunities to young developers.

  • http://www.websaviour.com/nexus/ dasil003

    This post is almost worse than the complete idiots posting about why Rails doesn’t scale because Ruby is slow. This is a terrible insult to the Twitter developers to suggest that they would never have thought of something so basic.

    It’s also an insult to Rails to imply that Rails favors one architecture over the other. Database normalization itself is what favors the former. It’s the more efficient, robust and flexible representation of the data. Of course that’s a theoretical perspective, as soon as you have to scale then you have far more important considerations.

    The worst part is this kind of article leaves the door open for the real idiots to come out in droves. Just look at phpimpact, so sure that Ruby is a technology more appropriate for “content management systems” or that there aren’t any Ruby jobs. I hate to break it to you phpimpact, but based on your level of discourse, I’d say your friend’s problem getting a programming job has nothing to do with Ruby.

    Please, for the love of god, would everyone just go read someone who knows what they’re talking about, study it, internalize it, then realize that off-the-cuff punditry actually makes the blogosphere stupider than listening to someone who actually has a clue:

    http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi.html

  • ChewyBac

    Rails is for people incapable of simply and correctly modeling OBJECTS. If you use rails you need to come to terms with the fact that you’re a mediocre(or worse) programmer, working most likely on a macintosh, an embarrassment to yourself and the rest of the programming world, desperate to be part of something and just a all around imbecile.

  • Anirudh Vyas

    Ruby is a dynamic language isn’t it? Is there any point to be discussed than this any further?

    Scala is the only possible candidate I see ever taking place of Java; If Ruby does make it; I will be sorry for commercial software world. Java and C# are still the best languages to go with (and plus FP paradigm missing in Java will be incorporated sooner or later) so yes, features are being added which help increase stability of a language.

    Regards
    Vyas, Anirudh

  • Anirudh Vyas

    Oh and on the issue of Rails sinking Twitter, I doubt it; Its the lead or manager; they suffer from band-aid syndrome which goes like … ” ok, i got an application to do, i will live in this silo and do it on time; and well it works … so i am happy”. Things catch on fast don’t they?

    Regards
    Vyas, Anirudh