Ruby Devs – You’re too trusting. Gems could screw you.

Have you met the developers of all the Gems you’re using? Do you know their personal stance on security and whether they use strong passwords or reuse their pet’s name on every website?

Artists Rendition: Me deploying

Artists Rendition: Me deploying

I don’t know if I’m the only one, but when there are updates to my Gems, I will generally happily update them – run my tests – and be on my merry way. I raise my sword and shout ‘Deploy!’ to my many underlings (well, I type a few commands and deploy – but that sounds nowhere near as impressive).

Note: If you’re a person who carefully reviews every code change in every gem you update, and all their dependencies, then this article isn’t for you… bastard.

Are we too trusting?

I worry that the workflow for updating Gems isn’t transparent enough – especially for new developers. I worry that we’re placing our trust in a system that has one major weakness: the security of individual Gem developers. I worry that someone will gain access to a developers account and push a malicious change to one of the Gems I use. And, most importantly, I worry that I will unwittingly use this Gem and get screwed.

The problem is we’re naively trusting of other developers’ code, and it’s only getting worse. Go count the number of other developers Gems you’re using. Scared yet? Nope? Keep reading.

Proof of concept

To prove the concept, I created a new Gem called innocent. It just has a Something::Innocent#perform_action method which takes a string, then returns it… usually. This works fine in version 2.0, but somebody gained access to my laptop; or my GitHub repo; or is holding my girlfriend hostage and demanded I give up my private keys; and pushed version 3.0 which also calls the Something::Evil#do_evil method.

If you check the code below, you’ll see all this does is read your database.yml and then raise it as an exception. But all you would need to do is email that off and you’ve got a serious issue at hand – and you’re not even aware of it.

This is obviously assuming you’re running a Rails stack here, but it would be just as easy to sniff out common config files and send them.

Here’s Something::Innocent at work, being defiled by Something::Evil.

module Something
    class Innocent
    def self.perform_action(string)
      Something::Evil.do_evil
      string
    end
  end
  class Evil
    def self.do_evil
      file = File.open("./config/database.yml", "r")
      raise file.read.inspect
      # I could just as easily email this information
        # Or I could browse directories for API Keys
        # Or I could email your wife and tell her about the other girl
    end
  end
end

If you want to check it out, an example project using the innocent Gem is available at https://github.com/snikch/innocent-project, and the innocent gem itself is at https://github.com/snikch/innocent.

Commit d3cff993b62e05d7e1cc is the ‘before’ point.
Commit ba1db02e4405b4fc614b is after bundle update.

Don’t flame me

I know that we’re the ones who should be checking the code we’re using in our projects, and that of course the responsibility in the end lies with ourselves. I’m not trying to deny that, but what I am saying is that we’re human and easily commit time saving mistakes in lieu of spending the time we should on some aspects of our work. Spending hours eyeballing code updates every week / month / decade is not something we want, or should need, to spend our time on.

My workflow — a step in the right direction

The workflow we’ve adopted at Learnable isn’t about necessarily solving this issue, but it is about risk mitigation. We no longer include git Gems that aren’t in a repository we own. This means that for us to update a Gem we need to merge the owners branch into our own fork’s master branch. When we do this, we get to see the changes that are being made and have a chance to spot any funny business.

This doesn’t solve the problem, but it does go one step towards a more transparent update process, where we can see the changes being made, and by having this in place it’s difficult for anyone to skip the code review process. The majority of our Gems aren’t pulled from a git repo anyway, so this really only provides one level of protection to a minority of the Gems we use.

Is it a tedious step? Yes.
Does it make our code safer? Maybe.
Will I be happy if this saves us from an attack at some point? Hell yes.
Is it worth it? I’m not sure.

I’d love to hear from people on this, with any ideas they’ve got to do with reducing this risk.

Won’t people notice?

It’s obvious to most people that this isn’t going to be an issue for popular, well contributed, Gems. Slipping code this dastardly past a strong community of users and contributors is nigh on impossible. A Gem with 100,000 users isn’t about to slip in a malicious commit without being caught.

A Gem with a few hundred, or thousand, users that commits often but only has one core contributer would be a more likely candidate – especially if it is only a small Gem that is really only on the periphery of your project.

Who wants to code review the Gem that provides slightly faster csv parsing for that one admin report your marketing guy wanted? Or, more appropriately, who wants to eyeball the Gem that was a dependency for the csv parsing Gem the marketing guy ‘needed’!

So…

Are we too trusting? I think so.
Can we do anything about it? You tell me.

Seriously. I want to know people’s opinions regarding this, so please contribute in the comments.

Post Script.

I’d been spending a bit of time thinking about this one day, and drafted an email to my colleauges then decided it was a non issue. The very next day the attack vector I’d been pondering occured to some popular WordPress plugins.

Post Post Script.

If anyone mentions code signing I’ll kick them in the shins.

Win an Annual Membership to Learnable,

SitePoint's Learning Platform

  • http://jeffancel.com/blog/ jeff ancel

    I think any step you can take to manage the threat is a good step, so long as it doesn’t detract from the business value of doing so. If, in an enterprise business, publishing a forward facing website, then you must make sure that you take the appropriate security measures before getting attacked.

    Shouldn’t your databases for instance only be accessible from the clients permitted to access the database? Usually through VPN, or LAN. Make sure the security around your infrastructure doesn’t allow attackers to get access to your assets arbitrarily; then it matters little if they have the information, especially if they can’t get to the asset. If your allowing 0.0.0.0/32 to have access to production databases, that are directly accessible via an IP/PORT, then I’m scared that nothing else matters here, and you’re doomed.

    • http://mal.co.nz Mal Curtis

      The database issue is a good example. We’re on Amazon EC2 for many of our projects / deployments, so the database servers are open to clients in the same availability zone – same for when we’re using RDS for database management. So in this case it’s available through LAN, but the LAN is inherently insecure as any other instance can gain access to it.

      • http://www.gemfury.com Michael Rykov

        I believe you can restrict access to certain EC2 security groups within one account, or limit it by EC2 account if you have clients that needs to access your EC2 machines from their EC2 instances.

        • Buffy

          You can lock down RDS by security group and/or ip address range. Same goes for normal ec2 instances.

    • http://www.namingthingsishard.com Bradley Grzesiak

      Remove `File.open(‘./config/database.yml’, ‘r’)` from the example, replace with `File.open(File.expand_path(‘~/.ssh/id_rsa’), ‘r’)` and you’re in for a world of hurt.

      Yes, it’s possible to run the web server as a different user than the deploy user, but does everyone do that? I think not.

  • http://rawsyntax.com Eric Himmelreich

    I take a similar approach. Fork any git-based gems to my account on github, and specify an exact version of each gem I use in my Gemfile, like

    gem ‘rails’, ‘3.0.7’ # for example

    And also version the Gemfile.lock file. That way my environment is locked down until I decide to upgrade.

  • Lonny Eachus

    YOU’RE DOING IT WRONG.

    Seriously, dude, I’m not trying to be sarcastic, but I really think you have to sit down and ask yourself: “Why would I want to update my gems in the first place?” I don’t know about “trust”, but that’s a pretty fundamental departure from good programming practices.

    I am having a hard time believing you even wrote this. You are describing ways to fix a process that you should probably not be doing anyway, and which 90% of Ruby/Rails developers should never be encountering.

    When you start a new project, or out of necessity add a new gem during the project, you should document what version it is, and stick with that version. NEVER update your gems in the middle of a project, or once a project goes live. Come hell or high water, unless you absolutely have to in order to fix a bug.

    Working code is working code. Not messing with code that works well, without VERY good reason, is one of the Primary Directives of successful programming. Updating your gems regularly is a clear violation. In fact it almost sounds as though you have a policy of regularly doing just that, and now you’re inventing hoops to jump through in order to keep that bad habit from getting you into trouble! Where did that come from? Just don’t do it in the first place. Problem solved.

    If you need different gem versions for different projects then use RVM. Or freeze your gems into individual projects.

    Then when you start a new project, you can start with a new batch of gems you have freshly examined and go from there.

    • http://hemju.com Helmut Juskewycz

      @Mal

      You made a very good point with this post. I think many Ruby Devs (sometimes myself included) like to update their Gems right away and yes that can/is a problem. Not only can bugs be included, it also poses a security risk. However, this problem is universal for every third party lib in every language (Java, C#, …). It is more a problem of the source of the libs.
      Maybe Bundler made it too easy to update Gems :)

      @Loni

      You made a very good point too. It should be considered best practice to only update third party libs when necessary (security fixes, bug fixes, performance gains, …). However, if you wouldn’t have started your comment with ‘seriously dude’ and using a very aggressive writing style, you would have been much more convincing.

      But one thing for sure, if you use third party libs/Gems, you have to trust them. So even if Bundler makes it dead simple to use other code, you should always ask if a new dependency is really necessary.

  • http://sanbi.shikamaru.fr Rémy CLOUARD (shikamaru)

    @Lonny Eachus
    What if updates bring security fixes ?

    What you describe solves part of the problem, you just do code review once. But that also mean you assume noone else is doing this review.
    You have to do this code review, even if it’s a tedious process, like everyone else, but that does not mean you are stuck to a particular version of each gem. It could go even worse if you do, because an attacker could take advantage of a vulnerability that has been fixed in a later version…

    At least just watch the changelogs for the gems you’re using, but I agree, upgrades shouldn’t be done in an automatic manner.

  • Lonny Eachus

    Helmut:

    I wasn’t trying to convince anyone of anything; I was simply stating facts as I know them. Further, “You’re doing it wrong” is a common semi-humorous phrase these days. It is not intended to be taken too literally. Kind of like “I Can Haz Cheezburger.”

    Rémy:

    Maybe I did not state it as well as I could have. But a security fix could be considered fixing a bug. There are also times when you feel you simply “must have” some new feature. I understand that. I was speaking of general guidelines, not absolute truths that must be followed without exception.

  • Kevin Baribeau

    Hey,

    Java developers have been doing something like this for a long time using maven and (mostly) a product called nexus.

    Basically nexus is a proxy for maven repositories (each repository is like a source for java packaiges (ie: rubygems)).

    Maven suffers from a lot of incidental complexity and is generally not well liked. However Nexus seems like it does a pretty good job, and is probably worth learning about so that the ruby community can start from good place on this problem.

    For example, it would be nice to have a local cache of your gem sources so that you don’t have to connect to rubygems.org or github every time you rebuild your gemset. Nexus acts like a cache in this way… ruby could use a tool that does the same thing.

  • http://rubylove.info milandobrota

    Another reason to read more code.

  • http://www.itmitica.com/en/ IT Mitică

    He’s right! The author of this article is right! I’ve been reverse engineering and code reading every single damn dll since 1999 because of this!!!

    Nah, just joking ;)

    Maybe the author forgets the part where the good code is being exploited instead of intentionally turned evil…? That means patches are a must.

  • bbenezech

    The sad part is that we are talking about something that never happened so far, when we could be drinking beers or having awesome anal sex.
    Github makes it ridiculously to scan for changes as they are pulled into master branch of the gems you follow.
    Following the gems you use is indeed a good idea.
    Trusting the core team and maintainers of the gem you use is another one. A lot of them are trustworthy, notably because they have a life-long public historic of their OS activities on Github. And if you don’t, because you’re using some edgy stuff with low visibility, fork. This is not dirty.
    Sometimes life is that simple.

  • http://jkingdon2000.blogspot.com Jim Kingdon

    It hasn’t usually been a big problem (if one is confining the issue to malicious code, rather than bugs, misfeatures, or changes which your project just doesn’t need yet). But I could see tools that let people recommend particular versions of gems (sort of analogous to a linux distribution in a way), let people vote on which versions they are using (a la debian’s popularity contest, although usage might be a poor proxy for people looking at the code), or let people say what they have reviewed (as long as this wouldn’t just help attackers know what is getting ignored).

    Trying to convince people to read gems before using them strikes me as unlikely to happen, and perhaps not a good use of people’s time. But if the effort can somehow be distributed, then it becomes more tractable (in a way this already happens, but there probably is some way to strengthen it).