An Interview with Andre Arko and Terence Lee from the Bundler Team

Can any of you remember what life was like before Bundler came along? Getting a Rails application to work was often a long, tedious trial and error process: you would start up the app and hope that you already had all of the required gems installed. Then when the application didn’t work, you would have to figure out which gems were missing and install them one after the other, hoping you didn’t break other apps.

All of this became much, much easier with the advent of Bundler. Now each Ruby application could specify a list of gems it required, along with the precise version of each gem. Bundler would then allow the Ruby application to access only the specified gems, and no other gems that might be present on your machine.

This week I was lucky enough to have the chance to talk with two members of the Bundler core team: André Arko and Terence Lee. We talked about how Bundler works, how they got involved with Bundler, when Bundler 1.1 will be coming out, how the Bundler test suite works, and a few other things…

Who?

Q: Who are you guys? Tell us a little bit about yourselves.

I’m André Arko. I live in San Francisco and work for Plex, mostly creating web apps and services. Since programming all day isn’t enough, I work on open source too — I’m on the core team for Bundler, and I’m the author of the jquery-rails gem, popularized by Rails 3.1.

Hey, I’m Terence Lee. I work at Heroku, maintaining the Ruby stack. Since my interests tend towards infrastructure, I maintain Bundler and recently took over the Resque gem. When I’m not going to an awesome Heroku or Ruby event, I live in Austin, TX, the taco capital of America.

Bundler Basics

Q: What does Bundler do, and why should someone use it?

Bundler helps you manage your application’s library dependencies in a consistent, reliable and predictable manner. As you probably remember, in the bad old days of Ruby development you sort of hoped that the Readme had an up to date list of the gems that your app required – and, of course, it never did. Then you would install those gems and try to run the app to find out which gems were not in the Readme… Then after about an hour you would discover that at least one of the gems was too new of a version, and it had a different API, and buried in the app somewhere a feature was now broken.

Q: And that was way, way back in 2009, right? It wasn’t that long ago, really.

It was everything before Rails 3.0, really. Unless you went out of your way to find Bundler and start using it, you were still stuck with that even in Rails 2.3.

How did you two get involved with Bundler?

Q: How was it that you two got started working on Bundler? Was it around that time?

I’ll go first since chronologically Terence started after me. I had been working at Engine Yard at the time, and I really wanted to play with the cool, new Rails 3 beta stuff, and I said “After work I’m going to come home, sit down and write a Rails 3 app!” I made it no more than 10 min into that Rails 3 app when I found two or three bugs in Bundler that caused exceptions and fixed them. It was really trivial stuff; it just so happened that that build had a really obvious bug that I ran into.

The next day at work I hunted down Yehuda and said: “Yehuda you should apply my fixes to Bundler!” And he said: “Oh no, I’m too busy – You have commit rights on Bundler now! Go fix!” It was only a month or two before my boss at Engine Yard came to me and said: “Hey can you still hit your project software deadlines if you’re still working on Bundler? You seem to be doing really well…” I said: “Sure I can do that!” And then I had 50% of my paid Engine Yard time was working on Bundler!

Q: And what about you, Terence?

Yehuda and Carl came to the Lone Star Ruby Conference, in Austin TX, and gave a talk on Bundler, I think this was during Bundler 0.7 or maybe 0.8. And for the same reasons André was talking about before… having to do a project without it and then trying it, I said: “This has to be the greatest thing since sliced bread!” When I started working at Heroku I ended up getting in touch with Carl and Yehuda and trying to notify them of issues that were Heroku specific. Some of that stuff came from the Bamboo stack, e.g. the read only file system, and the tests they had didn’t incorporate any of that, because normally you just don’t think about those things. I ended up taking over the support of Bundler on Heroku during the 1.0 rc days – and after 1.0 came out Carl came up to me and said: “Bro you should commit to Bundler” and he gave me commit bit. After that I decided: I guess I should do something with this. The first thing I did was fix the build, because it was broken.

When will Bundler 1.1 be released?

Back in October I wrote about why Bundler 1.1 will be much faster than Bundler 1.0 and how it uses the new RubyGems.org dependency API, but since then it still hasn’t been officially released. I decided to ask André and Terence about why it was taking so long…

Q: So when is Bundler 1.1 going to be released?

That’s what everyone wants to know – as far as I’m concerned the answer is as soon as it works, and happily almost everything works. Terence and I are going to be pairing on it in person tomorrow. There’s one tricky, strange regression that only happens when you run “bundle update” when you have a gemspec declared in your gemfile and child gems in the same repository from git that are declared as child dependencies in the gemspec that you refer to in your gemfile, and then sometimes it pulls the wrong gemspec.

Q: Wow – that’s a mouthful!

That was really fun to isolate, but we have a failing test case and we just need to track down exactly where in the internals where something has started going wrong since 1.0. As far as I know, that is the last bug.

Q: It sounds like you’re really close!

I hope so, but for some bugs, for example a caching resolver bug, it took us two or three months to figure out what was wrong.

Yea, that was just amazing. It turns out that the way the Bundler dependency resolver code works uses a lot of catch and throw. And this means that some things that you think aren’t important side effects turn out to be very important side effects.

Q: That’s the flip side of working on something so important and helpful: you really need to make sure it works before you put it out there.

Speaking as the person who had to yank every other point release of Bundler that I made – gosh that was probably 10 releases – I appreciated the importance of that more and more over time. Terence has actually acted as the “voice of reason” saying: “No! Don’t take that pull request! Things could break!”

Q: So you are the voice of reason, Terence?

I’m just much more conservative about stuff. Bundler as a whole has a lot of other stuff that I think is not the core part of what Bundler does. So earlier we were talking about what Bundler actually does, and I think that at the heart of it, it is the dependency management stuff. Once André and I figure out how to do a plugin architecture, which is somewhere on our roadmap, I would like to get all these great things that people want to have in Bundler, and move them out.

How does Bundler actually work?

Q: When most people think about Bundler, they think about “bundle install” or “bundle update” and updating dependencies. But what about the load path? How does Bundler actually restrict Ruby to only load the gems in the Gemfile? Does Bundler do that? Or is that actually handled by RubyGems itself?

That is actually in Bundler, but it’s only in Bundler because RubyGems doesn’t provide a mechanism to actually do such a thing.

The point behind Bundler is that the stuff in your Gemfile should be the only things you can load. To make this work, Carl and Yehuda wrote a method called cripple_rubygems!

[gist id=”1894412″]

[ Note: it turns out cripple_rubygems was later renamed to replace_entrypoints. ]

It goes in and kind of unwinds all of the things that RubyGems does when you require it. It inspects the state of the Ruby process and says “Oh, has RubyGems patched the require method?” If it has let’s unpatch it. “Has RubyGems modified the load path?” Well if it has let’s unmodify the load path. It undoes all of the hooks that RubyGems itself has made to the load path and gem management stuff, and it installs some stub methods for the “gem” method. Then Bundler loops over the fully resolved graph and makes sure every version that we did decide on is available in your load path so you can require it.

Q: Is it true that Bundler is intimately related to RubyGems – that they are attached at the hip?

It is. From the very start Bundler has never tried to manage dependencies in the abstract. It has always been about RubyGems.

The Bundler test suite

Q: Can you describe your approach to testing? How do you test that all possible Gemfiles would work? How do you test against a service, like the new dependency API in RubyGems.org?

The overall philosophy we inherited from the decisions that Carl and Yehuda made when they were working before either of us was involved in the project. It amounted to deciding to write only integration tests. It’s really weird that we have a Ruby library that has almost only integration tests, but close to every test in the Bundler suite is an integration test. It actually sets up a Gemfile and then it runs Bundler against the Gemfile and then it checks the results. You either test what was installed or you start a new Ruby process and check to see what the load path is.

One of the things we had to look at was: how do we test that the API endpoint is actually working? [To do this,] we used Artifice. If you check under spec/support/artifice [in the Bundler source code] you can see all the end points we have for the different test cases there. I went into RubyGems.org and looked at how they built the stuff that they put into Redis and what it returns and then tried to mimic that in a Sinatra app.

Q: Oh so you had to build a fake RubyGems.org?

Yup! I had different endpoints to match different use cases like, “What happens if the API is missing?” or “What happens when we add basic authentication to the endpoints?” Things like that. Say the API is down – we still need “bundle install” to work. So let’s fall back to downloading the index, what you do in Bundler 1.0.

The future of Bundler

Q: What’s coming next with Bundler?

The most important thing after 1.1 will be trying to document Ruby versions. If you look at other dependency managers like Leiningen and NPM there’s a way to specify what version of that language you are using. In Leiningen you can specify what version of Clojure you are using and it will actually go and download it. In NPM as well you can specify: I need to use Node 0.6. I would like to see some Ruby standard to say “I’m using MRI Ruby 1.9.3” or “I’m actually using Rubinius on this project.”

Also, what we’re hoping to do moving forward is to narrow the focus of releases and actually get them out the door faster.

Who else has helped with Bundler?

Q: Who else has contributed to Bundler that you’d like to mention?

Carl and Yehuda help out from a thought/advisory perspective.

Q: Do you mean they’re on the “board of directors” of Bundler?

…exactly!

Also, Santiago Pastorino discovered a particular performance issue and very helpfully debugged and provided us with a patch for that.

And, Joel Moss did the “bundle outdated” feature.

He basically wrote most of that feature and has been really good about fixing bugs and making sure that it keeps working.

Cowboyd has helped out a lot with fixing various issues with bundle install —standalone that he’s been running into as he’s used it. And Sferik has been on of our major pull request guys on the project, helping us sift through that stuff. Sorry but I’m sure we’ve forgotten a ton of people.

What do you need help with?

Q: What do you need help with on Bundler in the future, if people want to help out?

Stuff that would be really helpful for us would be help with managing the github issues list – we’re at 260 – having people verify “yes, this is a bug” or helping to come up with reproducible steps instead of having to do a back and forth with the original person.

Thanks!

Q: Thanks for your time… and thanks for all the hard work on Bundler!

Thanks for using it!

We’ll see you around…