Gemfile Mining: A Dive into Bundler’s Gemfile

Key Takeaways

• Bundler, the package and dependency manager for Ruby applications, uses a Gemfile to list the gems that an application needs to function. This Gemfile can be customized with a variety of options, such as group, path, and github.
• Gems can be specified in the Gemfile in several ways, including specific versions, version operators, the “twiddle-whacka” operator (which specifies any version greater than the one listed but less than the next minor revision), and compound requirements.
• Bundler allows for the grouping of dependencies into different groups, often used to install different gems in a development environment than those installed in a production environment. This can be specified on the same line as the gem or by placing the gem in a group block.
• The Gemfile offers various options for specifying gems, including Platforms (which allow for installation of a gem for a specific version of Ruby), install_if (which controls the installation of gems based on a boolean evaluation), Require (which specifies the name to require the gem by), and Override Source (which can override the source for individual gems served from a private gem server).

Bundler is fantastic, which is why it has become the de facto package and dependency manager for Ruby applications. I have used npm and golang vendoring and other language dependency managers, but none of them can even hold a candle to the simplicity Bundler offers.

As I am sure you know, at the root of Bundler’s goodness is the Gemfile. The Gemfile lists the gems that our application needs to get its work done. If you’re like me, I learned how to use the Gemfile and Bundler from various Rails (and other) tutorials. I saw people typing

  gem "a_gem"

Then, running bundle install and watching as all the dependencies were installed. Those two steps are so self-explanatory that I was never really pushed to investigate much more. Not to mention, Bundler was such a godsend after years of vendoring dependencies and other hair-pullingly frustrating approaches. rant about the old days and how easy you whippersnappers have it now

After the initial blindly-follow-the-steps Bundler tutorials, I’ve picked up on other Gemfile options, such as group, path, and github, as needed. However, the other day I decided to read through the Bundler docs, specifically around the Gemfile and I was amazed at all the options I haven’t used. I learned a few things, so I figured others might too.

In today’s post, I am going mining for options in the Gemfile. There’s really nothing in here that you can’t see in the excellent Bundler documentation, but you haven’t read that, have you??? rant about how this is why we can’t have nice things

The Basics

Always a good place to start, this is the stuff you probably already know.

Source

At the top of all Gemfiles is a line like this one:

source "https://rubygems.org"

This line tells Bundler to use the Rubygems site as the one and only source for all gems listed herein. Here’s what you may not know:

• You can have multiple source lines, but it’s not recommended, and they are searched in the reverse order that they are listed. I don’t really like any of that, but it’s there.
• If the source you use is a private gem server, you need to set the username and password using bundle config. DON’T PUT IT IN THE GEMFILE, because it will end up on Github and then your private gems will be in the open and chaos ensues.
• OK, you can put the user and password into the URL itself (https://user:pass@gem.server.com), but don’t. You”re going to do it anyway, aren’t you? rant about how young people never listen

Gems

Right. We all know how to specify a gem in the Gemfile, right?

gem "name-of-gem"

That line will pull in the latest version of the gem. Obviously, you should lock your dependencies to versions that you know work with your application. There are several ways to specify gem versions:

• Use a specific version: gem "name-of-gem", "1.0". You can find specific versions on Rubygems.org (provided that’s the source you”re using) by searching for your gem and looking at the “Versions” listed.
• Use a version operator: gem "name-of-gem", ">1.0". This pulls in any gem that is after 1.0.
• Use that weird squiggly-stabby operator: gem "name-of-gem", "~>1.3". This operator, which some call a “twiddle-whacka”, basically says “any version greater than this one, but less than the next minor revision”. Since I am sure you know what semantic versioning is, I don’t need to explain that the minor version is the 2nd number. In the example, it means “any version greater than or equal to 1.3, but less than 1.4”. It works with major versions, too, so ~>1 will grab the latest version before 2.It gets a bit weird when you are using pre-release versions, like ~>1.0.rc.1, which means it will match all the 1.0.rc pre-release versions.
• Use a compound requirement: gem "name-of-gem", "~>2.2", ">=2.2.1". Here, we want a particular version, but are OK with newer minor releases. Nobody does this.

Groups

Bundler allows you to apportion your dependencies into different groups. This is used, much more often than not, to install different gems into a development environment than those installed to a production environment. You can specify the group(s) for a gem on the same line as the gem or by placing the gem in a group block:

gem "my-dev-tool", group: :development

group :development, :test do
  gem "my-test-tool"
end

You can then avoid installing a particular group (or groups) into a particular environment using the --without command line option to bundle install.

Rarer Finds

Let’s talk about other options for specifying gems that you may or may not have seen, but probably aren’t 100% clear about. At least, that’s how I feel about them.

Platforms

Platforms allow you to tell Bundler to install a gem for a specific version of Ruby. Platforms are basically groups, but Bundler handles the installation (or non-installation) of gems based on your current version of Ruby. The available platforms are listed here. Here’s how you use it:

gem "my-java-gem", platform: jruby

If I bundle and my current Ruby is MRI, my-java-gem won’t be installed.

install_if

Another way you can control the installation of a gem is install_if. Passing a proc or lambda that evaluates to a boolean (meaning, “truthy” or “falsy”) will control the installation of the gems in the block. From the docs:

install_if -> { RUBY_PLATFORM =~ /darwin/ } do
  gem "pasteboard"
end

Maybe you have another environment variable that determines if you install some gem or gems. I can’t speak intelligently on why you do things, all I can do is show you the tools.

Require

If you’ve ever seen the require keyword in a gem line, like so:

gem "rspec", require: "spec"

That means that the gem name and the name you require the gem by are different. You see, when something calls Bundler.require, Bundler will require all the gem dependencies. So, in the example, you want it to require "spec", not require "rspec".

Also we sometimes see a line like this:

gem 'rspec', require: false

That tells Bundler not to require rspec when Bundler.require is called. Rails, for example, will call Bundler.require as a part of its magic, which is why all your gems are already required.

Bundler will, by default, install all gems in the Gemfile and load them into memory. If we don’t want to load a specific gem (maybe because we’ll be doing it dynamically from within our code) we can set the require keyword to false.

Override Source

I mentioned the source keyword that lives in the top of the Gemfile, but if you happen to have gems that are served from a private gem server, you can override the source for those gems individually. This can be done on a single gem basis or for multiple gems in a block:

gem "a-gem", source: "https://my-private-gem-server.com" 

source "https://my-private-gem-server.com" do
  gem "a-gem"
  gem "b-gem"
end

Git

Another, more common, way to override where a gem comes from is to use the git, github, or path options, which pull from a git repository, Github, or a file path, respectively.

When pulling a gem directly from git, it’s just a matter of specifying the path to the repository:

gem "my-git-gem", git: "https://github.com/me/my-git-gem.git"

You need to have access to the repo, so it needs to be public, on your network, etc. The block syntax works as well, if your git repository has more than one gemspec:

git "http://github.com/me/many_gems.git" do
  gem "a-gem"
  gem "b-gem"
end

By the way, those repositories must have .gemspec files. Otherwise, you have to supply a version:

gem "my-git-gem-with-on-spec", "0.1", git: "https://github.com/me/my-git-gem-with-no-spec.git"

What if you want to pull in a branch, tag, or specific commit? Bundler lets you use the branch, tag, and ref options, respectively:

gem "my-git-gem", git: "https://github.com/me/my-git-gem.git", branch: "the_new_stuff"
gem "my-git-gem", git: "https://github.com/me/my-git-gem.git", tag: "1.0.rc2"
gem "my-git-gem", git: "https://github.com/me/my-git-gem.git", ref: "5acdb"

By the way, the ref option takes the first 5 characters (or more) of the commit hash.

Finally, if you are pulling in a git repository that has a ton of submodules (people do this?), there is a boolean submodule option:

gem "my-gem-of-submodules", git: "https://github.com/me/my-gem-of-submodules.git", submodules: true

Custom Git Sources

As it turns out, you can define custom git sources. Bundler supplies 3 git sources out-of-the-box: github, bitbucket, and gist. This provides some pretty cool shorthand:

gem "my-git-gem", github: "me/my-git-gem.git"

Just supply the path to the repo on Github (or Bitbucket). What if the owner (or organization) is the same name as the git repository? It gets even smaller:

gem "rails", github: "rails" # Works b/c the repo is https://github.com/rails/rails.git

All the options that work for the git source are valid here as well. By the way, the github option uses the git protocol, which is not encrypted, so govern yourself accordingly.

If your repositories on Github are private, you can create a personal access token and use it in the Gemfile:

gem "my-private-gem", git: "https://<the_token>:x-oauth-basic@github.com/me/my-private-gem.git"

That makes using private git repos easy, but putting your token in the Gemfile is a recipe for disaster. To keep your token secure, set an environment variable (GITHUB_TOKEN) and reference it in the git option:

gem "my-private-gem", git: "https://#{ENV['GITHUB_TOKEN']}:x-oauth-basic@github.com/me/my-private-gem.git"

Much love to this gist for first showing this to me. Unfortunately, there is still a problem with this approach, as it will write the token to the Gemfile.lock file, which is likely under source control. So, what can we do?

There is also a way to use bundle config to set up the auth. If you change the line in your Gemfile to:

gem "my-private-gem", git: "https://github.com/me/my-private-gem.git"

and then run

bundle config github.com <the_token>

Bundler will use the token to access github on your behalf. So, all your private repos will be accessible. Incidentally, that writes values into ~/.bundle/config. Hmmm…maybe I need to do another article that dives into the Bundler CLI. rant about how my work is never done

Gist

I’ve never used a Gist as a gem source, so I wasn’t quite sure how it works. I made one to figure it out:

gem "gist-gem", gist: "e0bef10af2e0f9bb2ed703c721ca9d29"

That works. TIL this. I have no idea why someone would do this. But, I don’t get MySpace, either.

Finally, you can make custom git sources for other git servers, like Github Enterprise Server, Bitbucket Server (formally Stash), or other sources. Here’s an example from the docs:

git_source(:stash){ |repo_name| "https://stash.corp.acme.pl/#{repo_name}.git" }
gem "rails", :stash => "forks/rails"

Path

If you have a gem locally that you”re developing, you can pull it in to your bundler with the path option:

gem "mine-mine-all-mine", path: "~/my-gems/mine-mine-all-mine"

Of course, there’s a block option:

path "~/my-gems" do
  gem "mine-mine-all-mine"
  gem "acts-as-myspace"
end

Ruby Version

It is possible to specify the Ruby version and interpreter (or engine) that your application requires. For example, if you are bound to Ruby 2.1 or above, but don’t want to presume 2.2 is OK, place this in the top of your Gemfile:

ruby "~>2.1"

That version obviously refers to MRI, by default. What if you want to grab JRuby?

ruby "~>2.1", engine: "jruby", engine_version: "9.0.3.0"

Which means you want JRuby 9000. The ruby keyword also provides a patchlevel option that takes all the same gem version operators, if you know what patch level you need. By the way, if I ever submit a patch, it will be patch level nicotine, adams, or eye.

Some version managers, like RVM, also read the Ruby directive and install the appropriate Ruby version, so that Bundler will never complain that we’re using a different Ruby version.

All Done

So, that’s it. The Bundler Gemfile distilled. Who knew that there are so many options for installing gems in the Gemfile? I didn’t, so this effort taught me some new stuff and reaffirmed my love for Bundler, the best dependency manager in the world. I hope you learned something, too.

Here’s a question for you: Did I miss anything? Any other tricks that we should include here? What are the young folks doing with Bundler these days?

I would like to sincerely thank Enrique Gonzalez and Fred Heath for peer reviewing this article. Despite their youth, they are very useful members of society.

Frequently Asked Questions (FAQs) about Gemfile