Hacking mruby onto Heroku
Unless you’ve been living under a Ruby colored rock, you’ve likely heard about mruby, Matz’s latest experimental Ruby implementation. What I bet you didn’t know is that you can run mruby on Heroku right now. As a matter of fact you can run just anything on Heroku, as long as it can compile it into a binary on a Linux box.
If you’re new to mruby, or to compiling binaries take a look at my last article Try mruby Today. I cover getting mruby up and running on your local machine. If you are already up to speed then follow along as we use vulcan to package mruby as binary, wrap it up in a custom buildpack and then launch an app to use mruby on the Heroku cloud.
What’s a Buildpack?
When you deploy python, ruby or anything else to Heroku we detect the language and then run the corresponding buildpack. Simply put, buildpacks are what enables us to run any language or framework on Heroku. The goal of the system is to minimize lock-in and maximize transparency.
The buildpacks are where all the magic happens in the deploy process. In the ruby buildpack Heroku detects and configures your ruby version, installs gems, and runs
rake assets:precompile. All buildpacks are open source and this means that you can fork one or build your own from scratch.
If you want to know more there is a great blog post about buildpacks Buildpacks: Heroku for Everything.
Before we can put a new piece of software such as mruby into a buildpack, we’ll need to compile it so it can run on Heroku using a build server.
Your first Heroku Build Server
If you’ve got mruby compiled locally you can just ship it up to Heroku right? Not so fast, Heroku’s machines are probably different from your personal setup. To make sure software will run we will need to compile mruby on a Heroku machine, but how do we do that?
$ gem install vulcan
Note: You might need the heroku gem installed as well you can run
$ gem install heroku to get it. Vulcan should be updated shortly to not rely on it.
Now you need to provision a place on a Heroku machine where you can compile your code. We’ll call this a build server and is actually an app that runs on a standard Heroku dyno:
$ vulcan create vulcan-schneems
Note: you’ll need to pick a different name than
Now that you’ve got a build server provisioned let’s clone a fresh copy of the mruby source
Go to the mruby Github page. From we can grab the git url to clone the source code to our computer.
$ git clone git://github.com/mruby/mruby.git $ cd mruby
Note: mruby is in Alpha release and these instructions may vary. For the up-to-date information visit https://github.com/mruby/mruby/blob/master/INSTALL
Make sure that the
bin/ directory is empty before you try to compile anything on Heroku, if not the build may fail but it still looks like it worked because you had old files in there built on your machine.
Compile with Vulcan
Now that you’re in the mruby directory, vulcan will upload the contents of the directory to the Heroku build server when you try to build it. By default vulcan will try to run ./configure && make install in the current directory you are in. If you needed to specify a different set of commands you could do so with the
--command= flag. Since we don’t need the
./configure let’s just run it with
Protip: You can get the whole output by passing
--verbose to help diagnose errors. I highly recommend this option.
The last piece of configuration that we will need is
--prefix this is the path on the server that will be cloned once it is done building your software. Traditionally the output of a build can be specified when running
./config with a
--prefix flag. For example if you wanted the output to go to
/tmp/mydirectory you could run
$ ./configure --prefix /tmp/mydirectory $ make $ make install
Since mruby doesn’t have a ./configure command this won’t work, instead we have to tell vulcan what directory our executables are in. We could just say grab the current directory specified by
. (a period) but this download will be quite large. Instead lets only grab the output of the build. When running a build, vulcan places the uploaded files (from the current directory) into an
input folder, so the result of our build will be in the
./input/bin. So lets grab those files using the command
--prefix=./input/bin. We can put all of this together to form our vulcan build command:
$ vulcan build --command=make --prefix=./input/bin --verbose # ... >> Downloading build artifacts to: /tmp/mruby.tgz
(available at http://vulcan-schneems.herokuapp.com/output/4e584d28-6451-4ff3-a72f-1d0d509d639d)
TIP: For more docs on the vulcan CLI check out the vulcan source on github.
Once the command is finished the file will be available at the url given output by vulcan the server restarts. Your build server is running on a dyno which has ephemeral hard-drives. While they can be read and written to, are cleared when the server restarts. This is to help prevent runaway processes from filling up disks, and to enforce good scalable app design. A server restart is guaranteed to happen at least once every 24 hours. A copy of the file should also be copied to your
Unzip the file to verify that it has everything we want:
$ cd /tmp $ tar xvf mruby.tgz $ ls mirb mrbc mruby # ...
This should unzip the contents and now you should have the binary files
mrbc. If you were wondering about the
mrbc binary it can be used to compile mruby programs to bytecode. If you try to execute these any files on your computer you will likely get an error since they were not built on your computer. We can double check that they built just using another Heroku app.
Verify the Compiled Files
For this section you’ll need a Heroku app to verify the files compiled correctly. With the Heroku Toolbelt installed run:
$ heroku create Creating fast-sierra-6912... done, stack is cedar http://fast-sierra-6912.herokuapp.com/ | firstname.lastname@example.org:fast-sierra-6912.git
Note: Your app name will be different from mine, replace as needed for the following commands.
Now you’ll want to be able to run commands inside of your app. You can do this using the
run command, or you can open up a bash session using
heroku run bash like this:
$ heroku run bash -a fast-sierra-6912
fast-sierra-6912 with your app name.
heroku run bash spins up an extra dyno, and gives you an ssh session into it. This is different from ssh-ing into a regular server since it is completely isolated from any other dynos or web requests. You could delete the whole app’s drive while in
heroku run bash and it wouldn’t affect any other dynos.
Now that we are in a Heroku bash session we need to get the binary on the machine. We can do this with the
curl command and the
-O option (capital O as in Oscar). Use this command along with the binary url we got from vulcan earlier to download the binary data.
# on heroku bash $ curl -O http://vulcan-schneems.herokuapp.com/output/4e584d28-6451-4ff3-a72f-1d0d509d639d
Note: you’ll need to use your own URL here.
Then unzip the file
# on heroku bash $ tar xvf 4e584d28-6451-4ff3-a72f-1d0d509d639d x mirb x mrbc x mruby
Now you’ve got the binaries copied to Heroku execute them just as you did on your local machine in my last article Try mruby Today:
# on heroku bash ~ $ ./mirb mirb - Embeddable Interactive Ruby Shell This is a very early version, please test and report errors. Thanks :) >
It worked! We were able to compile a binary that we can use to run on Heroku. We can use this same technique to compile any library you can imagine. But we aren’t quite done yet. Now that we have a binary, let’s put it into a buildpack so that can re-use this binary on any app we want. Let’s get started.
Fork a Blank Buildpack
While we could start with a buildpack that has more going for it, it is much easier to see what is going on starting with a blank buildpack created by Heroku engineer Ryan Smith. Fork the “null” buildpack then change the name to something more appropriate like ‘mruby-buildpack’ to do this you can go to admin, select “rename” and then enter ‘mruby-buildpack’. Once you’ve done this clone the from your personal github repo onto your local machine:
$ git clone git://github.com/schneems/mruby-buildpack.git $ cd mruby-buildpack
Note: Your url will be different (not schneems/mruby-buildpack)
Once you have the repo locally, open it up, and look at it. There really isn’t much here 4 files and a folder in total. If you look in the
bin directory you will see 3 files:
release. These are the three stages of the build process and these files will be called by runtime during a deploy. We will get more into those later.
Build a test App
Let’s verify our ever-so-simple null buildpack works. We will make a new Heroku “app” and deploy with our custom null buildpack:
First make a new directory (make sure you’re not inside of your buildpack directory first).
$ mkdir null $ cd null $ touch README.md
Then initialize this directory as a git repo
$ git init $ git add . $ git commit -m "initial commit"
Now we’ll want to create an app on Heroku and tell it to build using our newly forked buildpack. We can do this by specifying the buildpack url when we create the app or by modifying the
BUILDPACK_URL in the config. Run this command but remember to point it at your buildpack url and not mine:
$ heroku create --buildpack http://github.com/schneems/mruby-buildpack.git Creating still-mountain-4026... done, stack is cedar BUILDPACK_URL=http://github.com/schneems/mruby-buildpack.git http://still-mountain-4026.herokuapp.com/ | email@example.com:still-mountain-4026.git Git remote heroku added
Remember: Use your github repo and not
Once you’ve done that you can verify the buildpack config variable is set:
$ heroku config === still-mountain-4026 Config Vars BUILDPACK_URL: http://github.com/schneems/mruby-buildpack.git
Great! You can change this url any time in the future, for instance if you wanted to use a specific branch of a buildpack. You can specify it using
# (a hash mark) and the name of the branch. If you had a branch named
total_re_write you could specify it using the url
http://github.com/schneems/mruby-buildpack.git#total_re_write, and the
heroku config:add command; but let’s not worry about that for now.
Deploy your project and make sure everything works:
$ git push heroku master Counting objects: 3, done. Writing objects: 100% (3/3), 212 bytes, done. Total 3 (delta 0), reused 0 (delta 0) -----> Heroku receiving push -----> Fetching custom git buildpack... done -----> Null app detected -----> Nothing to do. -----> Discovering process types Procfile declares types -> (none) -----> Compiled slug size: 4K -----> Launching... done, v4 http://still-mountain-4026.herokuapp.com deployed to Heroku To firstname.lastname@example.org:still-mountain-4026.git * [new branch] HEAD -> master
Notice the output
Nothing to do. comes straight from the compile command in your repo.
Before we add mruby to our buildpack let’s take a look at what each of the stages of the build does.
When Heroku calls the detect command on a buildpack it is run against any source code you push. This is where an app such as a ruby app could check for existance of a
Gemfile or a
config.ru to double check that it can actually run the given project. If the buildpack can build based on the files, it returns a
0 to Heroku which in *nix interprets as “everything ran as expected”. If the buildpack did not find the files it needs, it should return a non zero status, usually
1. If that happens Heroku will cancel the deploy process.
Since we’re manually specifying the buildpack url, it is safe for us to always return a 0.
This is where the majority of the build takes place, once the application type has been determined, different setup scripts can be run to configure the system such as installing binaries or running
The biggest thing to note about the compile stage is that the config environment variables are not available during the compile step. This is done on purpose since a well architected app should compile the same regardless of configuration. There is a Heroku labs feature to enable the environment variables in the compile process, though I recommend not using it. I’ve seen more than my fair share of un-reproducable “it worked on staging, why is it broken on production” problems by using this feature. Compiling your application devoid of configuration is considered best practice.
A cache directory is available in the compile stage. Anything put in the cache directory will be available between deploys. It would be a good idea to store downloaded items such as gems here, but we don’t need to cache anything just yet.
In the compile phase, we will need to download our pre-compiled binary, unzip it, and put it somewhere that we can find it later.
So when does the config get added to the project? We’ll it’s not detect or compile, so it’s probably…
The release is where the compiled app gets paired with their environment variables. You shouldn’t make any changes to disk here, only changes to environment variables. Heroku expects the return in YAML format with 3 keys
addons if there are any default addons,
config_vars which supplies a default set of configuration environment variables and
default_process_types which will tell Heroku what command to run by default (i.e.
You can get more information on these three steps through the buildpack documentation.
Now that you know how Heroku runs buildpacks, lets open your forked buildpack, add mruby to it and then do something.
mruby-buildpack/bin/detect and change
"mruby". That was the easy part. Take the binary you downloaded on to your local machine and put it somewhere publicly accessible like S3. This will need to be somewhere that Heroku can access it. You can find my copy at
https://s3.amazonaws.com/schneems-heroku/mruby.tgz. If you don’t have an S3 account, anywhere public will do for now, even dropbox.
mruby-buildpack/bin/compile. Since this file is a bash file we can write command line commands directly into it. If you preferd to write your scripts in ruby we could add extra files to the buildpack and have the compile step call them, but for this exercise we will stick to shell scripting. First we want to make sure we are in the correct build directory, this is passed in as the first argument to our bash script so we can get it using
# change to the the BUILD_DIR ($1) cd $1
Then we want to download the file from the public location using
curl -O like this:
# download the mruby binary (-O) silently (-s) curl https://s3.amazonaws.com/schneems-heroku/mruby.tgz -s -O
*Note: Remember to point at your url instead of mine. Also the
-s flag in
curl silences the download, if you want to see it happen take it out.
Now we need to make a directory to store our binary let’s put it in
vendor/mruby_bin. We can run
mkdir to do this:
# make a directory to untar (like unzip) the binary mkdir -p vendor/mruby_bin
Now we want to unzip the files to the proper directory using the
tar xvf command, where
-C is used to specify the output directory of the file:
# untar the binary to the directory we want tar -C vendor/mruby_bin -xvf mruby.tgz
Finally clean up any unused files
# clean up the unused files rm mruby.tgz
As a nice touch you can replace this line:
echo "-----> Nothing to do."
With this one:
echo "-----> Installing mruby like a boss"
The final compile file looks like this for me
#!/usr/bin/env bash # bin/compile <build-dir> <cache-dir> echo "-----> Installing mruby like a boss" # change to the the BUILD_DIR ($1) cd $1 # download the mruby binary (-O) silently (-s) curl https://s3.amazonaws.com/schneems-heroku/mruby.tgz -s -O # make a directory to untar (like unzip) the binary mkdir -p vendor/mruby_bin # untar the binary to the directory we want tar -C vendor/mruby_bin -xvf mruby.tgz # clean up the unused files rm mruby.tgz
That wraps up the compile section, but we aren’t done quite yet. If you remember when we got mruby working locally we modified our path so that we could call the
mruby command from anywhere. To enable this functionality on your Heroku app, we’ll need to modify the path to include
Before we call try pushing we’ll want to add the files in
vendor/mruby_bin to our path so we can access them from any directory. To do this we’ll need to set our PATH which is part of the environment configuration variables. Since these aren’t available to us during the compile stage, we’ll have to add them during the release. You’ll need to append the path to the existing
PATH to make sure we don’t break anything.
bin/release you can output default config variables in a YAML format like this:
#!/usr/bin/env bash # add our mruby_bin folder to the path on first deploy echo "---" echo "addons:" echo "config_vars:" echo " PATH: $PATH:/app/vendor/mruby_bin" echo "default_process_types:"
Notice that we added the full path
/app/vendor/mruby_bin which is an absolute file path (it starts with a
/). When your app is deployed it will live in
/app. Since we added our files to
vendor/mruby_bin we need to add the full location to the path. The colon character
: is a separator. The PATH can have multiple paths and they are searched in order.
One important caveat of changing environment variables in a buildpack, they are considered “defaults” and will not over-ride existing config variables. If you mess up setting a config_var on one app, you’ll need to make a new app (
heroku create #...) to test the full changes.
Now that we’ve got our compile and release all sorted out we’ll need to commit our changes to our buildpack and push it back up to github so that Heroku can access it.
$ git add . $ git commit -m "adding mruby like a boss" $ git push origin master
Double check that the changes show up on your github repo and double check that the buildpack url in your app points to the same github repo. When you’re done we’re all ready to deploy!
mruby Buildpack Deploy
Go back to your null app (not the buildpack folder, the app with the empty readme we created). Make a new commit
$ git commit -m "empty" --allow-empty
Then deploy and you should see your app in action:
$ git push heroku master ----> Heroku receiving push -----> Fetching custom git buildpack... done -----> mruby app detected -----> Installing mruby like a boss -----> Discovering process types Procfile declares types -> (none) -----> Compiled slug size: 2.9MB -----> Launching... done, v4 http://morning-taiga-3831.herokuapp.com deployed to Heroku
Let’s quickly verify our path is correct:
$ heroku config PATH: /usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/app/vendor/mruby_bin
You should see
/app/vendor/mruby_bin in there. Look’s good to me, let’s give it a spin and see if everything worked. Of course no language tutorial would be complete without a hello world, let’s run one on our Heroku app now:
$ heroku run "mruby -e "puts 'hello world' " " Running `mruby -e "puts 'hello world' " ` attached to terminal... up, run.1 hello world
Congrats! You just built a full functioning buildpack that runs a language so hipster it doesn’t have version numbers yet. You should take this opportunity to celebrate, and tell all of your programming buddies how great you are. Also don’t forget to point link them to this article while you’re gloating (of course).
Now that mruby is available on the system you can use it through
$ heroku run bash or directly via the
run command like
$ heroku run mirb.
Why run mruby on Heroku?
Besides the obvious “because you can”, mruby is a new implementation of my favorite languages. This means that it has different strengths and weaknesses. It’s meant to be run on small targets and to be lightweight. I ran through a number of tests comparing mruby and standard MRI ruby (cruby). If there are quite a few floating point calculations then mruby will kick cruby’s butt, but on a large number of other operations it will lag quite far behind. Again the point isn’t to be fast, but to be light.
While running the tests I checked ram usage and consistently saw cruby using roughly 4x the amount of memory as mruby. So if we could re-write our web apps in mruby, theoretically we could squeeze in 4 times the number of processes into one dyno, which would be pretty impressive.
It’s important to remember that mruby isn’t a replacement for cruby and is still just in alpha. It currently has no standard library, and does not have the ability to use
require, so don’t expect to be running rails apps on it any time soon.
Since it has a relatively lightweight footprint and we could run many processes concurrently on a single dyno, and because of it’s great floating point performance: mruby could be a good fit for parallelized map-reduce or machine learning jobs on Heroku’s scalable architecture.
As the language matures and gets some more real world usage, it will be exciting to see what types of devices mruby gets embedded in, what games choose to use it as a scripting language and what kind of crazy tasks it takes up in the cloud.
Wow, so that was quite a bit of work. Can you imagine writing a buildpack from scratch to handle any Rack/Rails app? Luckily we’ll take care of that for you. If you did need to make a custom buildpack it can be quite a bit easier to start from one of the existing (and not null) buildpacks.
If you need a binary on the system but don’t want to maintain a fork of a buildpack or instance if you wanted to put the mruby binaries in your git repo and change your config PATH to point at that directory.
So now you understand more of what goes into the Heroku build process and how compiling and packaging binaries works on Heroku. If you do something interesting with mruby or a custom buildpack on Heroku let me know about it: @schneems. At the end of the day the more you know about how your system works, the better you will be able to use it.