Recently, I had the need to investigate better methods of uploading pictures to our web application. The app is Rails-based and hosted on Heroku. Heroku, as I am sure many of you know, constrains its hosted applications with a read-only file system. In actuality, the read-only file system is an eventual constraint, meaning, Heroku will allow the application to write to a small, temporary space on the file system. However, that temp space is swept frequently and without predjudice, so any files in it don’t live very long.
Since Heroku does allow writing to this temp space, it’s possible to deploy an app that allows users to upload pictures, which your application can manipulate/resize to its needs. There are numerous examples of how to do this which gems like CarrierWave, which is where we started. Unfortunately, this approach does not scale with Rails. Uploading a file using Rails will consume a web dyno. If the file is large and takes more than 30 seconds to upload, then your app will start throwing timeout errors all over the place. This article goes into a bit more detail about this.
In short, if you have a serious Rails application hosted on Heroku that requires file uploads, you need to find an approach that uploads the file directly to S3 (or a dedicated image service) and performs image manipulations outside of your Rails app. Your life will be much better for it.
Options
In my research on this issue, I came across CarrierWave-direct, which allows direct uploading to S3. I tried, in earnest, to make it fit our existing flow and could not. Also, I beleive that Carrierwave will pull the file from S3 onto Heroku’s servers to perform its processing, which I wanted to avoid. I am not here to bash CarrierWave or CarrierWave-direct, but to bring attention to them before I go into how Transloadit fits into the picture.
Transloadit Basics
Along with every other developer on the planet, Node has caught my attention. I read about it daily, try to keep up with happenings in that community. This is where I first heard about Transloadit. I have heard many rave reviews of it’s service since then, so I decided to give it a go. The basics of Transloadit consist of four items: Assemblies, Robots, Steps, and Templates.
Assemblies
An assembly is, basically, an individual job on Transloadit servers. In our example today, an assembly will encompass the uploading and resizing of our image.
Robots
The Transloadit Robots (and I LOVE that they call them that) each have a special skill. Some robots resize images, some store files, etc. You bring the robots into your jobs as needed using Steps
Steps (or Assembly Steps)
Steps are one part of an assembly. Resizing an image is a step (performed by a robot) and writing the file to S3 is another step
Templates
Templates bring everthing else together. It is your job, as the developer, to create a template that includes the assembly steps, performed by robots, to satisfy the needs of your application. Templates are specified in JSON, so it should be in your wheelhouse. In other words, creating a template is very simple.
Example App: Uploading Pictures
Right, let’s bring this all together with a contrived application. This application will simply allow a user to upload a photo and create a thumbnail and “fullsize” version of the photo. I won’t bore you with the basic setup details of the Rails app, but the source is here
Setting Up Transloadit
First things first, go sign up for an account on Transloadit. They have a free plan that should get you through this demo (although, not much farther, unfortunately.)
Transloadit needs to know what to do with our pictures, so we need to go to our templates section and create a new template.
I mentioned above that we want a thumbnail and a fullsize version of the pictures, so we need to define a template that creates those versions. Looking through that more-than-adequate documention on Transloadit’s site, we can structure our template as:
[gist id=”2276701″]
Briefly running through that template, the first step creates our thumbnail image at 50X50 pixels. It uses the image/resize robot on the “:original” image. Specifying "use" : ":original" is important. By default, Transloadit will feed the output of one step into the next step, which in this case, would distort our full size image.
The second step, as you probably guessed, creates our fullsize image at 320X1000 pixels. The last step (“export”) writes our file to S3 using the /s3/store robot. This step has some interesting values.
The use parameter includes the two images we created plus the original, which means each assembly will write 3 files. Also, our AWS authentication (if you need to setup S3, this page will help) information is in this step. Finally, the path step defines how we will name the files. Each job will result in a folder named for the assembly ID and the files will be based on the step name that created them.
The template can be tested via Transloadit’s web site, which is really cool. Presuming you have S3 setup, go ahead and upload a file and check out the two versions it creates.
Workflow for Uploading a Picture
The diagram below shows a high-level flow of uploading a picture using Transloadit.
1) The user submits a form that contains a file input.
2) The file is uploaded to the transloadit service, and the robots do their thing.
3) Once the files are ready, they are moved to S3.
4) The form submission is redirected to the rails app, along with some extra parameters from Transloadit.
5) The request returns to the user.
Now, this is just ONE example of a workflow using Transloadit. There are many options, including NOT waiting on the file upload/processing to complete and just being notified when it’s all done. When looking into the service, explore all the options and use the best approach for your particular situation (duh).
The Code
For our application, we are going to use the minimal integration approach along with Transloadit’s jQuery plugin. Using this approach, we can take our vanilla web form:
[gist id=’2276857′]
and decorate it using Transloadit’s jQuery plugin, like so:
[gist id=’2276822′]
The wait parameter tells the plugin to not redirect to my redirect_url until the file processing is complete.
To make it work, we have to include some parameters in the form that specify our Transloadit API key as well as which template to use. The hidden parameters look like:
[gist id=”2276804″]
Here is our form with that information encoded and added to the form.
[gist id=”2276789″]
Obviously, you’ll need to change your key and template id information. Also, notice that the redirect_url is http://localhost:3000/pictures, which is where the form will be redirect to once the file upload and processing completes.
Handling the Redirect
With all the uploading goodness in place on the client, that last task is to handle the redirection of the form from Transloadit in our controller. Since each upload will have a unique assembly id, that value will be stored on the picture and used to generate a url for each photo.
The controller code itself is very simple:
[gist id=”2276947″]
Transloadit redirects the form with all of the paramters we need (assembly_id and file name information) which we extract and then apply before the vanilla create method. The redirect includes the parameters from our own form as well, allowing us access to the data inputted by the user. Pretty easy.
Gotcha
Looking back at the controller code, you may have noticed that I downcase the file extension when generating the file name. The URLs to resources on S3 are case-sensitive, and the Transloadit robots create their file versions with lowercase extension values. As a result, if you upload ‘IMAGE.JPG’ the thumbnail will be called ‘thumbnail_IMAGE.jpg’ and it will 404 if you don’t use ‘jpg’ in your urls.
Upload Away
I hope you found this article useful. Transloadit is a very powerful service that offers a slew of options for uploading assets (they do more than just photos) and allows you to keep the uploading process out of your main app. On hosts like Heroku, this can keep your processes free to handle other things, more core to your business and keep your users very happy.
Other Resources
There is an official Rails gem for using transloadit, that abstracts most of what you saw in this article. There is also a Ruby gem and SDKs for various other platforms.
Glenn GoodrichGlenn works for Skookum Digital Works by day and manages the SitePoint Ruby channel at night. He likes to pretend he has a secret identity, but can't come up with a good superhero name. He's settling for "Roob", for now.
