Getting Started with TransloadIt – Key Concepts

Transloadit is a file-processing service primarily designed for processing images, videos and audio.

It allows you to perform various operations — conducted by what it calls “robots” — on these files; for example resizing and optimizing images, extracting thumbnails from videos or converting files from one format to another.

TransloadIt also allows you to upload files directly to their servers for processing rather than via your application. Indeed you can bypass your server altogether, transferring the processed files straight to an Amazon S3 bucket, a Rackspace Cloud Files account or to your own server via (S)FTP. So in other words, your application need only be concerned with where to find the resulting files, without having to process or store them directly — great for running distributed applications or for those on clusters.

In this tutorial we’re going to take a detailed look at Transloadit, and in particular using it to handle images. We’ll look at some of the key concepts, such as constructing “assembly instructions” to tell it what we want to happen to our uploaded image files. We’ll “hijack” a file upload form to send a file directly to TransloadIt, run a set of instructions to generate a number of “derivatives” — different-sized images — and then save them to an Amazon S3 bucket. Finally, we’ll look at how we can use the results of these operations to notify our application about the newly generated files, so that we can update our database accordingly.

Before we look in detail at Transloadit, it’s worth taking a step back and discussing why you’d want to use a third-party service for something like this in the first place, rather than implement it all yourself.

Why Use a Third-Party Service?

Image handling isn’t all that difficult to implement yourself, and there are all sorts of libraries out there for just that. So why would you want to use a third-party service to handle it at all?

• For one thing, it allows you to concentrate on what your application is really all about. Why spend additional time and effort on, say, user profile photo functionality when you could be concentrating your development efforts on mission-critical functionality?

• Image processing can be quite resource intensive. In particular, resizing large images can eat up a lot of memory. Using a service such as Transloadit eases the load on your servers.

• Processing files usually requires additional libraries or programs to be installed on your server; for example gd or Imagemagick for images or ffmpeg for videos and audio. By using a third-party service, there’s no need to worry about installing, configuring or maintaining them.

• Transloadit “jobs” can be run in the background, rather than hold up execution of your application. While there are solutions to get around this in your own application — for example queues such as RabbitMQ or Beanstalkd, or job servers such as Gearman, using TransloadIt frees you from the additional development effort and installation steps required to implement them.

• The Transloadit servers are highly optimized to handle file uploads, so performance may well be much better than if you implement them yourself.

• Permitting files to be uploaded to your web server also has all sorts of security implications which you must take into consideration. By using a third-party service, these risks are somewhat mitigated.

• It’s decentralized. You can set it up such that uploaded files go via the service and straight to an external storage solution such as S3, without ever having to be stored on your web server. Whilst this might not seem a huge advantage in a small-scale application, you’ll reap the benefits as soon as it grows to the point where you have multiple services, for example a cluster.

Depending on your application, your preferred hosting solution, your expertise or the amount of time you happen to have on your hands, chances are not all of these will apply. Perhaps none; it’s not the optimum solution for all cases.

Naturally, there are disadvantages. Here are a few:

• It introduces an additional point-of-failure.
• It could mean entrusting sensitive files to a third-party.
• Finally, there’s the financial cost.

Bearing all of those pros and cons in mind, let’s start taking a detailed look at the service. Then, in part two, we’ll run through a practical example of how to use it.

Getting Started with Transloadit

Transloadit offer a free sandbox account that gives you a total of 1GB capacity to play around with — which will be more than enough to follow along with this tutorial, and then some. A number of paid plans are available, which give you varying amounts of capacity per month. Paid plans start at $19 per month, at time of writing.

The actual definition of capacity varies according to what you’re using the service for. To illustrate what that means in the context of image processing, you can determine the usage involved in processing one particular image upload by adding the file size of the original file, to the total size of the resulting processed images. So suppose you upload an 800kb image, which in turn is used to generate a large version of 400Kb, a medium version of 200Kb and a thumbnail of 30Kb. If you use the smallest possible version as the input for the next resize, then that would use (800Kb + 400Kb) + (400Kb + 200Kb) + (200Kb + 30Kb) = 2,030Kb towards your limit. These calculations become more complex when you start working with videos or audio, so you’ll probably want to consult their website.

If you plan to follow along and go through the practical example in part two, now might be a good time to sign up. Head on over to the site and hit “Get Started”.

Now let’s look at some of the concepts and the components that make up the service, in order to better understand the process.

The Concepts

Assemblies

An assembly is essentially a set of instructions which tells the Transloadit service what actions you want them to carry out on a particular file (or group of files).

In practice, you’d probably create different sets of assembly instructions for different types of files. Thereafter, you’d probably create subtly different assembly instructions for sub-types. For example, user profile pictures / avatars might require one particular set of sizes, and profile banners an altogether different set; so in this scenario you’d create a set of assembly instructions for each.

Assembly instructions contain a list of steps, defining precisely what processing to carry out and in what order. Each step is performed by a robot.

Robots

A “robot” is like a worker, which performs some sort of processing. There are a number of different types of robot:

Import Robots can take files from a variety of sources, and not just via form-based file uploads. For example, you can create a robot which imports files over HTTP, using (S)FTP or by downloading them from an Amazon S3 bucket.

Assembly flow logic robots can be used to filter incoming files — for example, you can set up a robot to reject files which are not images.

Image robots perform a number of different tasks on image files, most notably resizing. There are also robots for optimizing image files or taking screenshots.

Video and audio robots perform actions on those particular file types. Because this tutorial is focused on images, we won’t be looking at these in any great detail – but stay-tuned for a follow-up article later.

File Export robots are used to transfer the result of your assemblies to some form of permanent storage. It’s important to note that TransloadIt don’t host your files; although they make them available for download via S3 buckets after processing, they won’t be stored beyond 24 hours. It’s your responsibility to ensure your files are stored somewhere that you control. You can set up a robot to transfer your files to an Amazon S3 bucket, to Rackspace Cloud Files, or transfer them to your server using (S)FTP.

Building Assembly Instructions

Assembly instructions are defined in JSON format, or as a hash which you can use with one of the various available libraries. Most of the time you’ll probably use the official jQuery plugin to interact with the service from your client application, but there are other options available. You needn’t necessarily use it from a client-side application — there are server-side libraries available as well.

In part two, we’re going to look at how you construct assembly instructions in detail.

Templates

Assembly instructions tell the robots what to do with your files, but also include any additional information they’ll need to run them. This is a real issue when it comes to the file export steps in particular, since they’ll require credentials with which they can access your S3 bucket or (S)FTP service. Clearly defining these in client-side code would be disastrous, so for that reason you’d use templates.

Rather than define your assembly instructions when you call the TransloadIt API — for example, when using the jQuery plugin — you can instead create a JSON document containing the relevant assembly instructions on TransloadIt’s server, and then reference that template by its unique id. Templates are encrypted server-side for additional security.

You can also merge additional parameters, or override existing ones, into a template at runtime — all the while, keeping your sensitive information such as S3 credentials safe from prying eyes.

Signatures

Signatures get around the problem of exposing your TransloadIt authentication credentials in client-side code, in much the same way as templates protect your storage-related credentials.

A signature is essentially a short-lived access token, which you’d generate on the server, pass through to your client application and then use to authenticate with the service. If someone should obtain that value, then it would very quickly become useless. Signatures are also “bound” to a specific set of assembly instructions, or one particular template, making them even more secure.

Variables

You can use variables in your assembly instructions. These are particularly useful when determining file names of processed files. So for example, if you’re resizing an uploaded user avatar, you can provide the user id or username to form part of the resulting file name. Later we’ll see how the names of the various steps you can define, can also be used as variables.

Using Form Fields

When you’re using TransloadIt from a client application (e.g. an upload form) then variables can be derived from the form fields. So in the example above for uploading user avatars, you could place the user id in a hidden form field and then incorporate that into the instructions you send to TransloadIt’s servers.

Summary

In this first part, we’ve taken a high-level look at the TransloadIt service. We’ve looked at what it can do, and some of the pros and cons of using such a service to perform the sort of tasks that you might otherwise implement yourself.

We’ve looked at some of the concepts you need to understand in order to get the most out of TransloadIt and with that background knowledge, we’re ready to start building a practical example. Stay tuned for part two.

Frequently Asked Questions about Image Uploads and Manipulation with Transloadit