![Abstract tooth wheels](https://uploads.sitepoint.com/wp-content/uploads/2013/12/Fotolia_51801972_Subscription_XXL-256x300.jpg)
This is part 2 of How Asset Precompile works in Rails. In part 1 we started digging into Rails built-in support of packaging assets, how it compiles static assets (images), and how a digest is generated based on their content etc. This article contains a bit more in-depth coverage of the Rails asset pipeline.
Types of Assets
If you remember, there are three types of assets according to Sprockets
- Bundled Assets
- Processed Assets
- Static Assets
We covered about Static Assets in part 1. Bundled Assets are assets which require some processing in-order to get created. Assets such as CSS files and Javascript files are example of Bundled Assets. Processed Assets are parts of Bundled Assets. In simple words, a Bundled Asset is composed of different Processed Assets and is used to store the file after processing is taken from Bundled Asset.
For example, application.js
is a Bundled Asset at first and it consists of different Processed Assets (including itself) which are defined in it according to the Sprockets
structure. Bundled Assets are saved on disk and contains contents of Processed Assets. Bundled Assets serve as a container for different Processed Assets.
Now, let’s deep dive and see how Rails Asset Precompile accomplishes this work. For this article default application.js
is used as a reference asset and article focuses on how application.js
gets precompiled.
Before proceeding, please open following files in your favorite editor. If you like, you can click below to view them in the browser.
- actionpack-3.2.15/lib/sprockets/railtie.rb
- actionpack-3.2.15/lib/sprockets/assets.rake
- actionpack-3.2.15/lib/sprockets/static_compiler.rb
- sprockets-2.2.2/lib/sprockets/base.rb
- sprockets-2.2.2/lib/sprockets/index.rb
- sprockets-2.2.2/lib/sprockets/bundled_asset.rb
- sprockets-2.2.2/lib/sprockets/processed_asset.rb
- sprockets-2.2.2/lib/sprockets/context.rb
- sprockets-2.2.2/lib/sprockets/directive_processor.rb
rake assets:precompile
To precompile assets we type rake assets:precompile
. This is a rake task, and is added in actionpack-3.2.15/lib/sprockets/railtie.rb
at line 13 where the sprockets/asset.rake
file is loaded. This code is in Sprockets::Railtie
which is a subclass of Rails::Railtie
. As discussed in part 1, that Railtie is the core of the Rails framework and provides several hooks to extend Rails and/or modify the initialization process. It is also used to add rake tasks.
In actionpack-3.2.15/lib/sprockets/assets.rake
at lines 59-67, there is a rake task. This is the rake task which gets called when we run rake assets:precompile
. This rake task, in turn, call another rake task, which calls the internal_precompile
method.
In internal_precompile
at line 50 an instance of Sprockets::StaticCompiler
is created by passing different options and assigned to compiler
. The env
that is passed to Sprockets::StaticCompiler
is an instance of Sprockets::Environment
which was assigned to Rails.application.assets
in actionpack-3.2.15/lib/sprockets/railtie.rb
at line 23 (Which we discusses in Part 1.)
internal_precompile
calls the compile
on Sprockets::StaticCompiler
. This is the method that performs all necessary operations to precompile our assets and write them to disk.
Let’s head to the compile
method in actionpack-3.2.15/lib/sprockets/static_compiler.rb
. In this method you will see the following line:
env.each_logical_path(paths) do |logical_path|
env
is Rails.application.assets
which is instance of Sprockets::Environment
. In the above line paths
is basically Rails.application.config.assets.precompile
which was passed to Sprockets::StaticCompiler
in actionpack-3.2.15/lib/sprockets/assets.rake
.
each_logical_path
is defined in sprockets-2.2.2/lib/sprockets/base.rb
at line 332. This method, in turn, calls each_file
which is defined just above each_logical_path
.
each_file
performs an each
on paths
. paths
is list of directories which contain our assets. The paths for a newly created Rails 3.2.15 application in which you haven’t added any extra gems might look like this.
/home/ubuntu/Desktop/work/asset_pipeline_article/app/assets/images
/home/ubuntu/Desktop/work/asset_pipeline_article/app/assets/javascripts
/home/ubuntu/Desktop/work/asset_pipeline_article/app/assets/stylesheets
/home/ubuntu/Desktop/work/asset_pipeline_article/vendor/assets/javascripts
/home/ubuntu/Desktop/work/asset_pipeline_article/vendor/assets/stylesheets
/home/ubuntu/.rvm/gems/ruby-1.9.3-p194@exporter-imranlatif/gems/jquery-rails-3.0.1/vendor/assets/javascripts
/home/ubuntu/.rvm/gems/ruby-1.9.3-p194@exporter-imranlatif/gems/coffee-rails-3.2.2/lib/assets/javascripts
Your absolute paths will be different based on your Rails application path. The first 5 paths are from app/assets
and vendor/assets
. The last two paths are from the jquery-rails
and coffee-rails
gems, respectively.
If you are a gem author and want to include your gem’s assets paths in Rails.application.config.assets.paths
, then you need to create an engine class that inherits from Rails::Engine
. This is described here. paths
are first assigned to Rails.application.config.assets.paths
and later assigned to Rails.application.assets.paths
. You can check the code that assigns Rails.application.config.assets.paths
to Rails.application.assets.paths
here.
Back to each_file
in sprockets-2.2.2/lib/sprockets/base.rb
. each_file
performs each
on paths
and in that block each_entry
is called.
each_entry
is defined just above each_file
. each_entry
recursively iterates over each directory and collects all files in a that path. After collecting all files from a given path it executes the block passed to it. You can see following line in each_entry
method:
paths.sort_by(&:to_s).each(&block)
The above line calls the block which was passed to the each_file
method from each_logical_path
. That block calls logical_path_for_filename
, which is defined in the same file.
logical_path_for_filename
calls matches_filter
which, also defined in the same file. matches_filter
matches filename
according to the filters
passed to it. filters
are, basically, Rails.application.config.assets.precompile
, an array containing different filters according to which an asset should be processed.
By default, the array has two items. First, a Proc
that matches the name for Static Assets. The second item contains a RegExp
object that matches CSS and Javascript files.
If a filename is not matched, then there will be no asset for it. For Static Assets, like images, it should match the filter defined by the proc
in the filters
array. For Bundled Assets, the filename should match the RegExp
defined in filters
.
If we want to bundle CSS or Javascript files that do not match the regular expression, then we have to add it to Rails.application.config.assets.precompile
on our configuration files. We do this in config/environment/production
by using following line:
config.assets.precompile += %w( sitepoint.js )
The proc
in the filters
array only matches Static Assets or assets which don’t have a .css
or .js
extension. The RegExp
in filters
only matches files which are either application.css
or application.js
. Since “sitepoint.js” does not match either of these criteria, it won’t be bundled. Appending sitepoint.js
to Rails.application.config.assets.precompile
forces a match and bundling of the asset will be performed.
After getting a successful match for the filename
passed to the filters_matches
method, logical_path_for_filename
returns that result to the block passed to each_file
from each_logical_path
. This block yield
the filename to another block passed to it from compile
in actionpack-3.2.15/lib/sprockets/static_compiler.rb
file. That block calls find_asset
defined in sprockets-2.2.2/lib/sprockets/index.rb
. This method sets options[:bundle]
to true and calls find_asset
from its Base
class in sprockets-2.2.2/lib/sprockets/base.rb
.
find_asset
resolves the path based on the filename and assigns absolute path of file to pathname
.
logical_path
is just name of the file i.e. application.js
. find_asset
calls build_asset
, which decides how to handle the asset. If there are no processors to run then it treats it like a StaticAsset
.
If there are some processors to run, then it treats theasset as a BundledAsset
. The first time build_asset
is called, options[:bundle]
is true
, so a BundledAsset
is created. find_asset
is called again with options[:bundle]
set to false
, so a ProcessedAsset
is created.
Remember our discussion that BundledAsset
is actually saved to disk containing different ProcessedAsset
instances. Every BundledAsset
has at-least one ProcessedAsset
. In the case of a single CSS or Javascript file, such as sitepoint.js
, both the BundledAsset
and ProcessedAsset
point to the same file.
The initializer for ProcessedAsset
executes context.evaluate(pathname)
to get the contents of the file in pathname
. evaluate
in sprockets-2.2.2/lib/sprockets/context.rb
performs different operations on the files mentioned in path
. It gathers processors
to run on the file.
By default for Javascript file, the two processors
are Sprockets::DirectiveProcessor
and Sprockets::SafetyColons
. Sprockets::DirectiveProcessor
is used to run a directive processor on each CSS and Javascript file. It basically scans the CSS or Javascript file and captures files that are required in that file according to the Sprockets
syntax. Popular directives are require
, require_tree
, requite_self
etc.
evaluate
runs each
on the processors
array and a new instance of the processor
is created. The render
method is then called on each processor template. Sprockets::DirectiveProcessor
is defined in sprockets-2.2.2/lib/sprockets/directive_processor.rb
and it is a subclass of Tilt::Template
. When render
is called on Sprockets::DirectiveProcessor
it also calls evaluate
on Sprockets::DirectiveProcessor
. In evaulate
, process_directives
is called, which scans files and gathers directives that needs to run on the current file.
For our reference asset application.js
result might look like this:
[[13, "require", "jquery"], [14, "require", "jquery_ujs"], [15, "require_tree", "."]]
After getting directives and assigning them to @directives
, process_directives
is executed. As you can see, this line of code sends a process_<name>_directive
message to each directive:
send("process_#{name}_directive", *args)
All these methods belongs to Sprockets::DirectiveProcessor
. Let’s see what happens when process_require_tree_directive
is called.
process_require_tree_directive
calls each_entry
, which we now know returns paths of all files from path passed to it. require_tree
means that all the files in the given path should be required
. You might have noticed, in the each_entry
block, that if a filename matches the current filename, then it is skipped with the next
statement. What is the purpose of this statement? We will get back to this later.
context.require_asset
is used to assign assets to the @_required_paths
array. @_required_paths
are paths of assets on which the current ProcessedAsset
depends. After process_directives
completes, process_source
is called, which captures the source of the current ProcessedAsset
by removing the directive processors.
After evaluate
of all processors completes, source of the current ProcessedAsset
is returned and saved to @source
in ProcessedAsset
.
Then, build_required_assets
from sprockets-2.2.2/lib/sprockets/processed_asset.rb
is called. In build_required_assets
you will see following code:
@required_assets = resolve_dependencies(environment, context._required_paths + [pathname.to_s]) -
resolve_dependencies(environment, context._stubbed_assets.to_a)
resolve_dependencies
is used to resolve the dependencies of the current ProcessedAsset
. Until this point, we only have paths of files that are required for our current ProcessedAsset
. For application.js
asset context._required_paths
might look like this.
{{GEMPATH}}/jquery-rails-3.0.1/vendor/assets/javascripts/jquery.js
{{GEMPATH}}/jquery-rails-3.0.1/vendor/assets/javascripts/jquery_ujs.js
{{GEMPATH}}
represents absolute path of gems directory which holds different gems. You have noticed that in above paths there is no application.js
which should be there because of require_tree
directive. Take a look at the parameters passed to resolve_dependencies
. We are explicitly appending the current pathname
to the list of context._required_paths
. Remember, from our last discussion that, in process_require_tree_directive
we skip the current file.
As we discussed, a BundledAsset
will have at least one ProcessedAsset
. By default, our files don’t include any processors to run, so if we want it’s path to be included in context._required_paths
we have to add a directive processor to each file which is obviously not ideal. We picked the other path. Instead of including the path of the file during directive processing, we included it explicitly to context._required_paths
while calling resolve_dependencies
from build_required_assets
. If there are some directive processors to run on the file, then we skip including that filename from there, because we know we are adding it manually during the call to resolve_dependencies
.
resolve_dependencies
loops on all paths that are passed to it and, if it matches the path of the current ProcessedAsset
, then it is appended to assets
array. If the filename doesn’t match, the elsif
statement is executed calling the find_asset
method.
To collect the source and run processors on any file, we need to create an instance of ProcessedAsset
. The jquery.js
file is one of these paths. Let’s see how it is recursively processed and appended to the assets
array for the ProcessedAsset
instance of application.js
. Starting with the elsif
in resolve_dependencies
you will see following line
asset = environment.find_asset(path, :bundle => false)
As previously mentioned, when we call find_asset
by setting bundle
to false
, it means that we are creating an instance of ProcessedAsset
. A ProcessedAsset
is created for jquery.js
in the elsif
of resolve_dependencies
.
Remember, the initializer of ProcessedAsset
calls context.evaluate
which performs different operations on the asset. jquery.js
is a JavaScript file and doesn’t contains any dependencies so no dependencies will be added to it by calling context.require_asset
. After the processors have been run on jquery.js
, resolve_dependencies
is called in the same way as it was called for application.js
a few steps earlier.
Since jquery.js
doesn’t depend upon any asset, context._required_paths
is empty and we are explicitly adding the current pathname and passing the array to resolve_depedencies
. There is only one dependency of jquery.js
, which is jquery.js
itself. The if
statement is true and the current ProcessedAsset
instance which we can refer as self
is added to the assets
array, which is assigned to @required_assets
in build_required_assets
. For those assets that don’t have any dependencies and are part of some BundledAsset
, @required_assets
points to one item which is the asset itself.
After creating an instance of ProcessedAsset
for jquery.js
and building it’s required assets, the control returns back to the elsif
in resolve_depedencies
which is building the required assets for application.js
. The ProcessedAsset
instance of jquery.js
is assigned to asset
in resolve_depedencies
. We then loop through required_assets
of jquery.js
and append an instance of ProcessedAsset
to the assets
array.
These steps are executed for jquery_ujs.js
and other paths too. After appending instances of ProcessedAsset
for all the paths of application.js
to the assets
array for application.js
, we return that array from resolve_dependencies
method which is assigned to instance variable @required_assets
of application.js
‘s ProcessedAsset
.
ProcessedAsset
will finish executing and will be returned to the initializer of BundledAsset
. The instance of ProcessedAsset
for application.js
is assigned to @processed_asset
and necessary assets are assigned to @required_assets
. We know that each ProcessedAsset
has it’s source so we start assigning the source of all ProcessedAsset
instances to @source
. to_a
returns required_assets
and we are getting a string representation of the ProcessedAsset
instance, meaning, the source. This behavior is defined here. When we have gathered sources of all ProcessedAsset
instances, we again run some processors
based on the content_type
of current file. For Javascript files, we run the Sprockets::Processor (js_compressor)
processor, passing it the source. After processing @source
and getting new @source
, the digest is calculated based on the source / contents.
When the constructor of BundledAsset
is finished, control eventually returns back to compile
in actionpack-3.2.15/lib/sprockets/static_compiler.rb
. This is where the magic started and we are back here with our completed BundledAsset
. Next steps are easy, as BundledAsset
is saved on disk in the same way a StaticAsset
is saved in file_name-digest.file_ext
format.
The whole process that I described above continues for each file in the paths
array.
Unanswered Questions Answered
You might be wondering why we have created the digest for StaticAsset
beforehand but the digest for BundledAsset
only after performing all that processing. The answer to this question is very simple. As discussed in part 1, there is no processing on StaticAssets
, so we just copy and paste them to the public/assets
directory with a digest based on its contents. The contents of a StaticAsset
such as image etc. will not changed between copying / pasting so we can calculate its digest beforehand. Whereas calculation of digest for BundledAsset
require some steps to be taken before the digest can be calculated.
We discussed in part 1 of this article that the write_to
method is used to create one asset, so how does Sprockets
creates two versions of the same file? When we do rake assets:precompile
the :all
task is called from actionpack-3.2.15/lib/sprockets/assets.rake
at lines 59-67. This task has two rake tasks, one is called to create the digested version of assets and the other is called to create the non-digested version of assets. You can check those two tasks on lines 69-71 and on lines 73-75. Both tasks call the internal_precompile
method with the digest
parameter being set accordingly. The non-digested call to internal_precompile
doesn’t take much time because everything has already been cached.
Final Thoughts
I have done my best to explain the coding internals of the Rails Asset pipelines. Reading code is an art and fun at the same time. By reading other developer’s code, weocan increase our technical skills. It was a real pleasure to read the awesome code of Sprockets
.
Frequently Asked Questions (FAQs) about Asset Precompilation
What is the purpose of asset precompilation in Rails?
Asset precompilation in Rails is a process that combines and minifies your CSS and JavaScript files to reduce the number of requests a browser needs to make, and the size of the files it needs to download. This process significantly improves the performance of your application by reducing the load time of your web pages.
How does the asset pipeline work in Rails?
The asset pipeline in Rails is a framework that concatenates and compresses or minifies JavaScript and CSS assets. It also adds the ability to write these assets in other languages such as CoffeeScript, SCSS, and ERB. The pipeline makes these files available in the public directory for production.
What is the role of Sprockets in asset precompilation?
Sprockets is a Ruby library that aids in the asset precompilation process. It concatenates and serves JavaScript, CoffeeScript, and CSS files, and also compiles these files from other languages. Sprockets provides a directive processor to manage dependencies of files.
How can I configure the asset pipeline settings?
You can configure the asset pipeline settings in the config/application.rb file and the config/environments/ files. These settings include the paths that Rails will look into while processing assets, whether to compress assets, and whether to fallback to assets pipeline if a precompiled asset is missed.
What is the difference between asset precompilation in development and production environments?
In the development environment, assets are served as separate files in their original form. This makes it easier to debug the code. In the production environment, all assets are concatenated and minified into one single file each for CSS and JavaScript to reduce the number of browser requests and improve performance.
How can I precompile additional assets?
You can add additional assets to be precompiled in the config/initializers/assets.rb file. You can specify individual files or entire directories to be precompiled.
What is the purpose of the manifest file in asset precompilation?
The manifest file, usually named application.js and application.css, is used to define which assets should be included in the asset precompilation process. It uses Sprockets directives to require the necessary files.
How can I troubleshoot issues with asset precompilation?
You can troubleshoot issues with asset precompilation by checking the Rails server log for any error messages. You can also use the rake assets:precompile task in the terminal to check for errors.
Can I use third-party libraries with the asset pipeline?
Yes, you can use third-party libraries with the asset pipeline. You can include them in the vendor/assets directory and require them in your manifest file.
How does asset fingerprinting work in Rails?
Asset fingerprinting is a technique used by Rails to ensure that an asset file is unique. It appends a hash to the filename of each asset, which changes every time the file is updated. This ensures that when the file is updated, the URL changes, forcing the browser to download the new file instead of serving the old one from cache.
Imran Latif is a Web Developer and Ruby on Rails and JavaScript enthusiast from Pakistan. He is a passionate programmer and always keeps on learning new tools and technologies. During his free time he watches tech videos and reads tech articles to increase his knowledge.