Asset Pipeline Internals

Almost a year ago, I wrote about build management for Javascript projects.

In a hindsight a year proved to be a ton of time on the client-side.

Most notably Grunt (which I only mentioned briefly) took off like a rocket, and in the same manner Yeoman - which I almost instantly considered a swiss army knife for doing my client-side only projects.

Yeoman though, which relies on Grunt, is going through some fundamental changes and looks like it is being re-arranged and re-planned for a while now.

For what it’s worth I do support the new Yeoman changes, but instead of waiting for it to crystalize I tought it is time to re-evaluate what’s out there today and see if Yeoman can be replaced altogather (the answer is ‘Yes’, keep reading :).

Ember.js and Sinatra

My current use case is fairly reasonable. Ember.js on the client side, and Sinatra providing a slim and fast API layer for the data.

Moving from Backbone to Ember as my client-side stack for a new project created a deep need for a solid asset pipeline with an effortless development experience, and looks like the most reasonable solutions right now are Grunt, rake-pipeline, and Sprockets.

Since I personally think dealing directly with Grunt build configuration is like dealing with an Ant build configuration in terms of flexibility (I remember a dark night just trying to do a certain file operation with Grunt) I hope you’ll forgive me for letting that one pass.

Client-Side Development

Things that I care about:

  • Magical asset compilation: don’t care if it’s Coffeescript, sass, or less, detect and compile it for me. I care about Coffeescript and Sass.
  • Dependency management: module dependency should be done properly.
  • Smooth integration: with existing server-side stack.
  • No friction: automate everything and live-reload.
  • Production ready: bundling, minification, manifest, gzip, etc.

Note that I don’t include testing frameworks and runners integration here. I find that this area is so segmented (everyone has their own preference), that it may earn its own separate set of articles.

Here is the file structure we’ll use (a Ruby project):

assets/
  javascripts/
  stylesheets/
  static/
lib/
  my_app/
  my_app.rb
spec/
config.ru

Note I’m taking the Rails naming for the assets folder (javascripts, stylesheets). It makes moving this thing to Rails fairly easy if I’d ever want to completely separate the API from the UI.

How To Trigger a Build?

Assuming all of your assets are layed out and your pipeline configured, I came to a conclusion that there are two ways you can typically model your asset compilation for development

  • File watcher and reactors - Guard does this well generically.
  • On demand (per request) - Sprockets and asset-pipeline can work this way each using their own middleware for it.

Using Sprockets

Sprockets is the default asset pipeline in Rails for a long while now and is why I was sure it will work for me, and it did.

A couple notable things about Sprockets is its flexible asset pipeline (it can handle a lot already out of the box) and the way it does dependency with a require directive, effectively injecting the body of code instead of the require as it processes your assets; this is arguably different (or simpler) than other javascript module dependencies techniques (see require.js as another example).

Sprockets with Guard

This way, everything will be Guard-driven. This means that changing code will trigger a series of processing steps that will drive everything that needs to be done in reaction to that change (asset compilation, browser reload, etc).

Set up

Let’s start with our Gemfile. We’ll add sprockets, sprockets-sass for better sass integration (we want compass), compass itself, coffee-script and finally uglifier for minification. CSS minification happens through the built-in Sprockets Sass compressor.

# ..somewhere in your Gemfile
gem "sprockets"
gem "sprockets-sass"
gem "compass"
gem "coffee-script"
gem "sass"
gem "uglifier"

group :development do
  gem "guard-sprockets2"
  gem "guard-livereload"
  gem "rack-livereload"
end

Now we’ll bootstrap our development environment and Guard itself with:

$ bundle

And create a Guardfile :

require 'my_project/app'

guard 'sprockets2', :sprockets => MyProject::App.sprockets do
  watch(%r{^assets/.+$})
  watch('app.rb')
end

guard 'livereload' do
  watch(%r{lib/.+\.rb})
  watch(%r{views/.+\.(erb)})
  watch(%r{public/.+\.(css|js|html)})
end

An important note to make here is that for livereload we are watching the generated files (which the sprockets2 watcher generates), and not the source files. This is to avoid a race condition where we modified a file, Sprockets is generating the file and we pushed a live-reload event to the browser prematurely before it was finished.

Requiring my_project/app is a result of the Ruby application structure I use: namespace/app, and Namespace::App which I consider a healthy project structure.

The sprockets2 guard, where we specified a Sprockets Environment which :sprockets => ..foo.. is one of the things that makes Sprockets so flexible. This will make Guard understand what to compile and how.

A bit more about Sprockets flexibility - since the Environment we just passed derives out of Base (here), which mixes in Server (here) implementing the Rack-ish call(env), the Environment also becomes a natural Rack middleware that we can use to compile assets on-demand (we’ll see that on the second part of this walkthrough).

Though in our case, guard-sprockets2 uses the Sprockets Environment not as a middleware but as a gateway to compilation that it needs in order to build assets.

Sprockets Configuration

Ideally, we need to configure Sprockets in a central place, and offer it to whomever may be interested (we already saw how Guard made use of it).

We’ll do this within the configuration section of our Sinatra app.

class MyProject::App <  Sinatra::Base
  set :root, File.expand_path('../../../', __FILE__)
  set :sprockets, Sprockets::Environment.new(root)

  configure do
    AssetHelpers.configure! sprockets, root
  end

  configure :production do
    sprockets.js_compressor = :uglifier
    sprockets.css_compressor = :scss
  end

  helpers do
    include AssetHelpers
  end
...

To use uglifier Sprockets will try to require things automatically the best way that it can. What it means for us is that it’s enough to put uglifier in our Gemfile.

The goal of AssetHelpers here is to set up the paths for Sprockets to be aware of and compute on-demand path to assets when we’ll need those in our views (when we’ll do <%= asset_path 'application.css' %>):

module AssetHelpers
  def asset_path(source)
    "/assets/" + settings.sprockets.find_asset(source).digest_path
  end

  def self.configure!(sprockets, root)
    %w{ stylesheets javascripts images }.each do |thing|
      sprockets.append_path(File.join(root, 'assets', thing))
    end
    sprockets.context_class.instance_eval do
      include AssetHelpers
    end
  end
end

That’s basically it. You now have a fully configured Sprockets environment that does Javascript, Coffeescript, Sass and CSS, Compass, and more.

To start, just say

$ bundle exec guard start

It will watch everything you do in /assets and rebuild your assets pushing them into /public. You’ll miss another piece of you workflow though: LiveReload.

Using LiveReload

In Ruby, we need to do a couple of things in order to set up LiveReload:

  1. Detect changes
  2. Inject LiveReload javascript (optional)

As you’ve already notice, we’ve used guard-livereload for change detection. It connects to the browser through a websocket and actually talks to it from your console when things change.

All that’s left is to inject the LiveReload script into every page. Easily done through a Rack middleware (rack-livereload):

configure :development do
  require 'rack-livereload'
  use Rack::LiveReload
end

That’s it. If you managed to survive the long read, you should now have a good under-the-hood grasp of how toolsets like Yeoman works.

If you rackup now, you’ll discover that you have a fairly nifty Yeoman-like environment.

Using Sprockets as Middleware

When you use Sprockets as a middleware, it will recompile everything on every request you make. On a first glance, it sounds scary. On a second glance - you only do this in development mode; you only compile and concatenate sources - no minifying or such intensive operations.

If you, like me, don’t think this such a waste given today’s CPUs - then you’ll want to drop the guard-sprockets2 magic and use a simpler solution: Sprockets middleware.

As mentioned before, Sprockets’ Environment is already a middleware itself which is why it’s dead easy:

Within your config.ru:

require 'my_project/app'

map '/assets' do
  run MyProject::App.sprockets
end

run MyProject::App

Since we’ve now inverted the pipeline model (we will pull the changes by refreshing the browser), Guard will need to watch different files.

Hello again, Guardfile:

# we just killed off the `sprockets2` section

guard 'livereload' do
  # watch(%r{public/.+\.(css|js|html)}) -- dropping this
  watch(%r{^assets/.+$})  # in favor of this
  ...
end

And we can also remove guard-sprockets2 from our Gemfile.

That’s it - now Guard is watching for file changes to support LiveReload, and Sprockets is compiling and serving assets on demand. To me this feels much cleaner, and is also very similar to how Rails does it (minus some minor differences).

Now that the setup is almost done, you can take a look at how Rails (this case 4.x) chose to configure and provide for Sprockets.

The assets:precompile Trick

Everything we did now was to support a development workflow. If we moved out to using Sprockets as a middleware - how would we move a properly built, minified package to our production server?

Let’s create a rake task named assets:precompile similar to the Rails one in name, so that you can use the same infrastructure for this application stack as well.

If you’re on Heroku though, you’ll have to run a manual heroku rake for this one, but if you want to feel like a boss you can fake it and make the Heroku buildpack think this is a Rails app, and run the assets:precompile task automatically (I’ll leave this to you, but I’ve linked a good hint of how to do that).

All that is left is to fake out this task with Sprockets, in your Rakefile:

require 'my_project/app'
require 'rake/sprocketstask' # <-- important
require 'logger'

namespace :assets do
  Rake::SprocketsTask.new(:precompile) do |t|
    t.environment = MyProject::App.sprockets
    t.output      = "#{File.dirname(__FILE__)}/public/assets"

    # include sprocket manifests (main application files with 'require's)
    t.assets = %w{ application.js
                   application.css }
    # include anything in `/assets` that is not a manifest (images)
    t.assets << lambda do |path, filename|
      filename =~ /\/assets/ && !%w(.js .css).include?(File.extname(path))
    end

    # setting debug is good for CI logs
    t.logger = Logger.new($stdout)
    t.logger.level = Logger::DEBUG
  end
end

Most of the heavy lifting was done by just our Sprockets configuration.

This completes our picture for Sprockets. We’ll now study an arguably more “elegant” asset pipeline called rake-pipeline.

rake-pipeline

Why another pipeline? well, in my opinion rake-pipeline is more hackable than Sprockets. This makes it very inviting to use in less typical projects, and even not necessarily web projects.

The developer API is pure awesome. Here is a quick example of defining what you want to do with your assets:

match "app/**/*.coffee" do
  coffee_script
end

match "app/**/*.js" do
  uglify if production?
  concat "application.js"
end

This is a real pipeline defined through a DSL; i.e. the output of the coffee_script processor goes straight into the next step in the pipeline with the help of a good input/output abstraction.

Note that production? here is just my own addition (just checking an environment variable). You can mix and match with your own to enrich the DSL.

Setting up rake-pipeline

While rake-pipeline provides the foundation, rake-pipeline-web provides the extra pipeline steps, or filters, that we actually require on a typical web app. You also get a CLI, rakep which we’ll look at later.

As a convenience, here is our app file layout:

assets/
  javascripts/
  stylesheets/
  static/
lib/
  my_app/
  my_app.rb
spec/
config.ru
Assetfile

If Guard and dancing around configuration in our Sinatra app and Guardfile were the main focus point of the Sprockets-based setup, in this case, Assetfile will be it.

You can use this file verbatim as a starting point and tweak it to your own needs.

Here’s our Gemfile

gem 'sinatra'

group :development do
  gem 'rack-livereload'
  gem 'guard-livereload'
  gem 'rake-pipeline'
  gem 'rake-pipeline-web-filters'
  gem 'coffee-script'
  gem 'yui-compressor'
  gem 'uglifier'
  gem 'sass'
  gem 'compass'
end

This time, there’s plenty more I don’t need in production mode here. This is due to the fact that I will not compute asset paths in the views but use a predefined /application.css or /application.js - it might be less flexible but is a very lean solution.

And to rig everything up in development we configure this (LiveReload configuration lifted off the Sprockets example):

configure :development do
  require 'rake-pipeline'
  require 'rake-pipeline/middleware'
  use Rake::Pipeline::Middleware, 'Assetfile'
  require 'rack-livereload'
  use Rack::LiveReload
end

Asset Compilation

For asset compilation, you’ll get a middleware that compiles on demand, like with the Sprockets middleware.

Everyhing that you outline in your Assetfile will be used both in the middleware and the CLI (rakep), as well as filters you can write on the spot. Here’s a custom filter example:

class VersionFilter < Filter
  def generate_output(inputs, output)
    version = File.read("VERSION").strip
    inputs.each do |input|
      result = input.read
      result.gsub!(/VERSION: '.*'/, "VERSION: '#{version}'")
      output.write(result)
    end
  end
end

You can take a look at a few more filters here. In total, you have the stock filters and the rake-pipeline-web filters at your disposal.

Module Dependencies

You might have noticed that once we moved off Sprockets, we have a spot to fill for Sprockets’ require dependency management.

Typically on rake-pipeline This is done through a small and nifty Javascript library called minispade.

What you typically do is use require naively in your Javascript or Coffeescript code, and then run a rake-pipeline filter which swaps it out for the real thing.

require('model/playlist')

And now, tuck this in your Assetfile

match "app/**/*.js" do
  minispade :rewrite_requires => true, :module_id_generator => proc { |input|
      input.path.sub(/^app\//, '').sub(/\.js$/, '')
  }
  uglify if production?
  concat "application.js"
end

Note that it is assumed that your client-side application code lives under javascripts/app.

You’ll also need to make sure minispade.js lives on your /public folder untouched. It’s an extremely small solution for dependency management and you can just put it in a /static subfolder of your assets.

input "assets/static" do
  match "**/*" do
    # The block we pass to `concat` lets us adjust the output path
    # of any files it matches. Here we take each input and strip
    # off the `static/` prefix, so `app/static/index.html` ends up
    # in `public/index.html`.
    concat do |input|
      input.sub(/static\//, '')
    end
  end
end

With the above directive, we just told rake-pipeline not to touch such static content and just place it in the output destination. Just place this directive at the bottom of your Assetfile and it’ll be enough.

That’s it. As mentioned before this one feels a bit more hackable, a bit simpler.

rakep and Production

To build for production all you have to do is use the assets:precompile task and run the rakep command which you get for free with rake-pipeline.

Here’s an example Rakefile for this:

namespace :assets do
  task :precompile do
    `bundle exec rakep`
    puts 'compiled.'
  end
end

Wrapup

Hopefully, this urged you to dip into the implementation details of these tools and libraries. As a side-effect you now have a kick-ass asset pipeline implementation that you can carry from project to project, and can freely hack on without subjecting yourself to a one-size-match-all solution.

Asset compilation pipelines, for me, proved to be a good study case of how to write code for libraries that can integrate well with other tools and libraries, expose a good DSL, and be very modular and flexible.