Building Your Tools With Thor

Thor is not new; first built as a rake and sake replacement, first commit is well over 4 years ago.

Jump ahead several years and Thor is part of the foundation of the new-generation rails generator, and very popular tools such as Bundler and Foreman.

Recently, @wykatz emerged a fantastic looking (and much deserved) Thor website, and although I’ve started doing Thor based projects over two years ago, I think its the right time to write about Thor itself.

Today, Thor can serve as a rake replacement, great generator building framework, and a general purpose CLI toolkit.

Why Thor?

Every once in a while, I tend to re-evaluate the things that form my “engineering toolbelt”. One of those is Thor, so I set out to undermine my assumptions and find something that can compete with it in terms of development ease of use.

mixlib-cli

This framework marks as ‘most popular’ in ruby library sites. However, as I’ve tried figuring out whats there to use, I came up short with things like developer convenience, interactive input (menus) and colored output.
I closed it up by reasoning that since the Chef knife tool depends on it, the high rank it gets is probably biased by the incredible number of implicit downloads.

trollop

Having over 250 dependent gems, trollop looks to be the most popular framework around. It deserves a discussion of its own, it is minimal and simple, and definitely is my pick when building one-off CLI tools.

slop

It is similar to trollop in simplicity, I’ve used it before and its also a great one for small CLI tools.

commander

Commander goes a step further and gives you a nice DSL to work with, it has a similar version for Node.js which is fun to use (by same author). Having built a tool with the Node.js version, I haven’t found it completely in advantage over simpler toolkits such as slop and trollop.

Finding the right match

So initially, I picked up trollop, and set out to build a tool that will have:

  • Clear separation of concern
  • Testability
  • Maintainability
  • Interactivity (input, menus)
  • Aesthetics (commands, colors)
  • State

The first hurdle was designing an aesthetic CLI (yes, CLI has the notion of design, for some people :). It so happens that a command would have parameters and options and that options are dasherized (--opt-name=<VAL>) and that whole concoction doesn’t really look well when you write the code to take in:

$ app command any number of variables, without --options.

For smarter input I had to depend on highline and for coloring I would take rainbow which is great Ruby citizenship, but really, I wanted to simplify even dependencies.

For example, working out how I’m going to test my tool with each of these (are they expecting to be tested in any special way? is there a preliminary setup?) wasn’t a good use of my time for such a small project in my opinion.

Faces of Thor

Although Thor is a single tool, in practice, it can be used in several ways (that I’ve so far had experience with):

Task runner

# file: test.thor
class Test < Thor
  desc "example", "an example task"
  def example
    puts "I'm a thor task!"
  end
end

When test.thor lives in your directory This will result in you being able to say:

thor test:example

Generator

An example tool you’d like to build can be a project sekeleton generator. Given a few command line parameters, It’ll generate a best-practice folder layout, with boilerplate code living inside generated files. The most obvious example would be the Rails generator with rails new.

Most (if not all) of the things you see that the Rails generator does are exposed to you through Thor. Here’s an example:

# ...snipped...
source_paths << File.expand_path("../../../templates",__FILE__)
source_paths << Dir.pwd


method_option :version_bumper, :type => :boolean, :aliases => "-b"
desc "init SOLUTION_NAME [TEMPLATE_LOCATION]", "Initialize an Albacore build"
def init(solution_name, template_location="default.alba")
  say "Creating Albacore build for #{solution_name}.sln"
  vars[:solution] = VS::Solution.new "#{solution_name}.sln"
  vars[:env] = VS::Environment.new

  template 'Rakefile'
  template 'Gemfile'

  apply template_location

  say "Done. Run 'bundle install' once to set up your dependencies."
end

And here is a typical template making use of the context we set up in the Thor task init:

# parts of a template file.
append_to_file 'Rakefile', <<-EOF, :verbose=>false

desc "Build release"
msbuild :build => :assemblyinfo do |msb|
  msb.properties :configuration => :Release
  msb.targets :Clean, :Build
  msb.solution = "#{vars[:solution].name}.sln"
end

desc "Build release"
msbuild :build => :assemblyinfo do |msb|
  msb.properties :configuration => :Release
  msb.targets :Clean, :Build
  msb.solution = "#{vars[:solution].name}.sln"
end

Although these are very exciting, I’d like to focus on Thor being a command line toolkit.

Thor: a CLI framework

The minimal CLI app:

require "thor"

class CLI < Thor
  desc "shake!", "Shakes the world"
  def shake!
    say "grrr"
  end
end

CLI.start

In a CLI app context, you would inherit from Thor, which makes available desc and other primitives for you to base your tool on.

Thor maps the CLI onto a class.

If you look closely, one of the ka-ching moments of mine with thor is that it maps:

$ <receiver> <action> <parameters>
$ myapp shake earth and mars

to the familiar:

myapp.shake('earth', 'and', 'mars')

while options are specified as:

#...snip...
  method_option :quake, :aliases => "-q", :desc => "As a quake"
  desc "shake!", "Shakes the world"
  def shake!
    say "grrr"
  end

and invoke as:

$ myapp shake earth and mars -q

Easily enough, in code, you would use the options variable, available at the body of the action.

Building a tool

Lets ditch out the idiotic earth-shaking example and use Logbook to work out the details. For any of the points below referencing CLI code, feel free to point a browser tab at the actual code to get more context, its around 100LOC.

Structure

I’d like to sketch out my CLI tools as such:

- logbook/
   - bin/
      - lg
   - lib
      - logbook.rb
      - logbook/
         - cli.rb
         - book.rb
   - spec

Where the main points are: a near-empty shim in bin/shake containing something along the lines of:

require 'logbook/cli'
Logbook::CLI.start

A single cli.rb module containing all of the command-line specific mechanics; and a set of modules (here book.rb) that contain the domain logic (in the domain-driven-design sense).

The CLI module should only bridge between the ‘ugly’ world of tty I/O and the beautiful, clean, world of our domain layer. The immediate gain is with testing, of course.

Interaction

To interact with the user, Thor provides several primitives such as say, ask, and print_table, here is an example usage:

choices = config[:books].to_a
choices = choices.map.with_index{ |a, i| [i+1, *a]}
print_table choices
selection = ask("Pick one:").to_i

Thor provides many more fun primitives, take a look here and here. That made me do away with highline and score a dependency not being attached to my code.

Flexibility

As mentioned before, Thor will auto-map the commandline into your class, so this allows me to take something like:

$ lg add its been a long, long ride

to simply:

#...
desc "add MEMORY", "add a new memory"
def add(*memory)
  text = memory.join '
#...

Aesthetics

I wanted to use color, so I’ve started out with rainbow mentioned before. But knowing Thor already does this with the Rails generator, I quickly found:

say text, :green

and the more low level shell abstraction

shell.set_color(text, nil, true)

To set a text to “bold” (or “bright”) without really modifying its color (which apparently was not supported in say).

State

Addmittedly, this is something Thor does not provide. I needed to save state about the user for the application, the same way the heroku gem saves your login information, for example.

This means storing a .dotfolder or a .dotfile in your ~/.config. While I could do this easily, I found user_config to be minimal and excellent.

Testing

Dealing with commandline apps I/O, this might (and rightly so) intuitively sound frightening; so it might not be very obvious how to tackle it. Although aruba might cover it, it is best to design your tool in a testable manner.

Given that we’ve encapsulated most of our logic outside of the Thor CLI class, it should be very easy to test. The book module hands out all of Logbook’s functionality.

In this specific case it is exposed by CLI, in another imaginary case it could be exposed to a Web service easily enough.

Given that we already know how to test modules separately, we’ll only need to test the CLI layer with logic mocked out, for interactions with the user and state.

Capturing IO

Lets simulate a command and assert on output:

it "should not add when no book" do
  out = capture_io{ Logbook::CLI.start %w{ add so long, and thanks for all the fish. } }.join ''
  out.must_match /No book is set./
end

minitest does great with providing us capture_io which fetches STDOUT for us to match on. As you might have noticed, invoking a Thor action is also pretty easy.

Testing input is a different matter. Here, I’ve chosen to mock Thor’s yes? primitive and precook it with values:

any_instance_of(Logbook::CLI) do |cli|
  mock(cli).yes?(anything){ true }
end

This will cause the flow to proceed as if the user typed in ‘yes’. I do the same where it is also a free input with ask.

Extra Tricks

Thor will spit warnings when tasks are declared without desc annotations. This most commonly will happen when you’re mocking things (in my case, I use rr).

In addition, many times a CLI apps deals with filesystems/files, a good solution for testing would be fakefs.

You can take a look at my monkey patching in logbook’s spec_helper

Conclusion

Thor gives you an almost brainless mapping from CLI to your code, arguably without using a specialized DSL, or requiring you to write additional boilerplate validation code after you’ve used a framework that takes care of arguments.

It will give you an awesome mileage for what you’re trying to do, and it is very worthwhile investing the time to specialize in everything it has to offer.