Jump ahead several years and Thor is part of the foundation of the new-generation rails generator, and very popular tools such as Bundler and Foreman.

Recently, @wykatz emerged a fantastic looking (and much deserved) Thor website, and although I’ve started doing Thor based projects over two years ago, I think its the right time to write about Thor itself.

Today, Thor can serve as a rake replacement, great generator building framework, and a general purpose CLI toolkit.

Why Thor?

Every once in a while, I tend to re-evaluate the things that form my “engineering toolbelt”. One of those is Thor, so I set out to undermine my assumptions and find something that can compete with it in terms of development ease of use.

mixlib-cli

This framework marks as ‘most popular’ in ruby library sites. However, as I’ve tried figuring out whats there to use, I came up short with things like developer convenience, interactive input (menus) and colored output.
I closed it up by reasoning that since the Chef knife tool depends on it, the high rank it gets is probably biased by the incredible number of implicit downloads.

trollop

Having over 250 dependent gems, trollop looks to be the most popular framework around. It deserves a discussion of its own, it is minimal and simple, and definitely is my pick when building one-off CLI tools.

slop

It is similar to trollop in simplicity, I’ve used it before and its also a great one for small CLI tools.

commander

Commander goes a step further and gives you a nice DSL to work with, it has a similar version for Node.js which is fun to use (by same author). Having built a tool with the Node.js version, I haven’t found it completely in advantage over simpler toolkits such as slop and trollop.

Finding the right match

So initially, I picked up trollop, and set out to build a tool that will have:

• Clear separation of concern
• Testability
• Maintainability
• Interactivity (input, menus)
• Aesthetics (commands, colors)
• State

The first hurdle was designing an aesthetic CLI (yes, CLI has the notion of design, for some people :). It so happens that a command would have parameters and options and that options are dasherized (--opt-name=<VAL>) and that whole concoction doesn’t really look well when you write the code to take in:

1

$ app command any number of variables, without --options.

For smarter input I had to depend on highline and for coloring I would take rainbow which is great Ruby citizenship, but really, I wanted to simplify even dependencies.

For example, working out how I’m going to test my tool with each of these (are they expecting to be tested in any special way? is there a preliminary setup?) wasn’t a good use of my time for such a small project in my opinion.

Faces of Thor

Although Thor is a single tool, in practice, it can be used in several ways (that I’ve so far had experience with):

Task runner

1
2
3
4
5
6
7

# file: test.thor
class Test < Thor
  desc "example", "an example task"
  def example
    puts "I'm a thor task!"
  end
end

When test.thor lives in your directory This will result in you being able to say:

1

thor test:example

Generator

An example tool you’d like to build can be a project sekeleton generator. Given a few command line parameters, It’ll generate a best-practice folder layout, with boilerplate code living inside generated files. The most obvious example would be the Rails generator with rails new.

Most (if not all) of the things you see that the Rails generator does are exposed to you through Thor. Here’s an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# ...snipped...
source_paths << File.expand_path("../../../templates",__FILE__)
source_paths << Dir.pwd


method_option :version_bumper, :type => :boolean, :aliases => "-b"
desc "init SOLUTION_NAME [TEMPLATE_LOCATION]", "Initialize an Albacore build"
def init(solution_name, template_location="default.alba")
  say "Creating Albacore build for #{solution_name}.sln"
  vars[:solution] = VS::Solution.new "#{solution_name}.sln"
  vars[:env] = VS::Environment.new

  template 'Rakefile'
  template 'Gemfile'

  apply template_location

  say "Done. Run 'bundle install' once to set up your dependencies."
end

And here is a typical template making use of the context we set up in the Thor task init:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# parts of a template file.
append_to_file 'Rakefile', <<-EOF, :verbose=>false

desc "Build release"
msbuild :build => :assemblyinfo do |msb|
  msb.properties :configuration => :Release
  msb.targets :Clean, :Build
  msb.solution = "#{vars[:solution].name}.sln"
end

desc "Build release"
msbuild :build => :assemblyinfo do |msb|
  msb.properties :configuration => :Release
  msb.targets :Clean, :Build
  msb.solution = "#{vars[:solution].name}.sln"
end

Although these are very exciting, I’d like to focus on Thor being a command line toolkit.

Thor: a CLI framework

The minimal CLI app:

1
2
3
4
5
6
7
8
9
10

require "thor"

class CLI < Thor
  desc "shake!", "Shakes the world"
  def shake!
    say "grrr"
  end
end

CLI.start

In a CLI app context, you would inherit from Thor, which makes available desc and other primitives for you to base your tool on.

Thor maps the CLI onto a class.

If you look closely, one of the ka-ching moments of mine with thor is that it maps:

1
2

$ <receiver> <action> <parameters>
$ myapp shake earth and mars

to the familiar:

1

myapp.shake('earth', 'and', 'mars')

while options are specified as:

1
2
3
4
5
6

#...snip...
  method_option :quake, :aliases => "-q", :desc => "As a quake"
  desc "shake!", "Shakes the world"
  def shake!
    say "grrr"
  end

and invoke as:

1

$ myapp shake earth and mars -q

Easily enough, in code, you would use the options variable, available at the body of the action.

Building a tool

Lets ditch out the idiotic earth-shaking example and use Logbook to work out the details. For any of the points below referencing CLI code, feel free to point a browser tab at the actual code to get more context, its around 100LOC.

Structure

I’d like to sketch out my CLI tools as such:

1
2
3
4
5
6
7
8
9

- logbook/
   - bin/
      - lg
   - lib
      - logbook.rb
      - logbook/
         - cli.rb
         - book.rb
   - spec

Where the main points are: a near-empty shim in bin/shake containing something along the lines of:

1
2

require 'logbook/cli'
Logbook::CLI.start

A single cli.rb module containing all of the command-line specific mechanics; and a set of modules (here book.rb) that contain the domain logic (in the domain-driven-design sense).

The CLI module should only bridge between the ‘ugly’ world of tty I/O and the beautiful, clean, world of our domain layer. The immediate gain is with testing, of course.

Interaction

To interact with the user, Thor provides several primitives such as say, ask, and print_table, here is an example usage:

1
2
3
4

choices = config[:books].to_a
choices = choices.map.with_index{ |a, i| [i+1, *a]}
print_table choices
selection = ask("Pick one:").to_i

Thor provides many more fun primitives, take a look here and here. That made me do away with highline and score a dependency not being attached to my code.

Flexibility

As mentioned before, Thor will auto-map the commandline into your class, so this allows me to take something like:

1

$ lg add its been a long, long ride

to simply:

1
2
3
4
5

#...
desc "add MEMORY", "add a new memory"
def add(*memory)
  text = memory.join '
#...

Aesthetics

I wanted to use color, so I’ve started out with rainbow mentioned before. But knowing Thor already does this with the Rails generator, I quickly found:

1

say text, :green

and the more low level shell abstraction

1

shell.set_color(text, nil, true)

To set a text to “bold” (or “bright”) without really modifying its color (which apparently was not supported in say).

State

Addmittedly, this is something Thor does not provide. I needed to save state about the user for the application, the same way the heroku gem saves your login information, for example.

This means storing a .dotfolder or a .dotfile in your ~/.config. While I could do this easily, I found user_config to be minimal and excellent.

Testing

Dealing with commandline apps I/O, this might (and rightly so) intuitively sound frightening; so it might not be very obvious how to tackle it. Although aruba might cover it, it is best to design your tool in a testable manner.

Given that we’ve encapsulated most of our logic outside of the Thor CLI class, it should be very easy to test. The book module hands out all of Logbook’s functionality.

In this specific case it is exposed by CLI, in another imaginary case it could be exposed to a Web service easily enough.

Given that we already know how to test modules separately, we’ll only need to test the CLI layer with logic mocked out, for interactions with the user and state.

Capturing IO

Lets simulate a command and assert on output:

1
2
3
4

it "should not add when no book" do
  out = capture_io{ Logbook::CLI.start %w{ add so long, and thanks for all the fish. } }.join ''
  out.must_match /No book is set./
end

minitest does great with providing us capture_io which fetches STDOUT for us to match on. As you might have noticed, invoking a Thor action is also pretty easy.

Testing input is a different matter. Here, I’ve chosen to mock Thor’s yes? primitive and precook it with values:

1
2
3

any_instance_of(Logbook::CLI) do |cli|
  mock(cli).yes?(anything){ true }
end

This will cause the flow to proceed as if the user typed in ‘yes’. I do the same where it is also a free input with ask.

Extra Tricks

Thor will spit warnings when tasks are declared without desc annotations. This most commonly will happen when you’re mocking things (in my case, I use rr).

In addition, many times a CLI apps deals with filesystems/files, a good solution for testing would be fakefs.

You can take a look at my monkey patching in logbook’s spec_helper

Conclusion

Thor gives you an almost brainless mapping from CLI to your code, arguably without using a specialized DSL, or requiring you to write additional boilerplate validation code after you’ve used a framework that takes care of arguments.

It will give you an awesome mileage for what you’re trying to do, and it is very worthwhile investing the time to specialize in everything it has to offer.

I’ve been waiting for this to go public since Matz’ early announcements of him being working on it. This is very exciting.

Installing

  $ git clone https://github.com/mruby/mruby
  $ make

Compilation is a fantastic error-less breeze, around 20 seconds.

Hello mruby

Lets see how this thing should work.

1
2
3
4
5
6

$ cd bin
$ cat > hello.rb
puts "hello mruby!"
^D
$ ./mruby hello.rb
hello mruby!

Done. Quick sanity check for mruby, very cool. Lets try something more complicated.

1
2
3
4
5
6
7

require 'benchmark'
require 'digest/md5'
puts Benchmark.measure {
  puts "goo"
}
puts  Digest::MD5.hexdigest "hello mruby!"
puts "hello mruby!"

No rubygems so no errors while requiring any of these, and we can confirm (unsurprisingly) that we can’t treat it like a normal ruby. So we get:

1
2

$ ./mruby hello.rb
#<NameError: uninitialized constant Benchmark>

Yup, no benchmark, so lets use time. I’ve ported a benchmark sample from the computer language shootout, removing mruby incompatible things and bridging the gap between mruby and ruby so we can just toggle each and run.

You can take a look at it here: https://gist.github.com/2438540.

Its Not Faster

I had 0-knowledge about what mruby should do here. Since I’m familiar with Lua’s speed and mruby aiming for that space, I was hoping for a surprise.

A quick run shows that the mruby interpreter is considerably slower, and giving off different results, I don’t care about that for now:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  $ time ruby n-body.rb
  -0.16854649420142517
  -0.1685643614266391
  ruby n-body.rb  1.68s user 0.05s system 84% cpu 2.044 total

  $ time ./mruby n-body.rb
  -5.99182358773761e+154
  NaN
  ./mruby n-body.rb  5.32s user 0.04s system 90% cpu 5.900 total


  $ time ./mruby -b n-body.mrb
  -5.99182358773761e+154
  NaN
  ./mruby -b n-body.mrb  5.12s user 0.04s system 90% cpu 5.713 total

In mruby you can compile your code to bytecode. I’ve compiled it using mrbc which also lives in bin into n-body.mrb and ran it above using the -b flag: mruby -b.

Just a funny fact, if you dump the compiled content, for the magic number, you get:

1

RITE0009000000090000MATZ

The Real Fun Is Here

Lets look at what the executable mruby is, this is quite cool:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

if (args.mrbfile) {
  n = mrb_load_irep(mrb, args.rfp);
}
else {
  p = mrb_parse_file(mrb, args.rfp);
  if (!p || !p->tree || p->nerr) {
    cleanup(&args);
    return -1;
  }

  if (args.verbose)
    parser_dump(mrb, p->tree, 0);

  n = mrb_generate_code(mrb, p->tree);
  mrb_pool_close(p->pool);
}

if (n >= 0) {
  if (args.verbose)
    codedump_all(mrb, n);

  if (!args.check_syntax) {
    mrb_run(mrb, mrb_proc_new(mrb, mrb->irep[n]), mrb_nil_value());
    if (mrb->exc) {
mrb_funcall(mrb, mrb_nil_value(), "p", 1, mrb_obj_value(mrb->exc));
    }
  }
}

This shows why mruby is so awesome; you can embed it – and the perfect example for it is the mruby executable itself.

You might or might not have heard about the 501 manifesto. A 501 programmer is the one that runs out of the office at 5:01, regardless of any importance of him staying and keep his responsibilities aligned.

While I do believe in family values and socializing, I think that part of the manifesto is pretty generic, and the addendum of pitying open source or programmers who love what they’re doing is pretty insulting. We (programmers) are a culture now, and usually the passionate ones are the ones that are being mocked.

We’re all n00bs

Often, the industry and academia tries to push it into familiar engineering industry; carpentry, architecture, construction.

Experienced programmers know that in software, there is no material waste like in carpentry.

And unlike the unexpected nature and world-changing-beneath-your-feet dynamicity of software – using Gantt charts in construction relies on the fact that each step is known and predictable, measurable.

In addition, seasoned programmers know that architectural design patterns are not completely the same as software design patterns, and although being derived from the seminal works of Christofer Alexander, software design patterns must be used with care.

We’re in a young profession, relative to other traditional professions, such as carpentry. Its an engineering trade but its also a craft, its also creative; its a prose. As an engineer, to me, when I try to frame it into something academically correct I choose to go with undefined.

I’m happy that it takes all those forms together for me. Software is now omnipresent, it is consumed everywhere and hosted anywhere, its powering everything.

It cultivates.

On 501s

Before the 501 manifesto was even out, over lunch, I mentioned that when I leave work, it’s not over for me.

Mostly, I will continue sitting in front a computer and be productive (read: code) as long as I can keep my eyelids open.

I love what I’m doing, hell, I’m passionate about it; and I love being part of a culture of the same kind of people.

That I manage myself and I keep myself efficient and productive after work hours is a mantra for me. I choose to not watch cable-TV, not do Facebook, and not do instant-messaging.

On most occasions I don’t do phones. I choose to communicate and work asynchronously. I see the person on the other side of the phone call, or IM demanding my attention immediately as rude. You’ve got SMS, you’ve got EMail, and I’ll be pretty quick at that - so use them.

Over lunch, the response from the developer was to mock my culture. “Pfft. I could never even look at a screen after work hours”. To which I replied that he should be certain he’s in the correct profession.

It’s a Culture

Well, 501’ers, programming is a culture.

The first bullet points of the manifesto are pretty generic. There isn’t anything special about software as a profession in them. I could apply them to fishing if I wanted to.

My blood started jittering when it got to this part:

If you: Write a technical blog Contribute to open source projects Attend user groups in your spare time Mostly only read books about coding and productivity Push to GitHub while sitting on the toilet Are committed to maximum awesomeness at all times, or would have us believe it …we respect you for it. There’s probably some pity in there too, but honestly, it’s mostly respect.

The pitying part is supposed to be attenuated by “respect”, but I think including that part cancels all respect, and there is a slight hint of cynicism at “have us believe it”. Doesn’t matter if you’re “honest” about it or not.

Programming is a culture of problem solvers, efficient and creative people, who want to make a change in the world because they realize that nowadays, software is omnipresent and that from this property alone, change CAN happen.

And when I’ll have a kid I’ll feel so urgently to return to, I’ll probably be busy programming his/her toys.

If my job is inspiring me, I’ll stay late at work. If I want to make good use of my life, I’ll stay up late coding, and I’ll make sure I contribute something for others to use. Stamp it “open source”, I don’t care. I just like this culture.

So I pity you for participating in my culture and trying to call me out of it.

Oh, and I would fix that crappy HTML.

tl;dr

In a distributed storage environment you may no longer generate IDs in traditional ways. You need a fast service that generates IDs, which, given constraints such as bit size, isn’t obvious to build.

Twitter’s Snowflake does it following a set of their own constraints, but given more lenient constraints there are other ways such as compositing an ID with a time component and random jitter. There are also important risks and guarantees to take into consideration.

Parting Ways with Auto Increments

You want to generate IDs when you’re working in a sharded setup; in which non of the shards can take responsibility for generating IDs on its own.

Further, you might want an independent service or component to be responsible for ID generation, in a complex transactional usecases, where multiple systems might be affected.

In other cases you’ll want to perform intricate tracking which requires generation at the client side, or far from the server.

ID Generation Constraints

ID generation may seem trivial. Just UUID.new and you’re done, right? nope.
Lets take a survey of some important constraints, that make this thing quite complex.

1. Must have reasonable amount of bits ( < 64)
2. Fast incremental ID generation, oredered by date
3. Ability to be deployed to multiple machines
4. No common storage

When constraint (1) is relieved, we can go with TIME+UUID id; this will be great - given a reliable time service, ofcourse.

When constraint (4) is relieved, we can use a common storage for synchronization. A solution based on Redis is benchmarked later in the next post.

But hey! If you’re kinda lost, you might want to become familiar with what is used in the wild, like Twitter’s Snowflake, Flickr’s ticketing, Boundary’s Flake.

I’m going to ease constraint (1) in this post, which Boundary also did, and which Twitter enforced in Snowflake due to internal reasons as claimed by them.

Challenges

Lets take a look at a handful of challenges that aren’t initially obvious.

1. Time sync within a single machine. NTP can manipulate clock on its own, which is a Good Thing, but for this case may be dangerous
2. Time sync cross machines
3. Observation: when running on multiple cores (multiple workers/threads), same challenges exist as with multi-machine
4. Fast generation, at least 1000 IDs/sec
5. IDs must be strongly unique (no margin) cross machines with no synchronization mechanism other than NTP (no storage)
6. IDs should be time-(k)-sortable

Risks

• When single machine – SPOF
• When multiple machines – duplicate IDs.

How Snowflake-like Services Work

Here is how ID composition works in principle (number of bits and algorithm simplified for brevity).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

millis = get_system_millis

millis = millis * Math.pow(2, 12);
# current ID: 
# "[111011100000001100111110011010001111][000000000000]"  
# -> time bits + 12 free bits. 48 total bits.
var id2 = id * Math.pow(2, 8);
# current ID: 
# "[111011100000001100111110011010001111][0011][00000000]"  
# ->  8 free bits. 48 total bits. our node-id is 0011 (3)
#     4 bits allow us to have up to 16 node values.
var uid = millis + id2 + counter;
# current ID:
# "[111011100000001100111110011010001111][0011][00000110]" 
# ->  0 free bits. 48 total bits. our node-id is 0011 (3)
#     the 8 bits allow us to keep 256 values for clock
#     resolution.

Each generating node can be allocated a ‘node-id’ from the 4-bit space. This ID can be given ‘by hand’ per process, on its startup.
In Twitter’s Snowflake, for example, Zookeeper is used to manage the distributed nature of correct node-id allocation.

Clock Resolution

Something that isn’t quite trivial to bump into is this issue: since system clock is millis (may be microsecs), it is possible that in a high performance server, we’ll get several requests within a single clock. For this purpose, we keep a counter which we increase “manually”.

The next problem is being able to detecting multiple requests coming in on a single clock. That is done by keeping the last clock value that the last request bumped into. If the next clock value is the same, then we’re in a multiple-requests-per-clock scenario we should then increase the internal counters. Sadly, we can only have 8-bits for the counter which makes only 256 increments; if we bump into more requests per clock - we’re going to cancel the request.

System Performance

The above issue (Clock Resolution) clearly outlines that we can only do X requests per clock. From this we can extract that we can only do K * X requests per seconds; where K is number of clocks/sec. It is important to see that this has nothing to do with the application’s ability to optimize code / serve requests better.

Multi Core / Multi Instance Service

We keep an “id” per node; this is true for multicore systems aswell. This id field discriminates between facilities generating IDs in order to keep IDs unique, by keeping generated ID spaces physically (bits) apart.

Picking a Solutions & Benchmarks

I’ll publish another post within a few days with reasoning for solutions (code), benchmarks, and notes about k-sortability.

tl;dr

Find a tool that can compile coffeescript, take javascripts and vendor javascript libs, join them preserving order, and minify them into one file good for release. If that tool can do CSS/stylus/sass all the better. I pick sprockets eventually and show how you can tailor it to your needs.

Its also been quite a journey and I’ve not to make it tiresome and include all of the details. My best offer if you have questions is just follow me and tweet me up.

Follow @jondot

Custom made concoctions

This is the quickest way you can have at setting up a build for your code, and sometimes it is just enough. However, you’ll need to either copy-paste this each time, or extract it into a boilerplate/tool.

To do this, you can use plain cat and minify command line libraries with a Makefile, use closure compiler with ruby, or use uglifyjs with node Cakefiles.

Dedicated tooling

First some words about CommonJS and its module system. From the spec:

This specification addresses how modules should be written in order to be interoperable among a class of module systems that can be both client and server side, secure or insecure, implemented today or supported by future systems with syntax extensions. These modules are offered privacy of their top scope, facility for importing singleton objects from other modules, and exporting their own API. By implication, this specification defines the minimum features that a module system must provide in order to support interoperable modules.

stitch

With it, you can use CommonJS in your browser. Find it here: sstephenson/stitch. If you want to be using CommonJS out of Node’s environment (a good thing), it presents a problem - the browser doesn’t really understand CommonJS.

This is why we need to have an HTTP server serving on-the-fly content from stitch, which chews up the CommonJS export directive into a browser-compatible structure, or a CLI tool that watches for changes and rebuilds you files on demand. Stitch doesn’t come with those, but it does come with a couple great examples of how to easily make them on your own on the README.

hem

Quite a new tool, looks like hem was built to provide great build experience for Spine.JS. It too supports CommonJS (via stitch), which is my preferred way to do dependencies with javascript.

I was able to customize it to my own needs (I’m not doing Spine.JS at the moment) and make it work pretty quickly; however, I couldn’t help but feel that I was abusing it, just because the terminology in the config file is web app specific (I have no /public folder like a web app would have, nor is what I’m building considered a web app).

sprockets

The familiar pipeline from Rails. I’m already using it in my Rails projects so I have nothing really stopping me except a couple of things.

For one, it doesn’t use CommonJS (but then again I use that on Rails so..), and secondly, with the lack of Rails pipeline glue I have to manually rebuild the package every time I make an update via the sprockets commandline.

While this is not too bad, it becomes a problem when I don’t even have (or want) a server-side in my project.

PS: There is also a sprockets inspired Node based tool called interleave.

Brewer.js

This library touched my radar a night after I’ve published this article. Looks like it was created 2-3 days before that too.

It looks like a great solution, I may have picked it if that was the first solution that occurred to me. It works quite similar to sprockets and interleave, and injects dependencies (not CommonJS).

Lumbar

A great build management tool from walmartlabs. Feels like it packs a ton of punch, but may be too overkill for my needs.

It doesn’t come OOB with coffeescript support (it does with stylus support). Regarding CommonJS, it is not using stitch but it does provide their own version for it. If my project was intensively complex in terms of dependencies and build targeting, I’d use that, and I’d take the time to build the appropriate CoffeeScript plugin.

brunch

http://brunch.io. Since I’ve used it before, at first, this was my go-to solution. It offers almost zero developer friction, nice guidelines and best practices, and great preprocessor support for any language that I needed.

However, currently minification is utterly broken, and looks like the tool itself undergoes some kind of a rewrite. If you’re willing to make minification an external step, this may be an excellent choice.

Note: I kinda stayed away from tools such as Jammit, Squishit (for you .Net guys), and Juicer. They do what I need only to a degree.

What did i choose?

Basically we can narrow it down to one question.

Do we want to use CommonJS?

If so, then stitch, hem and Lumbar are in that space (brunch can use require.js).

If not, then sprockets or a custom build file will do the job just fine. Take note that to use sprockets easily (read: without coding too much), you can use the sprockets command line tool (which, given its history was taken out and pulled back in), and some kind of watch detection (guard-sprockets might be nice for you).

Eventually I settled on sprockets.

Here’s how you could set it up, too. Needless to say you need Ruby for this.

With an example directory layout such as this:

/
 - /app
    - index.js
    - your.coffee
    - your.js
 - /vendor
    /js
      - jquery.js
      - backbone.js
      - underscore.js
 - /build
    - output.lives.here.js
 - Gemfile
 - Guardfile

You should set up a Gemfile like so

1
2
3
4
5
6

source :gemcutter

gem 'sprockets'
gem 'guard-sprockets'
gem 'coffee-script'
gem 'uglifier'

Follow with bundle install and your environment is basically ready. We’ll want to do a guard init, and guard init sprockets.
Make sure to preprend with bundle exec when needed.

Lets edit our Guardfile to make sure we build the right bundle. In this case my code lives in /app and my vendored things in /vendor. I called my main javascript file (through which i require others) index.js

1
2
3

guard 'sprockets', :destination => "build", :asset_paths => ['app', 'vendor'] do
  watch (%r{^app/.*}){ |m| "app/index.js" }
end

We will watch everything (via the app/.* regex) and let sprockets build our root file (app/index.js). Again, the root file includes our sprockets require directives (see sprockets)

You’re Up!

If you go bundle exec guard start, you should have a compiled script ready every time you make changes to files in /app in your /build folder.

One last thing

The code that was generated is not minified. Well, turnes out guard-sprockets isn’t supporting that, so I’ve added support.

For this, you will have to modify your Gemfile to take my own fork of guard-sprockets (for more detail, see https://github.com/sstephenson/sprockets/issues/279):

1

gem 'guard-sprockets', :git => "git://github.com/jondot/guard-sprockets.git"

Now we finish up by updating our Guardfile, so that it knows we want minified content:

1
2
3

guard 'sprockets', :destination => "build", :asset_paths => ['app', 'vendor'], minify => true do
  watch (%r{^app/.*}){ |m| "app/index.js" }
end

Finito

Run bundle exec guard start again, make a modification, and you’ll see your code compiled and minified!.

In ruby, when you want to run and redirect process output, you have many options.
Here are some:

1
2
3
4
5
6
7
8

(1) result = `command`
# stdout, stderr if redirected. Buffered until completion.

(2) pipe = IO.popen("command", "r")
# stdout, stderr if redirected. Get chunks while running.

(3) Open3.popen3("command")
# all 3 streams (in, out, err). Get chunks while running.

Streaming with Sinatra

From 1.3.0, sinatra supports streaming. It gets really impressive on an evented server such as Thin, considering that such behavior was reserved either to dedicated frameworks such as cramp and goliath, or even nodejs.

Spawning and Streaming

A typical CI server would spawn processes and listen in on their output. Lets see how this might happen easily.

We’ll pick option (2) for redirecting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

require 'rubygems'
require 'sinatra'

cmd = 'ruby bomb.rb'

get '/' do
  stream do |outp|
    IO.popen(cmd, 'r') do |io|
      while line=io.gets
        puts line
        outp << line
      end
    end
  end
en

And here is our bomb.rb process.

1
2
3

$stdout.sync = true
5.times { puts "tick."; sleep 1; puts "tock."; sleep 1}
puts 'KABOOM!'

Its important to instruct ruby to flush content immediately with $stdout.sync = true otherwise, we would wait a long time to get a non-streaming content. Try it out.

Easily enough, you can open up several browsers to esperience how great this is. If you get a ‘stuck’ request, make sure to append some garbage to your url such as http://localhost:4567?r=123. I think this has to do with browser cache mechanisms.

1
2

foo = (hello)=>
  console.log(hello)

Which translates to this javascript code:

1
2
3
4
5

var foo;
var __bind = function(fn, me){ return function(){ return fn.apply(me, arguments); }; };
foo = __bind(function(hello) {
  return console.log(hello);
}, this);

function.apply

To understand what this does, we need to first understand what is function.apply.

From Mozilla Developer Network:

Calls a function with a given this value and arguments provided as an array.

So, each function can be specified of its this. Sounds weird? perhaps not.

In python, you need to pass self as a first argument to a method. Notice that by method I classically mean a function that is called on its object.

1
2
3
4
5

class MyClass:
    """A simple example class"""
    i = 12345
    def f(self):
        return 'hello world'

In cases like working with Backbone, those functions that are triggered as a reaction to a DOM event, may be be run in the context of the DOM/window, simply because the browser will invoke them in that context.

When that happens this will no longer be the Backbone Model that the mentioned function belongs to – that’s because there is no notion of ‘belongs to’, there are no classes in Javascript; but there is a form of those in CoffeeScript.

Fat Arrow

Using a fat arrow gives a function a sense of being a proper method. When we look at the transformed Javascript code, we see that CoffeeScript generates a __bind function, which will take your function, and give you back a function that will call function.apply with a specified this parameters.

In all cases where you would use =>, the current instance of a class will be bound as this for that function you are defining. That closes up the relation between a function and an instance; a function that belongs to an instance, would take it as its this argument, completing its destiny as a proper method.

But I don’t use CoffeeScript!

If you use plain Javascript, you can use underscore.js’s _.bindAll function. Lets see what it does.

1
2
3
4
5
6
7

_.bind = function(func, obj) {
    if (func.bind === nativeBind && nativeBind) return nativeBind.apply(func, slice.call(arguments, 1));
    var args = slice.call(arguments, 2);
    return function() {
      return func.apply(obj, args.concat(slice.call(arguments)));
    };
  };

After analyzing the CoffeeScript translated version, we identify the function.apply idiom here pretty easily. However, underscore will call ECMAScript 5’s native bind if available (nativeBind in the code).

MVC and Selectedness

rgarcia asked:

Not a backbone expert, but how “correct” is it to have a presentation variable like “selected” tied to a backbone model? …

bricestacy said:

…Second, the author makes the claim that Backbone is an MVC and makes things simpler and yet violates that. His way you literally have to go through the view, the collection, the model, a trigger, and back to the view to select a new image.

Note that bricestacy perhaps unknowingly, summarized the way it should work in MVC. You should go to the view, collection, model, and then your view should react back to events from your model!

When you develop a ‘desktopish’ MVC application, you’re not in the same game as you were when you developed serverside / frontend hybrid web applications.

Part of ‘true’ MVC is having a good, rich model – and that stands for a reason. Your model should truely represent your reality; figuratively speaking, it is the body of your application, and your view is what your application wears.

If you think of a model representing whats behind a view, ‘selectedness’ can very well sit in a model - as long as it is a concern of your domain model. Thus, since a view reflects a model, it will also represent the state of the view.

In this example, the selectedness sits in the collection, however it may sit in the view, if that is purely a view’s concern – but not that it isn’t so most of the times.

To understand why, imagine a secondary view binding to this exact ‘selectedness’; in this case you would have to bind ViewA against ViewB for the matter. This means that you cannot compose your entire view freely - you cannot use ViewA without ViewB !.

A model (I see collection as a certain case of a model) will always exist at the core of your application – and that’s why you want to bind against it, or any other model that keeps the selectedness state, and still keep the freedom in your views.

Here is some further relevant discussion

Persistence

Some notes were made about persistence. Its true that the model doesn’t give you an immediate 1:1 persistence story to be synced via Backbone. This post was not intended for that.

Even in ‘proper’ rich clients, there is always the question of how do you persist your client’s model.

Do you use the same domain model (read: backbone Models) in your servers? do you autogenerate a version from those, to your client application? do you build an brand new derivative for your rich client, desktop application?

In this case, the main focal point is to build a domain model that can properly represent the application, without a concern for server-side (backbone sync). The idea is to take care of that at a later stage, and not let it disrupt the richness of the application itself in terms of the model’s design.

Fat Arrows

Thanks to fesplugas with the sharp eyes, althought I’ve not used it in the CoffeeScript ports of the sample, you can (and should) use fat arrows (=>) when you want this to be bound – instead of _.bindAll. CoffeeScript gives you native support for that. More about this in future posts.

Backbone.js. By now you have probably already heard about it, and you’re assuming it can solve all of your problems.

Lets start by clearing up the dust. Backbone is a backbone. Backbone. It is a skeleton, an aggregation of best practices, a great MVC implementation, but at 4.6kb (min) it cannot possibly solve all of your problems.

Also, regarding some criticism with Knockout vs Backbone arguments, many Rails and Rubyists come to backbone - it doesn’t give you Rails-like productivity magically.

“Getting” Backbone

Coming from domain driven design, and having implemented the classic MVC infrastructure myself on not- so-welcoming sets of technologies, I find Backbone to be a very simple and elegant solution for implementing some of these ideas as tools for frontend/Web development.

It all becomes simple when I keep in mind the following about Backbone:

• It is an event bus, that also wires up your MVC, and makes it alive.
• It has Model, View, Controller (recently, Router) base classes.
• It syncs (not discussed here)
• It routes (not discussed here)

The Demo

The purpose of the example is to take on implementing a simple image gallery; and to show how a Model and a View can co-exist under the Backbone infrastructure – given that it is not something that Web or back end devs are used to (however is common on the desktop).

It is not a demo about fitting in a REST backend, so some of the things (like fetch) were mocked out.

Assuming that most of us at one stage or another used a gallery either with jQuery plugins, manually, or hacked it out in the ugly old DHTML days, it makes a good base line use case.

Image Gallery

In an image gallery we have a list of Thumbs, and a Front View. In terms of functionality, we want our thumbs to jump into the front when someone clicks on those. Lets drop to code.

Models

We’re going to build the Thumb model and Thumbs collection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

var Thumb = Backbone.Model.extend({
  defaults: {
    uri: '',
    state: ''
  },
  select: function(state){
  this.set({'state': state ? 'selected' : ''});
  }
});

var Thumbs = Backbone.Collection.extend({
  model: Thumb
  ,
  fetch: function(){
    return _.map(urls, function(url){ return new Thumb({uri: url})});
  }
  ,
  select: function(model){
    if( this.selectedThumb() ){

      this.selectedThumb().select(false);
    }
    this.selected = model;
    this.selected.select(true);
    this.trigger('thumbs:selected');
  }
  ,
  selectedThumb: function(){
    return this.selected;
  }
});

And in coffeescript:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class Thumb extends Backbone.Model

  defaults:
    uri: ''
    state: ''

  select: (state) ->
    st = ''
    st = 'selected' if state
    @set('state' : st)


class Thumbs extends Backbone.Collection
  model: Thumb

  fetch: ->
    _.map(urls, (url)-> new Thumb(uri: url))

 select: (model) ->
    @selected.select(false) if @selected?
    @selected = model
    @selected.select(true)
    @trigger('thumbs:selected')

  selectedThumb: ()->
    @selected

I’d like to discuss the coffeescript version only. We see an implementation of a Thumb and Thumbs. The most important points here are that Thumbs will trigger an event once a certain thumb on it was selected.

A triggered event is any string, conventionally named 'namespace:event'. Another note is that in CoffeeScript, @ is Javascript’s this. I could also have ported underscore’s _.map to a CoffeeScript comprehension but chose not to, in order to reduce friction for those of you not knowing CoffeeScript.

Looking at classic MVC, this is where we can “wire” the thumbs view and make it re-render itself.

And as a side note, you’re probably beginning to realize, that CoffeeScript looks and suits Backbone much, much better – both in terms of LOC and readability. You would be completely spot-on. PS - they come from the same author as well.

Constructing Views

Continuing on, we’re going to construct the Views; FrontView, ThumbView and AppView.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

var thumbs = new Thumbs;

var FrontView = Backbone.View.extend({
  template: _.template('<img src="<%= uri %>" />'),

  el: $('#front'),


  initialize: function(){
    this.model.bind('thumbs:selected', this.render, this);
  },

  render: function(){
    this.el.html(this.template(this.model.selectedThumb().toJSON()));
  }
});


var frontview = new FrontView({model:thumbs});



var ThumbView = Backbone.View.extend({
  tagName: 'li',
  template: _.template('<img src="<%= uri %>" class="<%= state %>" />'),
  events: {
    "click" : "selectThumb"
  },
  initialize: function(){
    this.model.bind('change', this.render, this);
  },
  render: function(){
    console.log('rendering');
    $(this.el).html(this.template(this.model.toJSON()));
    return this;
  },
  selectThumb: function(){

    thumbs.select(this.model);
  }
});

var AppView = Backbone.View.extend({
  el: $("#container"),
  render: function(){
    _.each(new Thumbs().fetch(),
        function(t){
          $('div ul').append( new ThumbView({model: t}).render().el)
        });
  }
});

Yet again, the CoffeeScript port:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

class FrontView extends Backbone.View
  template: _.template('<img src="<%= uri %>" />')

  el: $('#front')

  initialize: ()->
    @model.bind('thumbs:selected', @render, @) # important to give 'this' last param.

  render: ()->
    @el.html(@template(@model.selectedThumb().toJSON()))
    @ # important to give 'this' out, on rendering we'll access 'el'

thumbs = new Thumbs
frontview = new FrontView(model:thumbs)

class ThumbView extends Backbone.View
  tagName: 'li'
  template: _.template('<img src="<%= uri %>" class="<%= state %>" />')
  events:
    "click" : "selectThumb"
  initialize: ()->
    @model.bind('change', @render, @)

  render: ()->
    $(@.el).html(@template(@model.toJSON()))
    @

  selectThumb: ()->
    thumbs.select(@model)


class AppView extends Backbone.View
  el: $("container")

  render: ()->
    _.each thumbs.fetch(), (t)->
      $('div ul').append( new ThumbView(model: t).render().el)

Again, going about the CoffeeScript version (you must admit, it’s easier to reason about, and therefore it must be easier to read and maintain).
A view contains a template, an element which it may mount itself on in the DOM named el, and it may contain tagName which is a tag it might create on demand.

To keep you from wondering, the template can be based on any templating engine you might want to use. Here I’m using underscore’s template and for brevity I’m inlining the content.

The view can react to user action, and to model events.
You set up user initiated actions in events, and you bind to events in initialize.
This is where binding to thumbs:selected comes useful in FrontView.

Notice that some events come built in. A model knows when one of its properties were changed – now you can understand why you are restricted to using get and set on the model; these send built in events regarding things that happen in the model.
Any view can bind on its model-generated events – which is really the power of wiring a view to a model in MVC.

So simply: a user initiates an action on a view (which we wired through events), the model would change, and would trigger on it one or more of its properties. The view is already bound to events from a model, and it would react by re-rendering itself.

Conclusion

Last but not least, we’re going to initiate a render like so:

1
2

window.App = new AppView
window.App.render()

You might notice that the gallery starts out blank - no thumb in front view. I leave it as a trivial excercise to fix :).

Luckily, this is the same in Javascript and CoffeeScript – sans the ’;’.

Hope this helped. As further reference, discussion and corrections about the code here is everything in a Gist

Any questions/corrections, get in touch dotan[ruby-instance-var-symbol]paracode.com or tweet me up: @jondot.

OpenSSL::SSL::SSLError at /auth/facebook/callback
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

It means that open-ssl cannot find your ca_certs (which live under /etc/ssl/certs), which is correct if you’re on windows anyway.

If in rails (using Windows), try disabling verification:

1
2
3

Rails.application.config.middleware.use OmniAuth::Builder do
    provider :facebook, FACEBOOK_KEY, FACEBOOK_SECRET, {:client_options => {:ssl => {:verify => false}}}
end

You’re going to deploy on Linux any way (and you shouldn’t be developing on an env that isn’t identical to your prod env), but you can at least go on.

Being familiar with the compiler by alisey I kept wanting to not use batch files and use just Ruby. Similarly to how that works, it seems you can have it with execjs.

Well, here goes:

$ gem install execjs
# a little proof of concept
$ irb
irb(main):001:0> require 'execjs'
=> true
irb(main):002:0> ExecJS.eval "'hello world'.split(' ')"
=> ["hello", "world"]
irb(main):003:0>

Under the covers, since I’m on Windows, it uses Microsoft Windows Script Host (JScript). Now we’re a small step from a “CoffeeScript Compiler” in Ruby.

All that’s left is to read the CoffeeScript.js compiler source and call CoffeeScript.compile on loaded module.
Now if you take a look at the Rails 3.1 Gemfile you’ll see that it uses ruby-coffee-script that does exactly this!. While I initially thought of implementing this mechanism, I was happy to see that it’s already there in Rails 3.1. I’ll keep the blog post – for educational purposes.

Add to Gemfile:

gem 'rr'

run:

$ bundle install
$ cd vendor/plugins
$ git clone -b rspec2 https://github.com/josephwilk/rspec-rr.git

in your spec_helper.rb:

1
2
3
4

require 'rspec/rr'
...
...
config.mock_with :rr

Now in your controller specs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

describe CommentsController do
  include Devise::TestHelpers
  def mock_comment(stubs={})
  @mock_comment ||= mock_model(Comment, stubs)
  end

  describe "voting" do
  it "should up vote" do
    @user = new_user
    @user.aid = 'account'
    @user.save

    sign_in @user
    mock(Comment).find("4def8b7a53bfd22388000002") { mock_comment }
    put :up, :id => "4def8b7a53bfd22388000002"
  end
  end
end

One of the things people like about Vim is the fact that they customize and fit it to their specific needs. Vim gives you tons of control, at many levels of time investments on your part – be it a quick local vim script tweaking or a proper bundle written either in vimscript or ruby or python or whatever.

For this reason you need to realize that some bundles (vim plugins) may use python, ruby or even C. Command-T for example, uses ruby and C for optimizations. This means you need to find a recent Vim compiled with Ruby bindings, the correct ruby version that it was compiled with installed and visible (PATH) on your system – be it 1.8.7 or 1.9.x.

If you chose to follow the above link for the Vim binaries, then you will have a Ruby 1.9 capable Vim.

Setting Up Your Vim Depdencies

Because many Vim enthusiasts are Rubyists (and because Ruby is awesome), you may better off install a Ruby on your system. You can do that either via pik and have multiple Rubies, or install a single Ruby via the One-Click installer.

You can read all about your options at this site. Notice that this is a link to the Ruby DevKit. This is another dependency that you need in order to compile the C extensions.

To sum it up:

• Go look at this site
• Install Ruby (via One-Click or pik)
• Install DevKit

Setting Up Your initial Vim environment

Your vim environment is composed of your vimfiles and vimrcs. vimfiles is a folder with Vim specific packages such as plugins, color schemes and such; its where all the good stuff is at. You need to put your vimfiles either at the Vim directory (program files) or at your ‘My Documents’ special windows folder.

Lets clone them from my own repository. This repository is in itself a clone of a popular one, with my own win32 related tweaks.

$ git clone https://github.com/jondot/vimfiles-win32
Cloning into vimfiles-win32...
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 6 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.


$ cd vimfiles-win32
$ git submodule init
.. snip tons of output
$ git submodule update
.. wait until all completes
$ cd bundle\Command-T
$ rake make

What you just did is get my win32 repo, pull vimfiles and build Command-T. Command-T is an awesome quicksilver like (or if you’re more familiar - firefox’ awesomebar) file search plugin.

A Word About Vim Configuration

Currently the important stuff to know is that

• Your Leader (a leader is a command prefix) is \
• ESC is jj
• F1, F2, F3 do neat things for you (check it out)
• T (\t) is your Command-T shortcut
• CTRL-L gives you a =>
• behave mswin is on, meaning CTRL-C, CTRL-V and such work as you expect on windows (no consideration for terminal). Some vimmers find it very offensive since CTRL codes are bad for working in terminals across machines – this is clearly far from the case in Windows.

You can now locate your vimrc file and customize it to your desires. Be aware that there is a learning curve awaiting for you – but you don’t need to take it all in one go.

With qtools you’ll be able to perform both deployment and ongoing operation actions pretty easily – modeled

after the UNIX phylosophy:

__Write programs that do one thing and do it well. Write programs to work together. Write programs to handle

text streams, because that is a universal interface__

In practice the qtools are mostly conforming to that model.

The Tools

• qls - list queues. This is an enabler of pipe operations when you want to run batch commands.
• qcount - count messages in a queue.
• qcp - copy queue content to another queue.
• qgrep - grep-like tool for searching inside queue messages.
• qrm - remove a queue
• qtail - tail a queue (show contents and a live message feed)
• qtouch - create a queue
• qtruncate - truncate (empty a queue)

Most of the tools try to rely on their UNIX counterparts for name semantics.

An input is a queue path. If -n isn’t required and you dont specify a queue path via -n you’ll be prompted

for STDIN.

From what you’ll see below, you might be finally able to throw away all of those pesky vbs, bat, and powershell scripts

:).

Quick Examples

Some of the things you can do with the tools as a collection or separately:

Creating a new transactional queue with full permissions for Everyone, a limit of 400KB. Not all

parameters are required.

qtouch -n .\private$\foo_q -p FullControl -u Everyone -l 400 -t

Creating a set of queues from a text file (as part of deployment for example). Note that I use MSYS/Mingw’s cat to stream the text out.

# queues.txt  --snip-snip--
.\private$\xmltestqueue
.\private$\xmltestqueue


$ cat queues.txt | qtouch -p FullControl -u Everyone -l 400 -t

Counting number of messages in a single queue

$ qcount -n .\private$\xmltestqueue
OK: [.\private$\xmltestqueue] 3 message(s).

Counting number of messages in a list of queues (using qls with a pipe)

$ qls -f xmltest | qcount
OK: [.\private$\xmltestqueue] 3 message(s).
OK: [.\private$\xmltestqueue2] 0 message(s).

Removing, truncating and such in the same fasion (use qrm, qtruncate instead of qcount)

Grepping queue contents can be fun (you can also use qls to grep on several queues!):

$ qgrep.exe -n .\private$\xmltestqueue -e a
INFO: [.\private$\xmltestqueue] Listing results.
5/3/2011 7:11:40 PM     *** message id ***    foo:    f<a>sdf
5/3/2011 7:21:26 PM     *** message id ***    asdfadf:        <a>ef

Tailing a queue is also fun - you should see messages on the terminal as they’re added to the queue:

$ qtail -n .\private$\foo_q

Even More

You should check out each command with its various switches. I’ve only covered a small subset of what you could do with qtools.

Output

The qtools output was adjusted to be easily parsable by regex or simple matchers (by position and tabs) so

that if needed, it can be piped to a monitoring or logging system.

Graylog2 is an open source log management solution that stores your logs in MongoDB. It consists of a server written in Java that accepts your syslog messages via TCP, UDP or AMQP and stores it in the database. The second part is a web interface that allows you to manage the log messages from your web browser.

Although many of the common uses for graylog are for error tracking – and you’ll find a rack adapter that can automatically push your errors to your graylog, I intend to share how i’m using graylog in a different way.

Beyond Tracking Exceptions

I wanted a way to track business events but in a way that wouldn’t require much infrastructure to be written, I also didn’t want the system to stop working in the event of that infrastructure to be down.

Using graylog, I got the server side with its analytics capable front-end for free. I also benefited from the graylog protocol being UDP to assure that my system and the business event tracking system don’t influence eachother.

Having this kind of front-end was certainly a big plus:

Implementation was brainless

Going a step further in not inventing the wheel, I chose to use Log4r which will allow me to have several loggers as well as one or more special ‘business events’ loggers on demand and per configuration.

I then levearaged the log4r-gelf appender to track my business events. Configuration was an easy copy-paste-tweak:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

log4r_config:
  loggers:
  - name      : inbox_runner
    level     : INFO
    outputters:
    - stdout
  - name:       cpnel
    level :     INFO
    outputters:
    - gelf
  - name:       remrn
    level :     INFO
    outputters:
    - stdout
  - name:       eproc
    level :     INFO
    outputters:
    - stdout
  - name:       esend
    level :     INFO
    outputters:
    - stdout

  outputters:
  - type     : StdoutOutputter
    name     : stdout
    formatter:
      date_pattern: '%y%m%d %H:%M:%S'
      pattern     : '%d %l: %m '
      type        : PatternFormatter

  - type: Log4r::Gelf::GelfOutputter
    name: gelf
    gelf_server: "192.168.1.105"
    gelf_port: "12201"
      # Optional - showing default values
      # facility: "gelf-rb"
      # host: "#{Socket.gethostname}"
      # max_chunk_size: 'LAN'
      # level: 5 # GELF::UNKNOWN

I also went ahead and divided my events to sections, each getting its own logger instance.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# lib/logging/logger.rb
require 'log4r'
require 'log4r/yamlconfigurator'
require 'log4r-gelf'


module MyApp
  include Log4r
  cfg = YamlConfigurator # shorthand
  cfg.load_yaml_file(File.expand_path('../../config/node_config.yml', File.dirname(__FILE__)))

  module Activity
    def self.control_panel(code, h)
       log 'cpnel', code, h
    end

  private
    def self.log(app, code, h)
      Log4r::Logger[app].info("act:#{code}:#{h.to_json}")
    end
  end

end

Just for back reference, here are some useful links:

• graylog2
• log4r
• log4r-gelf
• gelf-rb

I want to share a trick that has to do with the second method; I’ve had my share of client-side templating. When you look at the typical link_to :remote => true, you realize that what rails does is send you back ‘executable’ javascript that modifies your page.

Now lets assume we have a FoosController and a show action that will render js:

1
2
3
4
5
6
7
8
9

def show
  @foo = Foo.get(params[:id])
  #...
  #...
  #this is important
  respond_to do |f|
    f.js {}
  end
end

The show action will now expect views/foos/show.js.erb to exist. Notice extension .js.erb. Within show.js.erb you can put your Javascript code mingled with your serverside ruby view code.

However what do you do when you want to load up a page, but then load the rest of it – or at least the dynamic parts of it via AJAX, as done by clicking on the link in link_to :remote => true ?

In previous versions of Rails you would do a remote_function call which will render all of the Javascript you require. However in the recent and better Rails the answer is so simple that it comes to this

1

<script type="text/javascript" src="<%= foo_path(@foo) %>"></script>

This will fetch the ‘RJS’ Javascript and execute it. It is enough to plug this at the end of your markup.
You might also recognize that with it you can do cross domain, and cache behind a CDN.

I’ve decided to start testing Albathor on seemingly random open source solutions. Here, I start with Shouldly.

C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871>dir

03/31/2011  06:03 PM    <DIR>          .
03/31/2011  06:03 PM    <DIR>          ..
03/31/2011  06:03 PM               104 .gitignore
03/31/2011  06:03 PM    <DIR>          build
03/31/2011  06:03 PM    <DIR>          lib
03/31/2011  06:03 PM             1,510 LICENSE.txt
03/31/2011  06:03 PM             2,038 README.markdown
03/31/2011  06:03 PM             1,365 Shouldly.sln
03/31/2011  06:03 PM             1,391 Shouldly2010.sln
03/31/2011  06:03 PM    <DIR>          src

Now lets albathor init the Shouldly solution.

C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871>albathor init Shouldly
Creating Albacore build for Shouldly.sln
      create  Rakefile
      create  Gemfile
      apply  C:/Ruby187/lib/ruby/gems/1.8/gems/albathor-0.0.1/templates/default.alba
Done. Run 'bundle install' once to set up your dependencies.

Lets look at our albacore build options with rake -T.

C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871>rake -T
(in C:/Users/Dotan/Downloads/tests/shouldly-shouldly-d5a7871)
rake build  # Build release

That’s it. Albacore is ready to build, lets verify:

C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871>rake build
(in C:/Users/Dotan/Downloads/tests/shouldly-shouldly-d5a7871)
I, [2011-04-08T19:17:39.301646 #7320]  INFO -- : Generating Assembly Info File At: C:/Use
Microsoft (R) Build Engine Version 4.0.30319.1
[Microsoft .NET Framework, Version 4.0.30319.1]
Copyright (C) Microsoft Corporation 2007. All rights reserved.

Build started 4/8/2011 7:17:41 PM.
Project "C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871\Shouldly.sln" on node 1
ValidateSolutionConfiguration:
  Building solution configuration "Release|Any CPU".
Project "C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871\Shouldly.sln" (1) is bu
..
..
..
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:05.11

And we have Shouldly driven by an Albacore build (+AssemblyVersion task as a bonus) in 10 seconds.

Auto Configuring Tests

Story not done yet. Now I’m going to re-do the process with a custom albathor template. The template is located here http://pastie.org/pastes/1747874. In it you’ll find the nunit command which will fetch, configure and find your tests project into your albacore build along with NUnit itself.

C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871>albathor init Shouldly http://pastie.org/pastes/1747874/download
Creating Albacore build for Shouldly.sln
      create  Rakefile
      create  Gemfile
      apply  http://pastie.org/pastes/1747874/download
Downloading nunit...
      create    tools/nunit.zip
Done. Run 'bundle install' once to set up your dependencies.

After this ran I have the following tasks magically pointing to the correct test assembly and tools (which were downloaded and unpacked for my by Albathor):

desc "Run tests"
nunit :nunit do |nunit|
  nunit.command = "tools/NUnit-2.5.9.10348/bin/net-2.0/nunit-console.exe"
  nunit.assemblies "src/Tests/bin/Release/Tests.dll"
end

And now I’m able to run the tests happily:

C:\Users\Dotan\Downloads\tests\shouldly-shouldly-d5a7871>rake nunit
(in C:/Users/Dotan/Downloads/tests/shouldly-shouldly-d5a7871)
NUnit version 2.5.9.10348
Copyright (C) 2002-2009 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
  OS Version: Microsoft Windows NT 6.1.7600.0
  CLR Version: 2.0.50727.4952 ( Net 2.0 )

ProcessModel: Default    DomainUsage: Single
Execution Runtime: Default
.........................................................
Tests run: 57, Errors: 0, Failures: 0, Inconclusive: 0, Time: 2.4151326 seconds
  Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

The fun thing is that I didn’t customize anything at all.

Cool? I’d love to hear your suggestions/opinion, tweet me up @jondot.

To get a feel for it simply:

$ git clone https://github.com/jondot/darkness
$ cd darkness
$ jruby darkness.rb

Then remember this:

• ESC quits.
• Use CTRL(Command)-L, CTRL(Command)-S to load and save.

Whats It Like?

The purpose of darkness was educational; to provide a showcase of converting an old demo app from Java to JRuby. However, it is fully usable and can become a cross platform, lightweight, alternative to those who want to write distraction free.

Converting your existing Swing application from Java to JRuby is very possible. I’ve started doing this with zero knowledge of referencing Java from my JRuby powered code. I’ve learnt everything on the fly.

Take a look at the sources of both the original and the JRuby version. You may like the reduction in code size and readability and tempted to do the same for your apps. I’ve also included in comments some pitfalls I’ve fallen into.

if you havn’t yet given JRuby a run, you should definitely go for it: http://www.jruby.org/

• Personal, tiny-smart-hackable, OpenID provider.
• OpenID provider test server.

Here are some scenarios that Passage might help you with:

• You have an internal / sandboxed / restricted environment, as a test server
• You don’t really care about providing your real ID to a web site.
• You wish to remain completely anonymous.
• You wish to implement your own authentication strategy (finger print, cell call)
• You wish to have full control over your identity and registration detail
• You want several identities
• .. The list goes on..

Quick Start

Passage can be used both as a gem, or as a deployable web (rack) app.

As a personal gem:

    $ gem install passage
    $ passage --ids myids.yml --auth pass_through

Or deploy it:

    $ git clone https://github.com/jondot/passage
    $ export PSG_IDS_FILE=myids.yml
    $ cd passage
    $ rackup 

Thats it!

If you’ve noticed, you’ve specified an identities file and an authentication strategy. Passage will load your identities from the file and will auto discover all of your authentication strategies and load the one you’ve selected. Lets discuss these quickly:

Your identities (identities.yml)

    $cat identities.yml.example
    'http://localhost:9292/ids/foo':
      email: foo@foo.org
    'http://77.127.240.49:9292/ids/foo':
      email: foo@foo.org
    'http://77.124.212.222:4567/ids/pookie':
      email: goo@goo.org
    !ruby/regexp /http:\/\/localhost:4567\/ids\/(.*)/:
      email: '#{$1}@foo.org'
      nickname: '#{$1}'

This is a basica YAML file with a twist: The line starting with !ruby/regexp is a proper regular expression with a capture group. The OpenID SReg fields following it (email, nickname) are using the capture group in order to have their own values without your intervention. You can identify these as being plain ruby interpolated strings.

Authentication Strategies

the auth folder holds authentication strategies. An authentication strategy answers to the following:

1
2
3

def auth_setup(req); end
def auth_fetch; end
def auth_validate!(identity, trust_root); end

If you’d want to make your own authentication strategy, a good starting point will be to copy-paste an existing one and tweak it to your use. Authentication strategies are auto-discovered, so just drop your folder in /auth, and configure your selected one via configuration. An example authentication strategy would be one that would present a user/password form and store credentials at the DB. If you think about it, that will “upgrade” your Passage instance from a personal to a full fledged OpenID server.

Here is a description of the existing authentication strategies

• pass_through - Lets authentication always pass. No need for any credentials or so.
• pass_phrase - Authentication will pass only if a proper pass phrase was specified.

As a Personal OpenID Provider

You can have your fixed identities at your domain, or even throw-away identities.

To set fixed identities in your domain, simply code them up in your identities.yml file and let passage know about it.

Using throw-away identities can maximize your anonymity around the Web. To have throw-away identities you can specify any identity you wish in your identities.yml manually, or specify a smart rule as a regex identity:

1
2

!ruby/regex /http:\/\/mydomain\.com\/ids\/(.*)/:
  email: 'myuser@gmail.com'

This will let you specify any user you wish to make up, at your domain.

As an OpenID Test Server

Passage is great (and being used internally) for integration tests against OpenID consumers (relying parties). Through Passage’s identity configuration system, you can load up fixed identities per test scenario, or you can load identities which follow certain rules (with regex enabled identities).

As an example, you can store as fixed identities in your dev environment:

1
2

'http://dev.com/ids/foo':
  email: foo@dev.com

Or, you can store the following if you have many dynamic users:

1
2

!ruby/regex /http:\/\/dev\.com\/ids\/(.*)/:
  email: '#{$1}@dev.com'

This makes use of regex enabled identities, in which you specify a regex as the user identifier, and any SReg property can make use of the captures that were made.

Conclusion

Passage aims to be a tiny but smart OpenID provider – one that you can easily tweak and customize.

I’m using passage mainly to stub out a provider in automated GUI tests, while in some projects I have to mimick a different OpenID provider behavior. I must say Ruby is a hugh enabler of giving me the ability to make quick changes and go on with my main goals.

I’ve jused pushed support for C# in rocco, about 3 LOC. Here is how it looks like over NChurn’s Program.cs file:

(Click for full view)

To have this effect with rocco, you need to streamline comments in your code in Markdown (much like what I’m writing this blog post in). Such as

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

    static Program()
    {
        // Static Maps
        // -----------
        // We're adding a key-value type initializer. Something like a poor's man IoC,
        // in this case -- reporters.
        _reporterMap.Add("table", x=> new TableReporter(x));
        _reporterMap.Add("csv", x=> new CSVReporter(x));
        _reporterMap.Add("xml", x=> new XMLReporter(x));

        // Continuing on, here we're looking at adapters.
        // Note the `auto` adapter -- a special kind of adapter.
        _adapterMap.Add("git", x => WithPath(x, new GitAdapter()));
        _adapterMap.Add("svn", x => WithPath(x,new SvnAdapter()));
        _adapterMap.Add("tf", x => WithPath(x,new TFAdapter()));
        _adapterMap.Add("hg", x => WithPath(x,new HgAdapter()));
        _adapterMap.Add("auto", x => WithPath(x, new AutoDiscoveryAdapter()));
    }

You may also use multi-line comments (note the opening block is /**) such as

1
2
3

/**
* this is a multiline block
*/

Finally to run rocco on your files do

$ rocco -l csharp *.cs

Enjoy.