Introducing jlbox: Realtime Testing with Julia, FactCheck.jl, Gulp.js, and ZMQ

When writing ruby code, I always enjoyed having my tests run side-by-side as I wrote code. I used a variety of combinations through the years, starting with the watchr gem, and eventually moving to guard. In julia, the language doesn't have as many tools available due to the infancy of it's package ecosystem. However, I combined a few tools to allow me to save my changes and see my test results immediately in a terminal window.

tl;dr:

  1. install node and zmq
  2. npm install -g jlbox
  3. 'jlbox init' in your project directory

Dependencies

First, you need to install node.js and zmq bidings if you don't have them already. I'm using homebrew on OS X, so the following command works:

1
2

brew install node
brew install zmq

Once node is installed, you will have two new executables: node and npm. From here, you can install the other node packages we need:

1
2
3

npm install -g gulp
npm install --save-dev gulp
npm install --save-dev zmq

Julia

I wasn't a fan of the existing Julia testing framework because I couldn't easily inspect the expected vs. actual results. FactCheck.jl nicely provides this functionality. In addition, you can run your tests as a module and reload it when you need to test again. This feature comes in handy as we'll see later on.

From your Julia command line, install FactCheck and ZMQ:

1
2

julia > Pkg.add("FactCheck");
julia > Pkg.add("ZMQ");

Julia ZMQ Binding

We're now going to setup a simple julia process that will bind to a ZMQ socket and take requests from node.js. When we get a request, we're going to reload the test/module file(s) and run our test cases.

Here's a file to get started; let's call this file gulp.jl:

1
2
3
4
5
6
7
8
9
10
11

using ZMQ
const ctx = Context()
const sock = Socket(ctx, REP)
ZMQ.bind(sock, "tcp://127.0.0.1:7752")

while true
   println("Waiting for input")
   msg = ZMQ.recv(sock)
   reload(bytestring(msg))
   ZMQ.send(sock, "SUCCESS")
end

gulpfile.js

Now we need to setup the gulpfile to watch our test cases and code, and run the corresponding test cases where appropriate.

My project structure looks like this:

1
2
3
4

src\
    - example.jl
test\
    - example_test.jl

Given this structure, we want to execute the example_test.jl file any time example.jl or example_test.jl changes.

The below gulpfile does the following things:

  1. Starts a julia child process which binds to a ZMQ socket. In the event your code causes Julia to crash, gulp will restart julia for you.
  2. Uses a node.js ZMQ package to connect to the ZMQ socket setup by the julia child process.
  3. Watches src/ and test/ directories for file changes.
  4. When a file change is detected, passes the test file to be re-run over the ZMQ socket to the running julia child process.

Here's the gulpfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

var gulp      = require('gulp');
var gutil     = require('gulp-util');
var zmq       = require('zmq');
var spawn     = require('child_process').spawn;
var fs        = require('fs');
var file_path;
var sock;

// watch all jl files in tests and scripts dirs
var paths     = {
  tests: 'test/**/*.jl',
  scripts: 'src/**/*.jl'
};

var setupJuliaProcess = function() {
  sock      = zmq.socket('req');
  var child = spawn("julia", ["--color", "gulp.jl"], {cwd: process.cwd()});

  // connect to socket
  sock.connect('tcp://127.0.0.1:7752');

  child.stdout.setEncoding('utf8');
  child.stdout.on('data', function(data) {
    data = data.trim();
    if (data) {
      console.log(data);
    }
  });

  child.stderr.setEncoding('utf8');
  child.stderr.on('data', function(data) {
    console.log(data.trim());
  });

  child.on('close', function(code) {
    gutil.log(gutil.colors.red("Julia exited, with exit code: ", code));
    gutil.log(gutil.colors.green("Restarting julia..."));
    setupJuliaProcess();
  });
};

// build child process
setupJuliaProcess();

gulp.task('juliaZMQ', function(){
  // if file name exists, send it across
  if (fs.existsSync(file_path)) {
    gutil.log("Sending " + file_path + " to julia");
    sock.send(file_path);
  }
});

var watcher = gulp.watch([paths.tests, paths.scripts], ['juliaZMQ']);
// regex to match any jl files in src dir
var re = /(.*\/)src\/(.+)\.jl/;

watcher.on('change', function(event){
  if (event.type !== 'deleted') {
    gutil.log(event.path + ' ' + event.type);
    var match = re.exec(event.path);

    // if no match, test file has been edited
    if (match === null) {
      file_path = event.path;
    }
    // src file has been edited, find the corresponding test file
    else {
      file_path = match[1]+'test/'+match[2]+'_test.jl';
    }
  }
  else {
    file_path = "";
  }
});

gulp.task('default', ['juliaZMQ']);

Write Julia Code

Below I'm making a module called Simple. It helpful to use a module so I can easily reload it in the test file.

In src/simple.jl:

1
2
3
4
5

module Simple
  function sum(num1::Int64, num2::Int64)
    return num1+num2
  end
end

For the test file a few things are done:

  1. We use a module again so it can be easily reloaded.
  2. We include our source file (also a module) relative to the project root. Gulp is going to start the julia process in the project root, so the path we provide to reload should be relative to that point.
  3. We use FactCheck to run our tests.

In test/simple_test.jl:

1
2
3
4
5
6
7
8
9
10
11

module SimpleTest
using FactCheck
reload("$(pwd())/src/simple.jl")
using Simple

facts("simple sum") do
    @fact Simple.sum(1,4) => 5
    @fact Simple.sum(1,36) => 37
end

end # module SumTest

Putting it All Together

Now we have all the pieces we need. From your terminal, run the following commands:

  1. In terminal, type: gulp
  2. Modify simple_test.jl or simple.jl and save.
  3. Profit

You should now see some output in your julia terminal window, with the tests automatically being re-run! Enjoy!