Livenode — Live-code your NodeJS application

I'm working on a NodeJS application for Kendo UI. If you've been following me for a while, you probably know that I'm not the biggest fan of NodeJS1—however, it seems that for this project Node is the best option, so I actually pushed for it.

A small annoyance that I'm trying to “fix” is the requirement to restart the server all the time in order to test changes. After some years with Common Lisp, I know there are better ways to do development than restarting the program — so here's Livenode, a module that allows to “live-code” a NodeJS application.

What's “Live Coding”?

In essence, it's a way to alter the program while it runs, without restarting it. The goal, however, is to end up with a program that works properly when we (re)start it. To be useful, it helps to have an editor that can be customized to send code to the server.

It's easier to show this in a screen-cast, so I made one. Before you watch it, let me warn you that I've been told that “anyone who's seen a Dracula movie is familiar with the accent”. Go figure how good am I at making screen-casts :-). Just ignore this aspect and check out the demo.

The challenge

The NodeJS module system (really, the require function) wraps all the code in a module into something like this:

(function (exports, require, module, __filename, __dirname) {
    // and here goes the module code
})(...);

It does that so we can have the “luxury” to have file-local variables and functions, and be selective about what we export. What this means, however, is that everything inside the module is “private” and in order to provide a facility to evaluate code in the environment of the module the only way is perhaps to export a function which takes a piece of code and evaluates it there, something like the following (and again, I'm adding the CommonJS wrapper, to show the whole picture):

(function (exports, require, module, __filename, __dirname) {
    // ... module code here
    exports.$__EVAL = function() { return eval(arguments[0]) };
})(...);

It might seem that's everything Livenode has to do to make my screen-cast work; but it's trickier than that. If the code that you evaluate is like var counter = 0; then the counter variable could not escape the $__EVAL function. There is no way in JS2 to introduce names in an outer scope. The counter would get created inside the $__EVAL function, but that wouldn't be of any use. What we need is to create a proper “file-local” variable, so that we can access it later from another function defined inside the module.

Solution: code rewriting

The only solution I could think of was to rewrite all code inside a module such that there are no free “file-local” variables; instead, the module environment would be stored in a local object. Here is an approximation of what happens:

var foo = something();
function getFoo() {
    return foo;
}
function incFoo() {
    foo++;
}

var $__CONTEXT = {};
$__CONTEXT.foo = something();
$__CONTEXT.getFoo = function getFoo() {
    return $__CONTEXT.foo;
};
$__CONTEXT.incFoo = function incFoo() {
    $__CONTEXT.foo++;
};
exports.$__EVAL = function() {
    eval(REWRITE(arguments[0]));
};

That would be the initial transformation — when the program starts, all modules will be silently rewritten like above before being evaluated. Later when we need to execute a single statement, we call the $__EVAL function of the appropriate module, which applies the same rewriting (but of course, without redefining $__CONTEXT).

Handling exports

For convenience, assignment to exports is treated in a special way. Here's why. Suppose you have the following code:

function foo() { return 1 }
exports.foo = foo;

$__CONTEXT.foo = function foo() { return 1 };
exports.foo = $__CONTEXT.foo;

Later you'll evaluate only the definition of foo, to make it return 2, for example:

function foo() { return 2 }

$__CONTEXT.foo = function foo() { return 2 };

If you evaluate only this definition, without evaluating the exports line again, then even though you've changed the local function, the module's exports will still point to the old definition. So, if exports would not be handled in a special way, after each change to an exported function you'd need to “export it again”, that is, to re-evaluate the line that assigns that function to exports (or, well, to restart the server, but that's what we want to avoid).

To make this more convenient, Livenode will transform assignments to exports like the following:

exports.foo = foo;

exports.__defineGetter__("foo", function() {
    return $__CONTEXT.foo;
}), exports.__defineSetter__("foo", function(v) {
    exports.__defineGetter__("foo", function() {
        return v;
    });
});

Seems like a lot of trouble, but it helps: by making it a getter, rather than assigning the computed value at the time the assignment is made, it allows you to redefine that thing in $__CONTEXT without bothering to re-export it. For correctness, it also defines a setter which makes it behave like an ordinary property.

Caveat emptor

Not 100% keeping semantics

The code rewriting should ideally keep the semantics intact, but I had to draw somewhere the line between convenience and correctness. For example, as you've seen in the previous section, it does this exports magic, so that you can redefine stuff without exporting it again. Strictly speaking, this doesn't always preserve code logic—but it works well in every practical case I can imagine. Here's an example where it would fail:

// module foo.js
var foo = "foo";
exports.foo = foo;
foo = "bar";

// normally:
require("foo").foo // → "foo"

// with Livenode:
require("foo").foo // → "bar"

I hope it's obvious why that happens.

Be wary about caching exports lookups

A related issue (not exactly an issue, just something to keep in mind) is that if you “cache” the exported values, your caches won't be updated when the value changes in the module. For example:

var foo = require("foo");
var my_bar = foo.bar;

   //... in some function...
   my_bar();

When you will change the value of bar in the "foo" module, the my_bar variable will continue to point to the previous value. This is exemplified in the screen-cast with the "handle" function.

exports[i] = stuff

Some people populate exports by iterating through some other object with a for/in, like this:

for (var i in stuff) {
    exports[i] = stuff[i];
}

// rarely seen, but this kind of for/in is valid syntax
for ($__CONTEXT.i in stuff) {
    exports[$__CONTEXT.i] = stuff[$__CONTEXT.i];
}

Note that in the transformation we don't do any special magic for exports this time. It would be buggy, because if we delay evaluation of $__CONTEXT.i (if we'd put it in a getter) then it'll only get the last value of the loop variable i. If you do this, you must remember to evaluate this loop each time you modify an exported function, so that other modules pick up your change.

Only for CommonJS

As you can see, Livenode is specifically designed around the NodeJS module system. This means that if your modules circumvent somehow the “standard” module system, you won't be able to evaluate statements at run-time into them. Ironically, I can't use it on some of my own code (for example UglifyJS 2.0 does not use CommonJS, instead it implements its own file loader).

The editor interface

You might have noticed in the screen-cast that I'm using Emacs. I've been with it for like 15 years and I know it fairly well, so I was able to program the interface into it without much effort—but Livenode itself doesn't have much to do with Emacs. It should be trivial to add support for Your Favorite Editor, assuming it's programmable. You need a JSON library3 and a WebSocket client library4. I'd love to see support for VIM or Sublime Text, but I don't know them well enough to write it myself—patches welcome, and please drop me a line if I can help!

The Emacs interface is pretty simple:

• Load livenode.el.

• M-x livenode to connect to the running process (assuming you started your application with livenode).

• C-M-x to evaluate the toplevel statement at cursor position. If it's an expression, you get the result in the minibuffer (or check *Messages* if it's too big). If there are warnings, it'll be highlighted in some reddish nuance. If there's an immediate error, you get it in the minibuffer.

• C-c C-r to evaluate the region. Feel free to select multiple statements.

• M-x livenode-gencode-region to do code rewriting on the region. The result goes in the minibuffer (and of course, *Messages*).

If you kill the server, Emacs disconnects and notices you in the minibuffer. You need to M-x livenode again to reconnect after you restart the server (it doesn't do it automatically for the time being).

Add your comment