Testing Grunt Plugin From Grunt

Writing tests for grunt plugin turned out to be less straightforward then expected. I needed to run multiple task configurations and wanted to invoke them all by typing grunt test in main directory.

Grunt normally exits after first task failure. That makes it impossible to store multiple failure scenarios inside the main project gruntfile. Running them from there would require the --force option, but grunt then ignores all warnings which is not optimal.

Cleaner solution is to have a bunch of gruntfiles in separate directory and invoke them all from the main project gruntfile. This post explains how to do that.

Table of Contents

• Demo Project
• Running Gruntfile From Subdirectory
• The Problem
• Explanation
• Solution 1: Duplicate Npm Repository
• Solution 2: Load Grunt Tasks From Parent Directory
• Calling Gruntfile From Javascript
• Exec
• Call Exec
• Unit Tests
• Disadvantages
• Fork
• Call Grunt
• Module Arguments
• Call Fork
• Write Tests
• Disadvantages
• Spawn
• The Problem
• Looping Suffixes
• Write Tests
• Disadvantages
• Conclusion

Demo Project

Demo project is small grunt plugin with one grunt task. The task either fails with warning or prints success message into the console depending on the value of action options property.

The task:

grunt.registerMultiTask('plugin_tester', 'Demo grunt task.', function() {
  //merge supplied options with default options
  var options = this.options({ action: 'pass', message: 'unknown error'});

  //pass or fail - depending on configured options
  if (options.action==='pass') {
    grunt.log.writeln('Plugin worked correctly passed.');
  } else {
    grunt.warn('Plugin failed: ' + options.message);
  }
});

There are three different ways how to write grunt plugin unit tests. Each solution has its own nodeunit file in test directory and is explained in this post:

• plugin_exec_test.js - the most practical solution,
• plugin_fork_test.js - solves rare edge case where previous solution fails,
• plugin_spawn_test.js - possible, but least practical.

All three demo tests consist of three different task configurations:

// Success scenario
options: { action: 'pass' }
// Fail with "complete failure" message
options: { action: 'fail', message: 'complete failure' }
//Fail with "partial failure" message
options: { action: 'fail', message: 'partial failure' }

Each configuration is stored in separate gruntfile inside test directory. For example, success scenario stored inside gruntfile-pass.js file looks like this:

grunt.initConfig({
  // prove that npm plugin works too
  jshint: { 
    all: [ 'gruntfile-pass.js' ] 
  },
  // Configuration to be run (and then tested).
  plugin_tester: { 
    pass: { options: { action: 'pass' } } 
  }
});

// Load this plugin's task(s).
grunt.loadTasks('./../tasks');
// next line does not work - grunt requires locally installed plugins
grunt.loadNpmTasks('grunt-contrib-jshint');

grunt.registerTask('default', ['plugin_tester', 'jshint']);

All three test gruntfiles look almost the same, only the options object of plugin_tester target changes.

Running Gruntfile From Subdirectory

Our test gruntfiles are stored in test subdirectory and grunt does not handle such situation well. This chapter explains what the problem is and shows two ways how to solve it.

The Problem

To see the problem, go to the demo project directory and run following command:

grunt --gruntfile test/gruntfile-problem.js

Grunt responds with following error:

Local Npm module "grunt-contrib-jshint" not found. Is it installed?
Warning: Task "jshint" not found. Use --force to continue.

Aborted due to warnings.

Explanation

Grunt assumes that grunfile and node_modules repository are stored in the same directory. While node.js require function searches all parent directories for required module, grunts loadNpmTasks does not.

This problem has two possible solutions, simple and fancy one:

• create local npm repository in tests directory (simple),
• make grunt load tasks from parent directories (fancy).

Although the first "simple" solution is somewhat cleaner, demo project uses second "fancy" solution.

Solution 1: Duplicate Npm Repository

The main idea is simple, just create another local npm repository inside the tests directory:

• Copy package.json file into tests directory.
• Add test only dependencies into it.
• Run npm install command every time you run tests.

This is the cleaner solution. It has only two downsides:

• test dependencies have to be maintained separately,
• all plugin dependencies have to be installed in two places.

Solution 2: Load Grunt Tasks From Parent Directory

The other solution is to force grunt to load tasks from npm repository stored inside another directory.

Grunt Plugin Loading

Grunt has two methods able to load plugins:

• loadTasks('directory-name') - loads all tasks inside a directory,
• loadNpmTasks('plugin-name') - loads all tasks defined by a plugin.

The loadNpmTasks function assumes fixed directory structure of both grunt plugin and modules repository. It guess name of directory where tasks should be stored and then calls loadTasks('directory-name') function.

Local npm repository has separate subdirectory for each npm package. All grunt plugins are supposed to have tasks subdirectory and .js files inside it are assumed to contain tasks. For example, loadNpmTasks('grunt-contrib-jshint') call loads tasks from node_mudules/grunt-contrib-jshint/tasks directory and is equivalent to:

grunt.loadTasks('node_modules/grunt-contrib-jshint/tasks')

Therefore, if we want to load all tasks of grunt-contrib-jshint plugin from parent directory, we can do following:

grunt.loadTasks('../node_modules/grunt-contrib-jshint/tasks')

Loop Parent Directories

More flexible solution is to climb through all parent directories until we find closest node_modules repository or reach root directory. This is implemented inside grunt-hacks.js module.

The loadParentNpmTasks function loops parent directories :

module.exports = new function() {

  this.loadParentNpmTasks = function(grunt, pluginName) {
    var oldDirectory='', climb='', directory, content;

    // search for the right directory
    directory = climb+'node_modules/'+ pluginName;
    while (continueClimbing(grunt, oldDirectory, directory)) {
      climb += '../';
      oldDirectory = directory;
      directory = climb+'node_modules/'+ pluginName;
    }

    // load tasks or return an error
    if (grunt.file.exists(directory)) {
      grunt.loadTasks(directory+'/tasks');
    } else {
      grunt.fail.warn('Tasks plugin ' + pluginName + ' was not found.');
    }
  }

  function continueClimbing(grunt, oldDirectory, directory) {
    return !grunt.file.exists(directory) &&
      !grunt.file.arePathsEquivalent(oldDirectory, directory);
  }

}();

Modified Gruntfile

Finally, we need to replace the usual grunt.loadNpmTasks('grunt-contrib-jshint') call in the gruntfile by following:

var loader = require("./grunt-hacks.js");
loader.loadParentNpmTasks(grunt, 'grunt-contrib-jshint');

Shortened gruntfile:

module.exports = function(grunt) {
  var loader = require("./grunt-hacks.js");

  grunt.initConfig({
    jshint: { /* ... */  },
    plugin_tester: { /* ... */ }
  });

  grunt.loadTasks('./../tasks');
  loader.loadParentNpmTasks(grunt, 'grunt-contrib-jshint');
};

Disadvantages

This solution has two disadvantages:

• It does not deal with collection plugins.
• If grunt ever changes expected structure of grunt plugins, you will have to modify the solution.

If you need collection plugins too, have a look at grunts task.js to see how to support them.

Calling Gruntfile From Javascript

Second thing we need to do is to invoke the gruntfile from javascript. The only complication is that grunt exits whole process on task failure. Therefore, we need to call it from child process.

Node module child process has three different functions able to run command inside child process:

• exec - executes command on command line,
• spawn - differently executes command on command line,
• fork - runs node module in child process.

The first one, exec, is easiest to use and is explained in the first subchapter. Second subchapter shows how to use fork and why it is less optimal then exec. Third subchapter is about spawn.

Exec

Exec runs command line command inside a child process. You can specify in which directory to run it, set up environment variables, set timeout after which the command will be killed and so on. When the command finishes its run, exec calls callback and passes it stdout stream, stderr streams and error if the command crashed.

Unless configured otherwise, command is run in current directory. We want it to run inside tests subdirectory, so we have to specify cwd property of options object: {cwd: 'tests/'}.

Both stdout and stderr streams content are stored inside a buffer. Each buffer has maximum size set to 204800 and if the command produces more output, exec call will crash. That amount is enough for our small task. If you need more you have to set maxBuffer options property.

Call Exec

Following code snippet shows how to run the gruntfile from exec. The function is asynchronous and calls whenDoneCallback after all is done:

var cp = require("child_process");

function callGruntfile(filename, whenDoneCallback) {
  var command, options;
  command = "grunt --gruntfile "+filename+" --no-color";
  options = {cwd: 'test/'};
  cp.exec(command, options, whenDoneCallback);
}

Note: if you installed npm into tests directory (simple solution), then you need to use callNpmInstallAndGruntfile function instead of callGruntfile:

function callNpmInstallAndGruntfile(filename, whenDoneCallback) {
  var command, options;
  command = "npm install";
  options = {cwd: 'test/'};
  cp.exec(command, {}, function(error, stdout, stderr) {
    callGruntfile(filename, whenDoneCallback);
  });
}

Unit Tests

First node unit test runs success scenario and then checks whether process finished without failure, whether standard output contains expected message and whether standard error is empty.

Success scenario unit test:

pass: function(test) {
  test.expect(3);
  callGruntfile('gruntfile-pass.js', function (error, stdout, stderr) {
    test.equal(error, null, "Command should not fail.");
    test.equal(stderr, '', "Standard error stream should be empty.");

    var stdoutOk = contains(stdout, 'Plugin worked correctly.');
    test.ok(stdoutOk, "Missing stdout message.");
    test.done();
  });
},

Second node unit test runs "complete failure" scenario and then checks whether process failed as expected. Note that standard error stream is empty and warnings are printed into standard output.

Failing scenario unit test:

fail_1: function(test) {
  test.expect(3);
  var gFile = 'gruntfile-fail-complete.js';
  callGruntfile(gFile, function (error, stdout, stderr) {
    test.equal(error, null, "Command should have failed.");
    test.equal(error.message, 'Command failed: ', "Wrong error message.");
    test.equal(stderr, '', "Non empty stderr.");

    var stdoutOk = containsWarning(stdout, 'complete failure');
    test.ok(stdoutOk, "Missing stdout message.");
    test.done();
  });
}

Third "partial failure" node unit test is almost the same as the previous one. Whole tests file is available on github.

Disadvantages

Disadvantage:

• Maximum buffer size must be set in advance.

Fork

Fork runs node.js module inside child process and is equivalent to calling node <module-name> on command line. Fork uses callbacks to send standard output and standard error to caller. Both callbacks can be called many times and caller obtains child process outputs in pieces.

Using fork makes sense only if you need to handle arbitrary sized stdout and stderr or if you need to customize grunt functionality. If you do not, exec is easier to use.

This chapter is split into four sub-chapters:

• call grunt from javascript,
• read command line arguments inside node module,
• start node module inside a child process,
• write unit tests.

Call Grunt

Grunt was not meant to be called programatically. It does not expose "public" API and does not document it.

Our solution mimics what grunt-cli does, so it is relatively future safe. Grunt-cli is distributed separately from grunt core and therefore is less likely to change. However, if it does change, this solution will have to change too.

Running grunt from javascript requires us to:

• separate gruntfile name from its path,
• change active directory,
• call grunts tasks function.

Call grunt from javascript:

this.runGruntfile = function(filename) {
  var grunt = require('grunt'), path = require('path'), directory, filename;
  
  // split filename into directory and file
  directory = path.dirname(filename);
  filename = path.basename(filename);

  //change directory
  process.chdir(directory);

  //call grunt
  grunt.tasks(['default'], {gruntfile:filename, color:false}, function() {
    console.log('done');
  });
};

Module Arguments

The module will be called from command line. Node keeps command line arguments inside process.argv array:

module.exports = new function() {
  var filename, directory;

  this.runGruntfile = function(filename) {
    /* ... */
  };

  //get first command line argument
  filename = process.argv[2];
  this.runGruntfile(filename);
}();

Call Fork

Fork has three arguments: path to module, array with command line arguments and options object. Call module.js with tests/Gruntfile-1.js parameter:

child = cp.fork('./module.js', ['tests/Gruntfile-1.js'], {silent: true})

The silent: true option makes stdout and stderr of the returned child process available inside the parent. If it is set to true, returned object provides access to stdout and stderr streams of the caller.

Call on('data', callback) on each stream. Passed callback will be called each time the child process sends something to the stream:

child.stdout.on('data', function (data) {
  console.log('stdout: ' + data); // handle piece of stdout
});
child.stderr.on('data', function (data) {
  console.log('stderr: ' + data); // handle piece of stderr
});

Child process can either crash or end its work correctly:

child.on('error', function(error){
  // handle child crash
  console.log('error: ' + error); 
});
child.on('exit', function (code, signal) {
  // this is called after child process ended
  console.log('child process exited with code ' + code); 
});

Demo project uses following function to calls fork and to bind callbacks:

/**
 * callbacks: onProcessError(error), onProcessExit(code, signal), onStdout(data), onStderr(data)
 */
function callGruntfile(filename, callbacks) {
  var comArg, options, child;
  callbacks = callbacks || {};

  child = cp.fork('./test/call-grunt.js', [filename], {silent: true});

  if (callbacks.onProcessError) {
    child.on("error", callbacks.onProcessError);
  }
  if (callbacks.onProcessExit) {
    child.on("exit", callbacks.onProcessExit);
  }
  if (callbacks.onStdout) {
    child.stdout.on('data', callbacks.onStdout);
  }
  if (callbacks.onStderr) {
    child.stderr.on('data', callbacks.onStderr);
  }
}

Write Tests

Each unit test calls the callGruntfile function. Callbacks search for expected content inside the standard output stream, check whether exit code was correct, fail when something shows up on error stream or fail if fork call returns an error.

Success scenario unit test:

pass: function(test) {
  var wasPassMessage = false, callbacks;
  test.expect(2);
  callbacks = {
    onProcessError: function(error) {
      test.ok(false, "Unexpected error: " + error);
      test.done();
    },
    onProcessExit: function(code, signal) {
      test.equal(code, 0, "Exit code should have been 0");
      test.ok(wasPassMessage, "Pass message was never sent ");
      test.done();
    },
    onStdout: function(data) {
      if (contains(data, 'Plugin worked correctly.')) {
        wasPassMessage = true;
      }
    },
    onStderr: function(data) {
      test.ok(false, "Stderr should have been empty: " + data);
    }
  };
  callGruntfile('test/gruntfile-pass.js', callbacks);
}

Tests corresponding to failure scenario are pretty much the same and can be found on github.

Disadvantages

Disadvantages:

• Used grunt function does not belong to official API.
• Child process output streams are available in chunks instead of one big block.

Spawn

Spawn is a cross between fork and exec. Similarly to exec, spawn is able to run an executable file and pass it command line arguments. Child process output streams are treated the same way as in fork. They are send to parent in pieces via callbacks. Therefore, exactly as with fork, using spawn makes sense only if you need arbitrary sized stdout or stderr.

The Problem

The main problem with spawn happens on windows. The name of command to be run must be specified exactly. If you call spawn with an argument grunt, spawn expects executable filename without suffix. Real grunt executable grunt.cmd will not be found. Otherwise said, spawn ignores windows environment variable PATHEXT.

Looping Suffixes

If you want to call grunt from spawn, you will need to do one of the following things:

• use different code for windows and for linux or
• read PATHEXT from environment and loop through it until you find the right suffix.

Following function loops through PATHEXT and passes the right filename to the callback:

function findGruntFilename(callback) {
  var command = "grunt", options, extensionsStr, extensions, i, child, onErrorFnc, hasRightExtension = false;

  onErrorFnc = function(data) {
    if (data.message!=="spawn ENOENT"){
      grunt.warn("Unexpected error on spawn " +extensions[i]+ " error: " + data);
    }
  };

  function tryExtension(extension) {
    var child = cp.spawn(command + extension, ['--version']);
    child.on("error", onErrorFnc);
    child.on("exit", function(code, signal) {
      hasRightExtension = true;
      callback(command + extension);
    });
  }

  extensionsStr = process.env.PATHEXT || '';
  extensions = [''].concat(extensionsStr.split(';'));
  for (i=0; !hasRightExtension && i<extensions.length;i++) {
    tryExtension(extensions[i]);
  }
}

Write Tests

Once you have grunt command name, you are ready to call spawn. Spawn fires exactly the same events as fork, so callGruntfile accepts exactly the same callbacks object and binds its properties to child process events:

function callGruntfile(command, filename, callbacks) {
  var comArg, options, child;
  callbacks = callbacks || {};

  comArg = ["--gruntfile", filename, "--no-color"];
  options = {cwd: 'test/'};
  child = cp.spawn(command, comArg, options);
  
  if (callbacks.onProcessError) {
    child.on("error", callbacks.onProcessError);
  }
  /* ... callbacks binding exactly as in fork ...*/
}

Tests are also almost the same as those in the previous chapter. Only difference is that you have to find the grunt executable filename before doing everything else. Success scenario test looks like this:

pass: function(test) {
  var wasPassMessage = false;
  test.expect(2);
  findGruntFilename(function(gruntCommand){
    var callbacks = {
      /* ... callbacks look exactly the same way as in fork ... */
    };
    callGruntfile(gruntCommand, 'gruntfile-pass.js', callbacks);
  });
}

Full success scenario test along with both failure scenarios tests are available on github.

Disadvantages

Disadvantages:

• Spawn ignores PATHEXT suffixes, custom code to handle it is needed.
• Child process output streams are available in chunks instead of one big block.

Conclusion

There are three ways how to test grunt plugin from inside gruntfile. Unless you have very strong reason not to, use exec.