Friday, January 6, 2012

JavaScript Testing with JSTestDriver

Js-test-driver is an open source JavaScript unit tests runner written in Java. The project was started at Google and is under active development. It is available under Apache License 2.0 license.

Js-test-driver is able to run from command line and reports results to the standard output. As a result, it is possible to fully automate JavaScript tests and run them on a continuous integration server.

Additionally, js-test-driver comes with IntelliJ IDEA plugin and Eclipse plugin. Of course, Eclipse plugin is compatible with Aptana Studio. If you use one of these IDEs, you can run tests directly from IDE and see progress and results in a view. It is very similar to java jUnit plugin.

This article starts with an overview of js-test-drivers architecture. Next three chapters explain configuration, how to run js-test-driver from the command line and how to run it from Eclipse. The longest chapter explains how to write tests and shows most important js-test-drivers features. We will also show how to debug tests in Firefox and Chrome. Finally, the conclusion contains our opinions on js-test-driver.

All used examples are available on Github.

Table of Contents


Overview

Js-test-driver testing has three components:
  • one or more web browsers,
  • small web server,
  • client that invokes tests.

Js-test-driver tests are written in JavaScript and run inside web browser. They are compatible with all major browsers. The tool is able to run them simultaneously in multiple browsers.

The tests are served to browsers by a small web server. The server may run either on the local machine or elsewhere. It does not matter too much. Each browser opens servers URL and server sends him a bunch of js-test-drivers JavaScript files. Those JavaScripts then periodically check the server for new commands.

Finally, the tests are invoked either from command line or from a plugin. In both cases, the client sends data to the server which forwards them to all active browsers. Js-test-drivers JavaScripts running in the browsers invoke tests and send their results back to the server. The server then forwards all results to the client.

Project Configuration

Unless specified otherwise, js-test-driver expects configuration in current working directory in a file named jsTestDriver.conf.

At minimum, the configuration file must contain:
  • list of JavaScript files under test in section load,
  • list of JavaScript files containing test cases in section test.

Unless specified otherwise, file paths are relative to the configuration file. Use '../' to climb up to parent directories. Files are loaded in the same order as specified in the configuration file.

Wildcard '*' is allowed. Files matching the wildcard are loaded in alphabetical order.

The number of spaces at the start of each line does matter. All sections elements in configuration file must be perfectly aligned. Tabs are not allowed.

For example, a project may consist of three files src/Utilities.js, src/Core.js and src/Main.js which have to be loaded in that order. All tests are located in src-test directory and the file TestsUtils.js must be loaded first. The configuration file would look like this:
load:
  - src/Utilities.js
  - src/Core.js
  - src/Main.js
  
test:
  - src-test/TestsUtils.js
  - src-test/*.js

All configuration file options are described on js-test-drives wiki page.

Running Tests - Command Line

This chapter explains how to run tests from the command line. If you do not want to use js-test-driver from the command line, skip to the next chapter.

To run the tests, start the server and connect browsers to it. Once it is done, use js-test-driver client to run test cases. You do not have to restart the server in between test runs, it may run permanently.

Start Server
Use the command java -jar JsTestDriver.jar --port <PORT_NUMBER> to start the server. The parameter port is mandatory, the server will listen on it. The rest of this chapter assumes that you used the port number 42442:
java -jar JsTestDriver.jar --port 42442

Capture Browser
Open the browser and put http://<servers ip address>:<PORT_NUMBER>/capture into URL bar. E.g. if you run the server on the local machine and used port number 42442, the URL contains:
http://127.0.0.1:42442/capture

The browser is now captured. Its top should have green color and contain something like this:
JsTestDriver
Last:1323466357645 | Next:1975 | Server:Waiting...

Start Both Server and Browsers
If you do not want to manually open each browser and edit its url, use the parameter --browser <browser 1>,<browser 2>,..., <browser n>. It will start each listed browser and fill correct capture URL into it.

For example, if Firefox, Chrome and Internet Explorer are installed on:
  • C:\Program Files (x86)\Mozilla Firefox\firefox.exe
  • C:\Users\AppData\Local\Google\Chrome\Application\chrome.exe
  • C:\Program Files\Internet Explorer\iexplore.exe
then you have to use the command:
java -jar JsTestDriver.jar --port 42442 --browser "C:\Program Files (x86)\Mozilla Firefox\firefox.exe",C:\Users\AppData\Local\Google\Chrome\Application\chrome.exe,"C:\Program Files\Internet Explorer\iexplore.exe

It will start the server and open all three web browsers. Each browser will be connected to the server.

Run Tests
The simplest possible way to run tests is the command java -jar JsTestDriver.jar --server <servers address:port> --tests all . It will read jsTestDriver.conf file in the current directory, send all configured files to the browser and run all tests.

If the server runs on port 42442 of your local machine, use this to run all tests:
java -jar JsTestDriver.jar --server http://127.0.0.1:42442 --tests all

Other Options
Of course, JsTestDriver has more command line options. For example, it is possible to run only some selected test cases instead of all of them. It is also possible to specify configuration file location and name.

All command line options are available on the projects wiki page.

Running Tests - Eclipse

The general workflow in Eclipse is the same as in the command line. You have to:
  • start and stop server,
  • start browsers and connect them to the server,
  • run tests and show results in a window.

However, you have to install and configure it first. If you do not want to use js-test-driver from the Eclipse, skip to the next chapter.

Installation
Open Eclipse and go to 'Help' -> 'Install new Software'. Click 'Add' button and add http://js-test-driver.googlecode.com/svn/update/ update site.

Open new update site and expand all categories. At the time of writing this post, there are two categories:
  • 'JS Test Driver Plugin for Eclipse' with version 1.1.1.c inside,
  • 'Uncategorized' with version 1.1.1.e inside.

Choose and install the version 1.1.1.e. The older c version had some bugs that prevent it from working.

If there is an even newer version, install that one. The plugin is a work in progress and the version 1.1.1.e have known bugs which are already fixed on the trunk.

Configuration
You have to configure two things to use js-test-driver plugin:
  • plugin itself,
  • launch configuration for the project.

To configure the plugin, go to 'Window' -> 'Preferences' and open 'Js Test Driver' preference page. Configure paths to browsers. That's it.

You have to create a separate launch configuration for each project you want to test with js-test-driver. Right click on the project, click 'Run As' -> 'Run Configurations ...'. Create new 'Js Test Driver Test' launch configuration and fill both 'Project' and 'Conf. file' fields.

Note: If you try to run tests and the launch configuration does not exists, the 1.1.1.e version does nothing. Future versions will automatically create new launch configuration, pre-fill fields and open it to edit.

Usage
Again, you have to start the server and connect browsers to it. Open Js Test Driver view: click 'Window' -> 'Show View' -> 'Other...' -> 'JsTestDriver'.

Start the server: The leftmost icon on the toolbar, the green arrow, starts the server. Press it.

Capture browsers: View top contains greyed browsers icons. Press any of them to open and capture selected browser.

Run tests: Run the project launch configuration to run all its tests. The view will show results in a tree grouped by hosting browsers. It is not possible to run only some subset of tests. The shortcut is available in various menus, but does nothing.

If you check the 'Run on Every Save' in the launch configuration, js-test-driver will run tests each time any file in the project is saved. The option was too much for me, but some people may find it useful.

Usage - Future Versions
It will be possible to run only some subsets of tests. There are three ways how to do it.

First, select one or more .js files with tests in package explorer, right click and choose 'Run As' -> 'Js Test Driver Test'. The plugin will run all tests defined in selected files.

Alternatively, you can open any .js file in an editor, right click and choose 'Run As' -> 'Js Test Driver Test'. The plugin will run all tests defined in the opened file.

Finally, highlight any test declaration in an editor, right click and choose 'Run As' -> 'Js Test Driver Test'. The plugin will run only that one test.

Writing Tests

This chapter shows how to write JavaScript unit tests. We will create simplest possible test case that declare some tests but does nothing. Then we will explain how assertions work and use them to show how js-test-driver cleans objects properties in between tests.

Final four sub-chapters explain logging, declaration of HTML in tests, dealing with events and asynchronous functions and how to load other text-based resources into tests.

All examples used in this chapter are available on Github.

Declare Tests
Use the function TestCase("TestCaseName") to create test case objects. It is not a constructor function, there is no new. The object name must be the same as the name supplied to the method. Any test case contains:
  • an optional function setUp - runs before each test case,
  • an optional function tearDown - runs after each test case,
  • any number of test methods - their names must begin with test.

All needed objects and functions are automatically loaded into the browser by js-test-driver server. You do not have to include them manually. However, if the file with test case contains syntax error, tests defined in it are ignored.

Following piece of code defines a test case with two tests. The tests are empty:
TestCaseDemo = TestCase("TestCaseDemo");

TestCaseDemo.prototype.setUp = function(){
  //initialization - before each test
};

TestCaseDemo.prototype.testFirst = function(){
  //the test
};

TestCaseDemo.prototype.testSecond = function(){
  //the test
};

TestCaseDemo.prototype.tearDown = function(){
  //clean up - after each test
};


Note: It is not possible to create a test suite.

Assertions
Js-test-driver comes with a respectable list of assertions you can use in your test cases. Each assertion has an optional argument msg which is returned as an error message if assertion fails.

Following test contains three test cases. First one always passes, second and third always fail:
AssertionsTestCase = TestCase("AssertionsTestCase");
 
AssertionsTestCase.prototype.testAlwaysPass = function(){
  var expected = 1, actual = 1;
  assertEquals("The vales should be the same", expected, actual);
  assertEquals(expected, actual);
 
  var myStr = "hello";
  assertString("The variable should contain a string", myStr);
  assertString(myStr);
};
 
AssertionsTestCase.prototype.testAlwaysFail = function(){
  assertEquals(1, 2);
};

AssertionsTestCase.prototype.testAlwaysFailWithMessage = function(){
  assertEquals("1<>2", 1, 2);
};

Assertions list linked above does not contain expectAsserts(count) assertion. This assertion tells the JsTestDriver how many asserts to expect within a test. It is useful when you need to check whether function under test called callback expected number of times:
functionUnderTest = function(number, callback) {
  for (var i = 0; i < number; i++) {
    callback(i);
  }
};

AssertionsTestCase.prototype.testExpectedCalls = function(){
  var parameterCheck = function(number) {
    assertNumber(number);
  };

  expectAsserts(3);
  functionUnderTest(3, parameterCheck);
};

Consult list of assertions to find out about all available assertions.

Properties Scopes
Properties scope is important for most non-trivial test cases. All tests should run independently of each other, so js-test-driver cleans all test case objects properties in between tests.

Each test has access to properties created in set up method. Tear down has access to properties created in both set up and test method. After tear down, js-test-driver cleans up all properties from the object and next set up method starts with an empty object again.

Next test case creates one property in set up, one in tests and one in tear down. Each method also verifies whether only appropriate properties are available:
ScopeTest = TestCase("ScopeTest");

ScopeTest.prototype.setUp = function() {
 assertUndefined("inSetUp", this.inSetUp);
 assertUndefined("inTest", this.inTest);
 assertUndefined("inTearDown", this.inTearDown);

 this.inSetUp = 1;
};

ScopeTest.prototype.testVariables = function() {
 assertEquals("inSetUp value", this.inSetUp, 1);
 assertUndefined("inTest", this.inTest);
 assertUndefined("inTearDown", this.inTearDown);

 this.inTest = 1;
};

ScopeTest.prototype.testVariables2 = function() {
 assertEquals("inSetUp value", this.inSetUp, 1);
 assertUndefined("inTest", this.inTest);
 assertUndefined("inTearDown", this.inTearDown);

 this.inTest = 1;
};

ScopeTest.prototype.tearDown = function() {
 assertEquals("inSetUp value", this.inSetUp, 1);
 assertEquals("inTest value", this.inTest, 1);
 assertUndefined("inTearDown", this.inTearDown);

 this.inTearDown = 1;
};

Logging
Js-test-driver provides a jstestdriver.console object. It has five logging methods: error, warn, info, debug and log.

Each takes any number of parameters of any types and forwards their values to the standard output. E.g., tests run from the command line print log messages to the shell. Tests run from the Eclipse print log messages to the console view.

Following test case shows how to use it:
ConsoleTestCase = TestCase("ConsoleTestCase");

ConsoleTestCase.prototype.testLogging = function(){
  var myArray = ['one', 'two', 3, "four"], undefinedV;
  
  jstestdriver.console.log("This is log message.");
  jstestdriver.console.debug("Printing out the value ", 2);
  jstestdriver.console.info("Array content is: ", myArray, " ");
  jstestdriver.console.warn("The value is undefined: ", undefinedV);
  jstestdriver.console.error(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
};

Standard output contains:
[LOG] This is log message.
[DEBUG] Printing out the value  2
[INFO] Array content is:  ["one","two",3,"four"]  
[WARN] The value is undefined:  undefined
[ERROR] 1 2 3 4 5 6 7 8 9 10

HTML DOM Interactions
Most JavaScript libraries interact with surrounding HTML document. Js-test-driver provides two different ways to create document object model needed for tests.

First, you may create it with JavaScript DOM API calls. This method is straightforward, but may result in long, difficult to read code. Additionally, created objects stay inside the document until you clean it.

Second, you can declare HTML inside doc comments and js-test-driver will do the rest. This method results in easier to maintain and read code.

JavaScript
JavaScript can create HTML objects in the test function or inside set up function. In either case, it is important to clean up all created objects from the document.

Next example creates HTML structure in the setUp function and cleans it in the tearDown. New structure is available in all tests:
JSDefinedHtml = TestCase("JSDefinedHtml");
 
JSDefinedHtml.prototype.setUp = function() {
 //create a new div and store it in objects variable
 this.mainDiv = document.createElement('div');
 this.mainDiv.id = 'main';
     
 //create inner div
 var innerDiv = document.createElement('div');
 this.mainDiv.appendChild(innerDiv);
 innerDiv.className = 'text';
 innerDiv.innerHTML = "Hello word.";
 
 //add whole thing to document body
 document.body.appendChild(this.mainDiv);
};
 
JSDefinedHtml.prototype.testHtml = function() {
 var mainDiv = document.getElementById('main');
 var child = mainDiv.childNodes[0];
  
 //stored object is the same as the one found in the document
 assertEquals(this.mainDiv, mainDiv);
 assertEquals("text", child.className);
 assertEquals("Hello word.", child.innerHTML);
};
 
JSDefinedHtml.prototype.tearDown = function() {
 //clean up document
 var mainDiv = document.getElementById('main');
 document.body.removeChild(mainDiv);
};

Declarative
Js-test-driver provides an alternative, easier to read and maintain way to add HTML into tests. Put valid HTML inside a doc comment and js-test-driver will place corresponding DOM into a variable or attach it to document body. Both document body and variable are automatically cleaned between tests.

All examples in this sub-chapter use an utility function findFirstDiv(element). The function is defined in TestUtils and returns first child div of supplied element:
TestUtils.prototype.findFirstDiv = function(element) {
  var result, child ;
  
  for ( i = 0; i < element.childNodes.length; i++) {
    child = element.childNodes[i];
    if (child.nodeType == 1 && child.tagName=='DIV')
      result = child;
  }
  
  return result;
};

The declaration /*:DOC <VARIABLE> = <HTML> */ creates a DOM corresponding to HTML and stores it in a variable. Next snippet creates two divs and puts them into the mainDiv variable:
HtmlDeclaredInTest = TestCase("HtmlDeclaredInTest");

HtmlDeclaredInTest.prototype.setUp = function() {
  //test utils object contains convenience functions
  this.testUtils = new TestUtils();
};

HtmlDeclaredInTest.prototype.testVariableHtml = function() {
 /*:DOC mainDiv = <div id='main'>
                    <div class='text'>Hello word.</div>
                  </div> */
  var child = this.testUtils.findFirstDiv(this.mainDiv);

  assertEquals("text", child.className);
  assertEquals("Hello word.", child.innerHTML);
};


The declaration /*:DOC += <HTML> */ creates a DOM corresponding to HTML and attaches it to document body. Next example creates two divs and attaches them to the body:
HtmlDeclaredInTest.prototype.testDocumentHtml = function() {
 /*:DOC += <div id='attached'>
             <div class='text'>Attached to document.</div>
           </div>*/

  var attachedDiv = document.getElementById('attached');
  var child = this.testUtils.findFirstDiv(attachedDiv);
    
  assertEquals("text", child.className);
  assertEquals("Attached to document.", child.innerHTML);
};

Both variable and document body are cleaned up between tests:
HtmlDeclaredInTest.prototype.testDocumentHtmlScope = function() {
  //This test method make sense only if it runs last. Theoretically,
  //it is impossible to guarantee order in which test 
  //methods run. If this method does not run last in your set up,
  //try to tweak its name.
  var attachedDiv = document.getElementById('attached');
  assertNull("Document should be clean.", attachedDiv);
    
  assertUndefined("Main div should be clean.", this.mainDiv);
};

If each test in test case needs to work with the same DOM/HTML, declare it in set up function:
HtmlDeclaredInSetUp = TestCase("HtmlDeclaredInSetUp");

HtmlDeclaredInSetUp.prototype.setUp = function() {
 //create a new div and store it in objects variable
 /*:DOC mainDiv = <div id='main'><div class='text'>Hello word.</div></div>*/

 //create a new div and append it to document
 /*:DOC += <div id='attached'><div class='text'>Attached to document.</div></div>*/

  this.testUtils = new TestUtils();
};

HtmlDeclaredInSetUp.prototype.testVariableHtml = function() {
  var child = this.testUtils.findFirstDiv(this.mainDiv);
  
  assertEquals("text", child.className);
  assertEquals("Hello word.", child.innerHTML);
  assertNull("Variable-only DOM", document.getElementById('main'));
};
 
HtmlDeclaredInSetUp.prototype.testDocumentHtml = function() {
  var attachedDiv = document.getElementById('attached');
  var child = this.testUtils.findFirstDiv(attachedDiv);
  
  assertEquals("text", child.className);
  assertEquals("Attached to document.", child.innerHTML);
};

Events and Asynchronous Functions
JavaScript programming tend to be asynchronous and event based. JavaScript functions often set up timers to delay real work or to periodically check the state.

Such function returns as soon as possible. All important work is done asynchronously in the background. The function takes callback function as a parameter and calls it whenever the work is done or an interesting event happens.

Few examples of asynchronous JavaScript programming:
  • all AJAX operations,
  • waiting for a browser action,
  • anything done in timers.

A simplified asynchronous function may look like this:
/**
 * Works asynchronously and calls the callback when everything is done.
 * 
 * @param callback called when the work is done 
 * @return nothing
 */
AsyncEvent.prototype.asynchronousFunction = function(callback) {
    //the timer simulates an asynchronous operation 
    window.setTimeout(callback, 2000);
};

As the function under test returns before the work is done, standard TestCase used in all previous examples is unable to test it. Instead, you have to use slightly more complicated AsyncTestCase.

Split your test case into multiple steps. Each step is represented by a function. Add all steps/functions to queue which is passed to your test as a parameter. In general, an asynchronous test case looks like this:
AsynchronousTest = AsyncTestCase("AsynchronousTest");
 
AsynchronousTest.prototype.testSteps = function(queue){
 //define common variables here
 var something = 'hello';
 
 queue.call('Step 1: initialization', function(callbacks) {
  //first step code goes here
 });

 queue.call('Step 2: do something', function(callbacks) {
  //second step code goes here
 });

 // ... 
 
 queue.call('Step n: final checks', function(callbacks) {
  //last step code goes here
 });
 
};

Js-test-driver runs those steps in the order they have been submitted to the queue. The next step starts only after previous one fully finished and all its callback functions have been called.

Each step function has one parameter called callbacks. The object passed to this parameter defines add(callback) function. It registers callback and returns wrapper around it. It is important to use wrapper instead of original callback in all subsequent code.

Important: js-test-driver does not wait for non-registered callbacks. They are ignored. On the other hand, if you fail to use wrapper instead of original callback, js-test-driver will not know that callback has been called. The step will run for 30 seconds and then fail.

Following step takes one second to finish:
var callback = function() {};

 queue.call('Step 1: run for 1 second', function(callbacks) {
     //register callback function
     var callbackWrapper = callbacks.add(callback);
     //set one second timer on the wrapper
     window.setTimeout(callbackWrapper, 1000);
 });

We will create test for previously defined simplified asynchronous function:
/**
 * Works asynchronously and calls the callback when everything is done.
 * 
 * @param callback called when the work is done 
 * @return nothing
 */
AsyncEvent.prototype.asynchronousFunction = function(callback) {
    //the timer simulates an asynchronous operation 
    window.setTimeout(callback, 2000);
};

Our test adds two steps into the queue. First step sets up a callback and passes it to the asynchronous function. Second step verifies whether callback run:
AsynchronousTest.prototype.testAsynchronousFunction = function(queue){
 //create our callback function
 var callbackFinished = false,
 callbackFunction = function() {
     callbackFinished = true;
 };

 // set up asynchronous test 
 queue.call('Step 1: set up', function(callbacks) {
     // js-test-driver wraps our callback and creates a 'real' callback
     var callbackWrapper = callbacks.add(callbackFunction);
     
     // pass wrapped callback to the function under test
     var testee = new AsyncEvent();
     testee.asynchronousFunction(callbackWrapper);
 });

 // check whether callback function has been called
 queue.call('Step 2: assert callback', function() {
     assertEquals(true, callbackFinished);
 });

};

Other Resources
Some tests may need access to additional resources. Js-test-drivers server is able to load css, js, xml, html and rdf files and serve them to tests. Use the serve section of the configuration file to list all additional files. They will be available under /test/path/filename url.

The main difference against previously discussed load and test sections is that each file listed in them is automatically loaded into all captured browsers. The files listed in serve section are served only if explicitly asked for.

Following configuration loads src-test/cssresource.css file into the server:
serve:
  - src-test/cssresource.css

The file cssresource.css is now available under /test/src-test/cssresource.css url.

The css file itself is very simple, it changes any divs color to red:
div {
  color: #FF0000;
}

We will create a test that:
  • creates new div and attaches it to the document,
  • check the divs color - should be undefined,
  • loads cssresource.css,
  • check the divs color - should be red.

Dynamic css file loading and the function that returns HTML elements color are defined in TestUtils.js file. As their content is long and unrelated to js-test-driver, we show only their API here. Full implementation is available on Github.
/**
 * Loads an external css sheet into brower. The callback function is 
 * called after the sheet was loaded and applied to the DOM.
 * 
 * Note: the detection on when is css really loaded is browser dependent 
 * Discussion: http://www.phpied.com/when-is-a-stylesheet-really-loaded/
 * 
 * @param src full filename, including path
 * @param callback to be called after the work was done
 * 
 * @return link element used to load the script 
 */
TestUtils.prototype.loadCss = function(src, callback) {
  ...
};

/**
 * Finds current visible color of the element and returns it
 * in hex format. 
 * 
 * Hex format: #rgb with all three values in hex
 *  
 * @param element to be investigated
 * @return elements color in hex format 
 */
TestUtils.prototype.elementColor = function(element) {
  ...
};

As the function loadCss is asynchronous, we have to use AsyncTestCase:
LoadResourceTest = AsyncTestCase("LoadResourceTest");
 
LoadResourceTest.prototype.setUp = function(){
  this.testUtils = new TestUtils();
};

The test function attaches new div to html and adds three steps to the queue. First step only verifies divs color. Second step calls the synchronous loadCss function and waits for the callback to be called. The final step checks wheter divs color changed to red:
LoadResourceTest.prototype.testLoadCss = function(queue){
  // define HTML and common variables
  /*:DOC += 
I am colored.
*/ var documentDiv = document.getElementById('colored'); var callbackFunction = function() { }; queue.call('Step 1: no css sheet available', function(callbacks) { // css sheet is not available yet assertEquals('', documentDiv.style.color); }); queue.call('Step 2: load css sheet', function(callbacks) { // wrap the callback var callbackWrapper = callbacks.add(callbackFunction); // asynchronously load css this.link = this.testUtils.loadCss("/test/src-test/cssresource.css", callbackWrapper); }); queue.call('Step 3: css is already applied', function() { // elements color should change to red var currentColor = this.testUtils.elementColor(documentDiv); assertEquals('#ff0000', currentColor); }); };

The tear down function cleans up css loading elements from the HTML:
LoadResourceTest.prototype.tearDown = function(){
 //clean up
 var head = document.getElementsByTagName('head')[0];
 if (this.link) {
  head.removeChild(this.link);
 }
};

Note: js-test-drivers wiki claims that the serve section supports also images. That is incorrect.

Debugging Tests - Firefox

The most popular JavaScript debugger available for Firefox is Firebug. If you do not have it, install and enable it.

Start js-test-driver and load js-test-driver tests into firefox. Firebugs scripts panel must be active while tests are loading:
  • Start js-test-driver server and capture Firefox browser either from the command line or from the eclipse plugin.
  • Go to captured tab and click 'Tools' -> 'Web Developer' -> 'Firebug' -> 'Open Firebug'. Firebug will open in bottom half of the screen.
  • Click on the 'Scripts' tab. It will activate the Scripts panel:
    Script Panel was inactive during page load
    Reload to see all sources
    
  • Run tests in js-test-driver. You can do it either from the command line or from the eclipse plugin.
  • Refresh scripts panel. Click on the HTML panel and then again on the script panel.

If you see the message No Javascript on this page, wait for a second or two. JavaScripts have not been loaded yet. Alternatively, go to the HTML panel and locate 'runner' frame in the source code. That should do it.

Js-test-drivers test page is now loaded in firebug. Generated HTML is available in the HTML panel and scripts panel contains all loaded JavaScript files.

Debug tests in the scripts panel:
  • Click third button from the left on the toolbar.
  • Select JavaScript file containing the test.
  • Click line number to place breakpoint.
  • Right click on the breakpoint to add condition to it.
  • Rerun tests - execution will stop on the line with breakpoint.

All firebugs features are now available:
  • Hover over variable/property to see its value.
  • Add variables and expressions to watch view to see their values.
  • Investigate stack trace to see who called which function and why.
  • Investigate, temporary disable/enable or remove breakpoints in the Breakpoints view.

Of course, we listed only most important Firebugs features. Read its documentation or one of numerous tutorials to find out more.

Debugging Tests - Chrome

Chrome comes with built-in debugger, no installation is necessary.

Start js-test-driver and load js-test-driver tests into Chrome:
  • Start js-test-driver server and capture Chrome browser either from the command line or from the eclipse plugin.
  • Go to captured tab and click 'Tool icon' -> 'Tools' -> 'Developer tools'. Chrome developer tools will open in bottom half of the screen.
  • Click on the 'Scripts' tab. It will activate the Scripts panel.
  • Run tests in js-test-driver. You can do it either from the command line or from the eclipse plugin.

Js-test-drivers test page is now loaded in Chrome. Generated HTML is available in the elements panel and scripts panel contains all loaded JavaScript files.

Debug tests in the scripts panel:
  • Click third button from the left on the toolbar.
  • Select JavaScript file containing the test.
  • Click line number to place breakpoint.
  • Right click on the breakpoint and choose 'Edit breakpoint...' to add condition to it.
  • Rerun tests - execution will stop on the line with breakpoint.

All chrome features are now available:
  • Add variables and expressions to watch view to see their values.
  • Investigate stack trace to see who called which function and why.
  • Investigate, temporary disable/enable or remove breakpoints in the Breakpoints box.

Again, we listed only most important Chrome developer tools features. Read its documentation or one of numerous tutorials to find out more.

Continuous Integration and Maven

Integration with Jenkins continuous integration server (previously known as Hudson) was described by Christian Johansen on his blog.

Maven integration can be achieved with jstd-maven-plugin. The plugin assumes that the server with captured browsers is already running somewhere. It is not able to start the server by itself.

Would be Great

It would be great if js-test-driver would be able to run purely from maven or ant task with no further configuration. The only part of js-test-driver life-cycle that is system configuration dependent is the browser.

That requires web browser embeddable in a java runner. We spend a half day trying to make it happen. We have not succeeded and this chapter contains log of our attempts.

We tried two different approaches:
  • using HTMLUnit,
  • using combination of Rhino and env.js.

HtmlUnit
HtmlUnit is the most popular java headless browser. Unfortunately, newest version is not compatible with js-test-driver. Timers are called and running, js-test-drivers hearth is beating. However, for a reason we have not found yet, HtmlUnit waits for servers command forever and seems to ignore 'run tests' command.

Rhino + Envjs
Rhino is an open source JavaScript parser and interpreter written in Java. It has shell, debugger, compiler and parser into abstract syntax tree. It does not have browser specific object and APIs.

Envjs is a pure JavaScript library which simulates browser environment. Load it into the Rhino and you have created a functional headless browser. It is easy and works for most purposes.

Unfortunately, js-test-driver relies heavily on timers and latest version 1.2 of envjs has bug that prevents them to work properly. Envjs version 1.3 is still in preview phase.

Conclusion

Main js-test-driver features are exactly what we were looking for. We wanted to write small, self contained tests to validate JavaScript behavior. We wanted to run them from continuous integration tool. Cross browser testing with js-test-driver is easy and comfortable which is what we needed.

While command line works as expected, Eclipse plugin is still in beta. It is usable, but it has some missing features. More importantly, it has some bugs that need to be fixed. The good news is that its source code is fairly easy to understand. Plus, its active maintainers are responsive to issues and patches.

There is not too much work remaining to have it polished. Most core work have been already done. The plugin seems to be very promising, just little rough on the edges for now.

Our last complain is the documentation. There is not much of it, and what was written provides only minimum information. The tool has great features and some caveats, but you have to dig inside to find them. You cant find them neither on wiki nor anywhere else. Then again, js-test-drivers team answers questions raised in their Google group or in issues list fast and fully.

19 comments:

Anonymous said...

Fantastic write-up! The main problem inhibiting use of JSTD is the lack of documentation. The information on the wiki is much lacking and out-of-date. This post patches part of that void.

One issue I've found is that any global variables set by one test (by the code under test) are persisted into the next test. This can easily make tests dependent on each other, which unit tests should not be. Is there any clean solution to this without explicitly clearing the variables in tearDown, or do you just consider that kind of code faulty any way?

Meri said...

Hi,

thank you :).

Generally, it is better to avoid global variables. But, they are commonly used and it is not possible to refactor every single library that uses them. I guess we just have to live with them.

The only idea I had so far:

If it would be possible to list all global variables, it would be possible to clean them. You would store a list of global variables in the setUp method. The tear down would read it again and compare with the stored version. It would remove what is new and does not belong to JsTestDriver.

Unfortunately, there may not be a reliable way to list all global variables. This thread contains a solution that should work in Firefox and Chrome, but I think that it will not work in IE. I did not had a time to try it, so maybe you will be lucky.

If it works in at least one browser, you may implement it. If test results in that one browser are different from results in other browsers, that hints to some global variables problem.

It is not really elegant. I will think about it more, but I doubt that there will be an easy solution.

Javascript Keyboard said...

Hello Meri...
By the way I would recommend you using the .data() method to attach this state as metadata of the form instead of polluting your global scope with global variables. The problem is im not a javascript expert, to be honest i jst started doing JavaScript about two days ago, that is why im in need for help.

Cheers!!
Jackie

dicasdolampada said...

That is a KICK ASS article.
Thanks a lot. Will be very useful.

Anonymous said...

great article! what about mocking? iv'e seen recommendations about sinon.js, is there a built in capability in js-test-driver?

Meri said...

Hi Anonymous,

as far as I know, there is no such support. Js-test-driver is mostly a test runner. If you need mocks, advanced asserts or something similar, the best is to combine it with other library.

Good luck,
Meri

Lior said...

Hi Meri, thanks for the article, it's been very helpful.

I've been trying to write an async test using the example you provided here.
In my first call to the queue I need to activate an initialization function, which is asyncronic (it uses $.getScript) and in the 2nd queue call I want to check using assert that the script loaded.

It seems that the assert in the 2nd queue call gets called before the initialization function finishes (it itself has a callback and when I place the assert in that callback, it works).
It just seems like the queue.call doesn't wait for the callback from the previous queue.call to return.

I also tried to define an async function using AsyncEvent.prototype.asynchronousFunction but it says that AsyncEvent is undefined.

Any help would be greatly appreciated!

Thanks

Meri said...

Hi Lior,

your problem sounds like an interesting puzzle. Could you please post simplified version of the code that does not work?

It would be easier for me if I had a working example to look at.

With Regards,
Meri

realjoker said...

Great tutorial, thanks!
I was wondering: is there a good reason why assertEquals ("this should fail","000011",11) does not fail?
Thanks

Meri said...

Hi realjoker,

assertEquals uses compare_ function to compare expected with actual value. You can find it here.

Basically, if the operator === returns false and at least one value is not an object, the == operator is used. If both values are objects, then the function compares properties of both objects.

Both string and number in your case are not objects, therefore they end up compared with ==. JavaScript == operator converts different data types to the same type before making the comparison, so the case evaluates to true.

I assume that assertEquals was meant to work like == operator with added special logic for objects.

What you want is === like behavior. I guess that there are use cases for both, both are "standard" in JavaScript :).

Hope that helps,
Meri

realjoker said...

Thanks for your answer.

I had seen the compare_ function before posting and what I was wondering is the "design" reasoning behind the implementation of such behaviour. I mean, why has JSTD team decided to implement it that way?

Another few questions:
1. how to install/use the JSTD coverage plug-in in eclipse?
2. is there a way to clear the Eclipse console within a test method? (i.e.: clean the console within the setUp function of a test suite)
3. can javascript code written inside JSP files be tested or it must first be put in js files?

Thanks in advance

Meri said...

Hi realjoker,

I do not know much about JsTestDriver team motivations and decisions. I know only what they wrote on thir page or google group. I do not recall to read something about this particular function.

1. I do not know, I did not tried.
2. I do not think so. As far as I know, the plugin does not manipulate the console beside necessary "add standard output".
3. JsTestDriver does not have jsp compiler in it, so I think that it is not possible. It is probably possible to write such plugin or otherwise integrate those two technologies, but as far as I know you would be first to do that.

Sorry for not helping you more :(,
Meri

Phil Caithness said...

For mocking, use Sinon.
We are using jsTestDriver and Sinon together and so far so good.
Sinon will give you a bunch of other goodies as well.

Martin Jacobs said...

Thanks for such a good tutorial.

Got it working with the GreeterTest tutorial given by the JsTestDriver site, by trial and error. However, I don't fully understand why, and there seem to be some anomalies.

Some notes on using it in Eclipse ...

1 Documentation is very poor (as noted above), which is frustrating for noobs like me

2 Start server

When I copied to terminal
java -jar JsTestDriver.jar --port 42442 I got the error message that it could not access JsTestDriver.jar. I can't find a file called JsTestDriver.jar on my system.

I could not resolve this, so I gave up and went back to Eclipse. If this means I missed a vital step, I'd appreciate hearing about it, and how to put it right.

3 JsTestDriver console and browser capure

The console should start with a red line and "NOT RUNNING", so click the green arrow. Line goes yellow with the message http://127.0.0.1:42442/capture. Click on the icons for the browser apps you want to capture (provided the paths are specified in Eclipse -> Preferences -> JS Test Driver)

In my case, clicking on the icon launched the browser app with a green line along the top with the message

JsTestDriver
Last:1356609704008 | Next:1956 | Server:Waiting...


It also shows the page only partly loaded. Unless I am mistaken, it's OK to leave the browser as it is (in a partially loaded state), which doesn't seem right, but it seems to work. I don't know if there's an error here (something to do with my security settings?).

The JsTestDriver console should now show green and at least one of the browser icons should be lit up.

4 JsTestDriver.conf

I could not get any tests to run (I'd always get Run : 0/1) with

server:http://localhost:42442

load:
- src/*.js
- src-test/*.js


... but I did run the tests with


server:http://localhost:42442

load:
- src/*.js
- src-test/*.js

test:
- src/*.js
- src-test/*.js


timeout: 10


With this last version, the tests passed or failed as expected.

If anyone can shed any light on how this is working, or how to make it work better, I'd appreciate it.

Tom said...

Great article, the AsyncTestCase and -serve functionality are new to me, and every example was useful and clear. For running the tests against a headless browser try www.browserstack.com Best part is that you can run your tests against most browser versions at once.

Meri said...

@Martin Jacobs

2 Start server
You need to download jar file from JsTestDriver test page. It has version number in a name, so you have to you something like this: java -jar JsTestDriver-<version number>.jar --port 42442 .

3 JsTestDriver console and browser capure
I think that the half loaded state is OK. At that point, the framework is loaded, but tests are not. The browser is waiting for tests to come.

4 JsTestDriver.conf
Your first version of configuration file loads all tests into the browser, but the framework does not know they are tests. It considers them to be part of the source code, so it does not run them.

Only files inside "test" section are considered test cases.

hemcoined said...

Additionally, created objects stay inside the document until you clean it.
http://www.hemcoind.com/loading-arms.html

shraddha said...

Great Article!

Ankit Raturi said...
This comment has been removed by the author.

Post a Comment