jUnit: Rules

Rules add special handling around tests, test cases or test suites. They can do additional validations common for all tests in the class, concurrently run multiple test instances, set up resources before each test or test case and tear them down afterwards.

The rule gets complete control over what will done with the test method, test case or test suite it is applied to. Complete control means that the rule decides what to do before and after running it and how to deal with thrown exceptions.

First chapter shows how to use rules and second shows what build-in rules can do. The third chapter describes third party rules libraries I found and the last one explains how to create new rules.

Table Of Contents

• Using Rules
• Example Rule - Temporary Folder
• Declaring Test Rules
• Declaring Class Rules
• Using Rules Inside Tests
• Default Rules
• Expected Exception
• Timeout
• Error Collector
• Test Name
• Third Party Rules Libraries
• Custom Rule
• New Rule
• From Scratch
• Full Class
• Extending Build-in Classes
• External Resource
• Verifier
• More About jUnit

Using Rules

This chapter shows how to declare and use rules inside a test case. Most rules can be applied to each test method separately, once to the whole test case or once to the whole test suite. Rules run separately for each test are called test rules and rules applied to the whole test case or suite are called class rules.

We will use temporary folder rule as an example, so first subchapter explains what it does. Second subchapter declares it as test rule and third one as class rule. Last subchapter shows how to access the folder from inside the tests.

Example Rule - Temporary Folder

Temporary folder rule creates new empty folder, runs test or test case and then deletes the folder. You can either specify where to create the new folder, or let it be created in system temporary file directory.

Temporary folder can be used as both test rule and class rule.

Declaring Test Rules

Test rules e.g., rules that run for each test method separately, have to be declared in public field annotated with @Rule annotation.

Declare test rule:

public class SomeTestCase {
  @Rule
  public TemporaryFolder folder = new TemporaryFolder();
}

The above folder rule creates new folder before every test method and destroys it afterwards. All tests are able to use that directory, but they are not able to share files through it. Since we used constructor with no parameters, the folder will be created in system temporary file directory.

Test rule does its work before methods annotated with @Before and after those annotated with @After. Therefore, they will have access to temporary folder too.

Declaring Class Rules

Class rules e.g., rules that run once for the whole test case or test suite, have to be declared in public static field and annotated with @ClassRule annotation.

Declare test case rule:

public class SomeTestCase {
  @ClassRule
  public static TemporaryFolder folder = new TemporaryFolder();
}

The above folder rule creates new folder before running the first test method and destroys it after the last one. All tests are able to use that directory and they are able to see files created be previously running tests.

Class rules are run before anything inside that class. E.g. methods annotated with @BeforeClass or @AfterClass will have access to temporary folder too. The rule runs before and after them.

Using Rules Inside Tests

Rules are classes as any other and tests are free to call their public methods and use their public fields. Those calls are used to add test specific configuration to the rule or read data out of it.

For example, temporary folder can be accessed using newFile, newFolder or getRoot methods. First two create new file or folder inside the temporary folder and the getRoot method returns temporary folder itself.

Create temporary file and folder:

@Test
public void test1() {
  // Create new folder inside temporary directory. Depending on how you 
  // declared the folder rule, the directory will be deleted either 
  // right after this test or when the last test in test case finishes.
  File file = folder.newFolder("folder");
}

@Test
public void test2() {
  // Create new file inside temporary folder. Depending on how you 
  // declared the folder rule, the file will be deleted either 
  // right after this test or when the last test in test case finishes.
  File file = folder.newFile("file.png");
}

Default Rules

JUnit comes with five directly useable rules: temporary folder, expected exception, timeout, error collector and test name. Temporary folder have been explained in previous chapter, so we will briefly explain only remaining four rules.

Expected Exception

Expected exception runs the test and catches any exception it throws. The rule is able to check whether the exception contains the right message, the right cause and whether it was thrown by the right line.

Expected exception has private constructor and must be initialized using static none method. Each exception throwing test has to configure expected exception parameters and then call the expect method of the rule. The rule fails if:

• the test throws any exception before the expect method call,
• the test does not throw an exception after the expect method call,
• thrown exception does not have the right message, class or cause.

The last test line throws an exception. Expected exception rule is configured right before causing the exception:

@Rule
public ExpectedException thrown= ExpectedException.none();

@Test
public void testException() {
  // Any exception thrown here causes failure
  doTheStuff();
  // From now on, the rule expects NullPointerException exception
  // to be thrown. If the test finishes without exception or if it 
  // throws wrong one, the rule will fail.
  thrown.expect(NullPointerException.class);
  // We well check also message
  thrown.expectMessage("Expected Message.");

  // this line is supposed to throw exception
  theCodeThatThrowsTheException();
}

Bonus: the expected message method accepts also hamcrest matcher argument. That allows you to test the message prefix, suffix, whether it matches some regular expressions or anything else.

Timeout

The timeout rule can be used as both test rule and class rule. If it is declared as test rule, it applies the same timeout limit to each test in the class. If it is declared as class rule, it applies the timeout limit to the whole test case or test suite.

Error Collector

Error collector allows you to run multiple checks inside the test and then report all their failures at once after the test ends.

Expected-vs-actual value assertions are evaluated using the checkThat method exposed by the rule. It accepts hamcrest matcher as an argument and thus can be used to check anything.

Unexpected exceptions can be reported directly using addError(Throwable error) method. Alternatively, if you have an instance of Callable to be run, you can call it through checkSucceeds method which adds any thrown exception into errors list.

Test Name

Test name rule exposes test name inside the test. It might be useful when you need to create custom error reporting.

Third Party Rules Libraries

Rules are decoupled from the test class, so it is easy to write libraries of general purpose rules and share them between projects. This chapter describes three such libraries.

System rules is rules collection for testing code that uses java.lang.System. It is well documented, available in maven and released under Common Public License 1.0 (the same as jUnit). System rules allows you to easily:

• test content of System.err and System.out,
• simulate input in System.in,
• configure system properties and revert their values back,
• test System.exit() calls - whether it was called and what return value was,
• customize java SecurityManager and revert it back.

A big set of useful rules is available on aisrael account on github. Its documentation is somewhat limited, but you can always look at the code. All rules are released under MIT license:

• starting and stopping in-memory derby database,
• starting and stopping default java HttpServer,
• starting and stopping Jetty server,
• running stub jndi,
• some support for dbUnit tests.

Another undocumented set of rules on github. I will not list them here, because their names are self-explanatory and they do not have specified license. Look at the rules directory to see their list.

Custom Rule

This chapter shows how to create new rules. They can be implemented from scratch by implementing the TestRule interface or by extending one of two convenience classes ExternalResource and Verifier available in jUnit.

We will create a new rule from scratch and then rewrite it using ExternalResource class.

New Rule

New rule ensures that all files created by tests are properly deleted after each test finishes its work. The tests themselves have only one responsibility: report all new files using the ensureRemoval(file) method exposed by the rule.

How to declare and use the DeleteFilesRule rule:

@Rule
public DeleteFilesRule toDelete = new DeleteFilesRule();

@Test
public void example() throws IOException {
  // output.css will be deleted whether the test passes, fails or throws an exception
  toDelete.ensureRemoval("output.css");
  // the compiler is configured to create output.css file
  compileFile("input.less");
  checkCorrectess("output.css");
}

From Scratch

Each rule, including class rules, must implement the @TestRule interface. The interface has exactly one method:

public interface TestRule {
  Statement apply(Statement base, Description description);
}

Our job is to take statement supplied in the base parameter and turn it into another statement. The statement represents a set of actions e.g., test, test case or test suite to be run. It might have already been modified by other declared rules and includes before and after test or class methods.

The second description parameter describes the input statement. It can tell test class name, test name, annotations placed on it, it knows whether we are dealing with test or test suite etc. We will not need it.

We need to create a new statement which will do three things:

• Empty the list of files to be deleted.
• Run underlying test, test case or test suite represented by the base parameter.
• Delete all files reported by tests inside previously run statement.

The statement is a class with one abstract method:

public abstract class Statement {
  public abstract void evaluate() throws Throwable;
}

Since underlying statement can throw an exception, the code to delete all files must run from finally block:

public class DeleteFilesRule implements TestRule  {
  
  public Statement apply(final Statement base, final Description description) {
    return new Statement() {
      
      @Override
      public void evaluate() throws Throwable {
        emptyFilesList(); // clean the list of files
        try {
          base.evaluate(); // run underlying statement
        } finally {
          removeAll(); // delete all new files
        }
      }
    };
  }
}

Both referenced methods emptyFilesList and removeAll are declared outside of new statement, directly inside the DeleteFilesRule class:

public class DeleteFilesRule implements TestRule  {

  private List<File> toDelete;
  
  private void emptyFilesList() {
    toDelete = new ArrayList<File>();
  }

  private void removeAll() {
    for (File file : toDelete) {
      if (file.exists())
        file.delete();
    }
  }

  /* ... the apply method ... */
}

The last thing we need is a public method able to add files to be deleted:

public void ensureRemoval(String... filenames) {
  for (String filename : filenames) {
    toDelete.add(new File(filename));
  }
}

Full Class

Click to expand

public class DeleteFilesRule implements TestRule  {

  private List<File> toDelete;
  
  public void ensureRemoval(String... filenames) {
    for (String filename : filenames) {
      toDelete.add(new File(filename));
    }
  }
  private void emptyFilesList() {
    toDelete = new ArrayList<File>();
  }

  private void removeAll() {
    for (File file : toDelete) {
      if (file.exists())
        file.delete();
    }
  }

  public Statement apply(final Statement base, final Description description) {
    return new Statement() {
      
      @Override
      public void evaluate() throws Throwable {
        emptyFilesList(); // clean the list of files
        try {
          base.evaluate(); // run underlying statement
        } finally {
          removeAll(); // delete all new files
        }
      }
    };
  }
}

Extending Build-in Classes

JUnit contains two convenience classes ExternalResource and Verifier meant to simplify the above process even more.

External Resource

The ExternalResource helps when you need to do some kind of preprocessing and postprocessing around the underlying test statement. If you need preprocessing, override the before method. If you need postprocessing, override the after method. The after is called from finally block, so it will be run no matter what.

Our DeleteFilesRule could be rewritten like this:

public class DeleteFilesRule2 extends ExternalResource  {
  
  /* ... list, ensureRemoval and removeAll methods ... */

  @Override
  protected void before() throws Throwable {
    toDelete = new ArrayList<File>();
  }

  @Override
  protected void after() {
    removeAll();
  }

}

Verifier

The Verifier has only one method verify to override. That method runs after the wrapped test finished its work and only if it did not thrown an exception. As the name suggests, the verifier is good if you want to run additional checks after the test.

More About jUnit

Previous post about jUnit 4 features:

• jUnit: Dynamic Tests Generation