Home

Downloads

Development

Mutations

FAQ

Do you want to know how good your JUnit tests are?

Jumble will tell you: from 0% (worthless) to 100% (angelic!).

Jumble is a class level mutation testing tool that works in conjunction with JUnit. The purpose of mutation testing is to provide a measure of the effectiveness of test cases. A single mutation is performed on the code to be tested, the corresponding test cases are then executed. If the modified code fails the tests, then this increases confidence in the tests. Conversely, if the modified code passes the tests this indicates a testing deficiency.

Example

Here is some example Jumble output for a Java class called "Foo", which has some JUnit tests in a class called "FooTest".

Jumble starts by running the unit tests (in FooTest.class) on the unmodified Foo class to check that they all pass, and to measure the time taken by each test. Then it will mutate Foo in various ways and run the tests again to see if they detect the mutation. It continues this process until all mutations of Foo have been tried. The output might look like this:

  Mutating Foo
  Tests: FooTest
  Mutation points = 12, unit test time limit 2.02s
  ..
  M FAIL: Foo:31: negated conditional
  M FAIL: Foo:33: negated conditional
  M FAIL: Foo:34: - -> +
  M FAIL: Foo:35: negated conditional
  ......
  Score: 67%

This says that Jumble has tried 12 different mutants of Foo and the unit tests (in FooTest) correctly detected the changed behaviour in 8/12 cases (indicated by a '.'), but failed to detect the change in the other 4/12 cases. For example, when an if (C) decision on line 31 was mutated to if (!C), the unit tests still passed, so they are probably not testing that if statement thoroughly enough.

Overall, 67% of the mutations were detected by the unit tests, which means that they probably need to be improved.

Installation and Usage

Release 1.3.0 of Jumble is available as an Eclipse plugin. Just add http://jumble.sourceforge.net/update/ to Eclipse as an update site, and you will be able to install Jumble. Then you should be able to right-click on any class and do Jumble Mutation Tester / Analyse tests of this class to mutate the class and analyse its tests. Note that you click on your source code class, NOT on the JUnit tests.

For command line usage, just download jumble_1.x.y.jar via the Sourceforge Files release system, and run it with java -jar jumble_1.x.y.jar --help to see the help message.

Key to Symbols

The web interface to Jumble [not released on Sourceforge] displays the following symbols as visual feedback of how thoroughly each class is tested.

	X, no corresponding test class was found.
	No corresponding test class was found, but class possibly contains SWING/AWT code which is hard to JUnit test.
	F, very lazy programmer, JUnit code is worthless.
	E
	D-
	D
	C-
	C
	B
	A-
	A, aim for this as a minimum in production code.
	A+
	A++, all mutations were detected.
	Class was abstract or class did not contain any possible mutation points.

Jumble History

JUnit [1] has become the de facto unit testing framework for the Java language. A class and its corresponding JUnit test is a sensible granularity at which to apply mutation testing. With Java it is feasible to perform mutations at either the source code or byte-code level. Jester [2] is a mutation tester operating at the source code level. While Jester proves useful, it is hampered by the costly cycle of modifying the source, compiling the source, and running the tests.

Jumble is a new mutation tester operating directly on class files. It uses the byte-code engineering library (BCEL) [3] to directly modify class files thereby drastically cutting the time taken for each mutation test cycle.

Jumble has been designed to operate in an industrial setting with large projects. Heuristics have been included to speed the checking of mutations, for example, noting which test fails for each mutation and running this first in subsequent mutation checks. Significant effort has been put into ensuring that it can test code which runs in environments such as the Apache webserver. This requires careful attention to class path handling and co-existence with foreign class-loaders.

At ReelTwo, Jumble is used on a continuous basis within an agile programming environment with approximately 400,000 lines of Java code under source control. This checks out project code every fifteen minutes and runs an incremental set of unit tests and mutation tests for modified classes.

References

[2] Ivan Moore, "Jester: A JUnit Test Tester", XP2001, 2001.

[3] Markus Dahm, "Byte Code Engineering with the BCEL API", Freie Universität Berlin, Institut für Informatik, B-17-98, 2001.