Do you want to know how good your JUnit tests are? Jumble will tell you: from 0% (worthless) to 100% (angelic!). Jumble is a class level mutation testing tool that works in conjunction with JUnit. The purpose of mutation testing is to provide a measure of the effectiveness of test cases. A single mutation is performed on the code to be tested, the corresponding test cases are then executed. If the modified code fails the tests, then this increases confidence in the tests. Conversely, if the modified code passes the tests this indicates a testing deficiency.
ExampleHere is some example Jumble output for a Java class called "Foo", which has some JUnit tests in a class called "FooTest". Jumble starts by running the unit tests (in FooTest.class) on the unmodified Foo class to check that they all pass, and to measure the time taken by each test. Then it will mutate Foo in various ways and run the tests again to see if they detect the mutation. It continues this process until all mutations of Foo have been tried. The output might look like this:
Mutating Foo Tests: FooTest Mutation points = 12, unit test time limit 2.02s .. M FAIL: Foo:31: negated conditional M FAIL: Foo:33: negated conditional M FAIL: Foo:34: - -> + M FAIL: Foo:35: negated conditional ...... Score: 67%
This says that Jumble has tried 12 different mutants of Foo and the
unit tests (in FooTest) correctly detected the changed behaviour in
8/12 cases (indicated by a '.'), but failed to detect the change in
the other 4/12 cases. For example, when an
Overall, 67% of the mutations were detected by the unit tests, which means that they probably need to be improved.
Installation and Usage
Release 1.3.0 of Jumble is available as an Eclipse plugin.
Just add
For command line usage, just download Key to SymbolsThe web interface to Jumble [not released on Sourceforge] displays the following symbols as visual feedback of how thoroughly each class is tested.
Jumble HistoryJumble was developed in 2003-2006 by a commercial company in New Zealand, Reel Two (www.reeltwo.com), and is now available as open source under the GPL licence.JUnit [1] has become the de facto unit testing framework for the Java language. A class and its corresponding JUnit test is a sensible granularity at which to apply mutation testing. With Java it is feasible to perform mutations at either the source code or byte-code level. Jester [2] is a mutation tester operating at the source code level. While Jester proves useful, it is hampered by the costly cycle of modifying the source, compiling the source, and running the tests. Jumble is a new mutation tester operating directly on class files. It uses the byte-code engineering library (BCEL) [3] to directly modify class files thereby drastically cutting the time taken for each mutation test cycle. Jumble has been designed to operate in an industrial setting with large projects. Heuristics have been included to speed the checking of mutations, for example, noting which test fails for each mutation and running this first in subsequent mutation checks. Significant effort has been put into ensuring that it can test code which runs in environments such as the Apache webserver. This requires careful attention to class path handling and co-existence with foreign class-loaders. At ReelTwo, Jumble is used on a continuous basis within an agile programming environment with approximately 400,000 lines of Java code under source control. This checks out project code every fifteen minutes and runs an incremental set of unit tests and mutation tests for modified classes.
References[1] Erich Gamma and Kent Beck, "Test Infected: Programmers Love Writing Tests", The Java Report, 3, 7, 37-50, 1998.[2] Ivan Moore, "Jester: A JUnit Test Tester", XP2001, 2001. [3] Markus Dahm, "Byte Code Engineering with the BCEL API", Freie Universität Berlin, Institut für Informatik, B-17-98, 2001.
|