FAQ
What does a . output mean?
It means that the test cases detected the mutation. This is good.
What does a M output mean?
It means that the mutation (which follows the M) was not detected by the test cases. This generally indicates the tests are in some way deficient.
What does a T output mean?
The mutation caused the unit tests to take much longer than expected (possibly having put the code into an infinite loop). Jumble gives the tester the benefit of the doubt in such situations, and counts this mutation as having been detected.
Can Jumble ever incorrectly report an arithmetic problem?
Yes. Certain combinations of operations and constants give the same result for different examples. For example,
y = x * sign; y = x / sign;
give the same result for sign = 1 or sign = -1.
Can Jumble ever incorrectly report a conditional problem?
Yes. Certain pieces of code can be modified without invalidating the code. One example of this is where a condition is used to select different code for performance reasons rather than correctness. Suppose f1(x) and f2(x) compute the same function but exhibit different performance depending on x. Valid code might then be:
int foo(int x) {
return x < 10 ? f1(x) : f2(x);
}
mutating the condition x < 10 would not change the correctness of the code only the speed. Unless there is a very sensitive speed test this modification is likely to pass unit tests.
A similar situation arises in certain probabilistic algorithms. The following code executes the internal block half the time, but negating the condition does not change the semantics:
void foo() {
if (new Random().nextBoolean()) {
// do something ...
}
}
Note, however, the selection of a specific seed for the random number generator would render this code testable.
Why does Jumble only make one mutation at a time?
A single mutation is generally harder for a test class to detect than multiple simultaneous mutations. Also, if multiple mutations were made there is a slight chance they would cancel each other out.
Is any code exempt from mutation?
Jumble tries not to make mutations within assertions. Specific methods can also be excluded from testing, by default main and integrity are excluded.
Can I Jumble an interface?
Not really, since an interface contains no code to mutate. Jumble will report a score of 100% for interfaces.
What's the situation with abstract classes?
For the class AbstractFoo, Jumble will look for a test named DummyFooTest. This is done because there could be an actual abstract test class named AbstractFooTest.
The recommended Jumble conventions are to put some unit tests for the AbstractFoo methods into the abstract AbstractFooTest class, then inherit that class into the unit tests for each subclass of AbstractFoo (say FooA and FooB). This means that JUnit will test the AbstractFoo methods for each subclass of AbstractFoo. However, the methods in AbstractFoo will not be mutated when Jumble analyses the subclasses of AbstractFoo, because it is a design principle of Jumble to analyse each class just once. So we create a DummyFooTest class that inherits AbstractFooTest, provides a "dummy" implementation of any missing methods, and concentrates on testing the methods of AbstractFoo.
Of course, you can override these default Jumble naming conventions by explicitly specifying your test classes on the command line, like this:
java -jar jumble.jar AbstractFoo FooTest1 FooTest2
I cannot test something Jumble complains about?
Just because something is difficult to test does not mean it is impossible to test. Testing is generally a finite resource so it is appropriate to concentrate testing where it is most important. At best Jumble is just another indicator of where potential weaknesses in the testing regime exist. Difficulty in writing tests can be genuine, or due to laziness (e.g. not wanting to parse standard output), or symptomatic of a poor design.
There are a few situations where the limitations of Java and/or JUnit make testing hard. These include testing GUI code, audio code (indeed any code where the JVM leaves uninterruptible threads hanging around), code which is conditional on operating system and/or native code, and out of memory conditions. Testing algorithmic complexity can also be difficult in a reasonable time; for example, making sure your algorithm really is quadratic rather than cubic.
Remember the purpose of Jumble is to identify potential problems, if after examining a failure you convince yourself there is never going to be a problem (e.g. because you have a formal proof) then the failure could be ignored.