In a recent entry, Martin Fowler explains the reason behind JUnit reinstantiating your JUnit class before invoking each test method. He concludes by showing the following example:

[TestFixture]
public class ServerTester
{
private IList list = new ArrayList();
[Test]
public void first() {
list.Add(1);
Assert.AreEqual(1, list.Count);
}
[Test]
public void second() {
Assert.AreEqual(0, list.Count);
}
}
and pointing out that this example passes with JUnit but fails with NUnit and TestNG.
 Here is why.
 Remove mentally the annotations from the code above ([TestFixture], [Test]) and read the code just like if it were standard Java/C# code.  And now, predict its output, knowing that first() is invoked first and then second().
 It seems pretty obvious this code will fail in Java and in C#.
 So why should your test framework behave differently?
 By the way, here is the correct way to write this test with TestNG:
@Test
public class ServerTester
{
private IList list = null;
@Configuration(beforeTestMethod = true)
public void init() {
list = new ArrayList();
}
public void first() {
list.Add(1);
Assert.AreEqual(1, list.Count);
}
public void second() {
Assert.AreEqual(0, list.Count);
}
}
Quick explanation:
The @Test annotation at the class level indicates that all the public methods in this class are test methods, so there is no need to add individual annotations @Test on each method.
The @Configuration annotation indicates that the method init() should be invoked every time before a test method is invoked.
How would you feel if Java invoked your constructor every time before invoking your methods?

A comment in my previous entry asks why statics can cause problems. Here is a quick example of a program that wants to handle a -verbose option:

public class Foo {
private static boolean m_verbose = false;
public static void main(String[] argv) {
if (/* argv contains -verbose */) {
m_verbose = true;
}
runProgram();
}

Now imagine you invoke your program twice in the same build file as follows:

<java classname="Foo" line="-verbose"/>
... and somewhere else
<java classname="Foo" />

You will probably wonder why the second invocation keeps being invoked in verbose mode…

The solution is of course:

<java fork="yes" classname="Foo" line="-verbose"/>
... and somewhere else
<java fork="yes" classname="Foo" />

If you forget to fork a new JVM, the static variables set in the previous invocations will not be reinitialized. The problem with the fork flag is that it creates a new JVM each time, which dramatically increases the time of your build.

Therefore, it is much cleaner to avoid static variables as a rule of thumb, for example by storing all your command line variables in a singleton class and make sure this singleton gets reinitialized each time your program is invoked.

Now, does this mean you should never use static methods/fields? Of course not. Actually, I like to use static methods as much as I can as long as they don’t modify static state, because const static methods decrease the coupling of your classes.

But this will be the subject of a future entry.

TestSetup is a JUnit class you can use if you want to introduce initialization in your JUnit tests that only get invoked once, and not once before test method, which is what JUnit does (making it difficult to make sure that an initialization method is only invoked once during the lifetime of the class). I wrote more about this oddity not long ago.

A recent weblog entry about TestSetup shows how to use it, which is pretty straightforward. I still have a big problem with this solution, and what JUnit forces you to do to work around this “design choice”: it forces you to introduce static methods in your code.

I don’t have anything against static methods in general, I tend to use them as much as possible whenever they don’t need access to any non-static fields since it decreases the coupling between my classes. However, static methods are particularly evil when they modify state, and by extension, force you to declare static fields.

Which is exactly the only choice that JUnit leaves you.

The problem with non-const static methods is that they break badly in several cases:

• Multi-thread access.
• Different objects thinking they are accessing private content, while it is actually shared.
• Reuse of the same JVM.

This last point is particularly important. By forcing you to introduce static methods in your code, JUnit guarantees that if somebody ever invokes your class several times from an ant task while not providing the flag fork=”yes”, your code will break in mysterious ways.

Classes in object-oriented programming obey very simple rules:

• If you invoke your initialization from the constructor, it will only be invoked once.
• If your content is not static, it is guaranteed to be private to your instance.

JUnit violates these two principles and I just can’t get used to it. Instead of righting a wrong with another wrong, why not return to the most intuitive way of doing this and letting test writers implement methods that are guaranteed to be invoked once?

Of course, these is also an alternative. 🙂

Imagine I am trying to test a server.  In order to do this, my test class will contain the following test methods:

• Check that we are running the correct JVM.
• Check that the server started correctly.
• About twenty methods making various calls to the server.

Obviously, the first two test methods listed above should be run before everything else.  The way you would do this with JUnit is to put the first two methods in the initialization code, which will probably have to be static since JUnit instantiates a new object before each test method invocation, and we can’t use setUp() either since it’s invoked before each test method.  This initialization code will set two booleans and the twenty test methods will have to test for these two booleans before proceeding, and if one of them is false, it will fail.

Now let’s assume that on a test run, I used the wrong JVM.  In this case, JUnit will probably report this in the initialization code and it will then report twenty more failures for each test method.

This is very bad for a variety of reasons:

• The initialization code is not in a test method per itself.  This is bad because report of its success/failure will be done in a separate track.
• When QA reads the result of this test run, they will see "1 SUCCESS, 21 FAILS" and they will, rightfully, get really scared.
• Upon reading of this result, QA will have to provision for fixing 20 tests, while the reality is that only one test failed.

This is why I believe that a test framework needs to provide support for "dependent test methods", where you can mark test method b() as depending on the successful run of test method a().  If a() failed, then b() will be marked as a SKIP, and not a FAIL.

With such a feature, the test run will be marked "1 SUCCESS, 1 FAIL, 20
SKIPS", which is much more accurate.

Here is how I would write this test using TestNG:

@Test
public correctVM() {}
@Test
public serverStartedOk() {}
@Test(dependsOnMethods = { "correctVM", "serverStartedOk" })
public method1() {}
...

This is okay but having to list all the methods that we depend for each new test method on is error-prone, so instead, let’s use TestNG’s groups:

@Test(groups = { "init" })
public correctVM() {}
@Test(groups = { "init" })
public serverStartedOk() {}
@Test(dependsOnGroups = { "init.* })
public method1() {}
...

We have gained a lot of flexibility with groups.  For example, imagine that I want to add another init test method, such as "firewall is on".  All I need to do is add this test method and declare that it is part of the group "init".

Also, note that I used a regular expression in the "dependsOnGroups" declaration, as a reminder that you can actually define several init groups (such as "initOS", "initJVM", etc…) and they will automatically be run before any test method is invoked.

But we can do even better.

In the above example, I don’t like the fact that whenever I add a new "real" test method, I need to remember to specify that it depends on the group "init.*".  In TestNG, the traditional way to indicate that an annotation should apply to all test methods is to move this annotation at the class level.

Also, I don’t like the fact that "init" methods and "real" test methods are in the same class, so I’d like to use inheritance to provide a cleaner separation of roles.

Therefore, I will restructure my tests like
this:

@Test(groups = "init")
public class BaseTest {
  public correctVM() {}
  public serverStartedOk() {}
}

This is now the base class for my tests.  The @Test annotation is now on the class, which means it applies to all the public methods inside that class (so there is no need to repeat it on each individual method).  Therefore, each public method automatically becomes part of the group "init".

Next, I write my test class as a subclass of BaseTest:

@Test(dependsOnGroups = { "init.*" })
public class TestServer extends BaseTest {
  public method1() { ... }
  public method2() { ... }
  ...
}

Here again, the @Test annotation is now on the class, which means that it applies to all the public test methods, making it easier to add new testing methods.  Also, since this class extends BaseTest, it "sees" not only the methods that are being inherited, but their annotations as well, so TestNG will include all the methods from the base class that belong to the "init.*" group to determine which methods need to be run first.

What’s the overall result?

• Two classes that have responsibilities that are very clearly delineated.
• Very easy maintenance and evolution, since adding test methods (whether they are init or "real" test methods) boils down to just adding these methods to the right class (no need to annotate them).  A newcomer doesn’t even need to know about that annotations that are needed to get all this to work.
• A report that will accurately reflect the result of the test runs and will correctly identify the real failures from those that are caused by a cascade effect, and whose resolution should therefore be postponed until all the FAILs have been resolved.

I have moved the following projects to java.net:

• Doclipse
• TestNG
• J15

It will make it easier for users to download the latest versions from CVS or use the mailing-lists to ask questions.

Update: if you are looking for a Java project, read this.

Can you spot why the following program:

public class Split {
public static void main(String[] argv) {
String STRING = "foo  bar";
String[] s = STRING.split(" ");
for (int i = 0; i < s.length; i++) {
System.out.println(i + " '" + s[i] + "'");
}
}
}

displays:

0 'foo'
1 ''
2 'bar'

The reason is that split() works a little differently from StringTokenizer: 
it accepts a regular expression as a separator.  In the code above, I
define this regular expression as " " (one space character) but the input string
contains two of them.  Therefore, we can solve this problem by using the "
+" as a regular expression ("at least one space character").

Still, the fact that split() can return empty strings is deceiving,
especially if you are converting your code from StringTokenizer.

There
are a couple of good things about this behavior, though:

• You can reconstitute
  the original string if you need to.
• It makes it easier to parse strings with records that can
  be empty, such as lines from a log file.

Can you think of any other use?

I have just added support for concrete collections to
J15:  use Ctrl-1 on the creation of a concrete collection and J15 will add the
correct Generic parameters.

Before:

List l2 = new ArrayList();
Float n2 = new Float(42);
l2.add(n2);

After:

ArrayList<Float> l2 = new ArrayList<Float>();
Float n2 = new Float(42);
l2.add(n2);

Before:

Map m = new HashMap();
m.put(new Integer(42), "Cedric");

After:

HashMap<Integer, String> m = new HashMap<Integer, String>();
m.put(new Integer(42), "Cedric");

Let me know if you can think of other conversions that this plug-in could make
for you.