Archive for April, 2009

Advanced parallel testing with TestNG and data providers

TestNG allows you to run your test methods in separate threads. You can configure the size of the thread pool and the time-out and TestNG takes care of the rest. For example, consider the following test class invoked with a thread pool size of 2:

public class A {
public void g1() { log("g1"); }
public void g2() { log("g2"); }
public void g3() { log("g3"); }
public void g4() { log("g4"); }

The output:

Thread:9 g4()
Thread:8 g2()
Thread:8 g3()
Thread:9 g1()

As you can see, TestNG created a pool of two threads and it is dispatching all the test methods on each of these threads as they become available. You can also configure the threading strategy (“each test method in its own thread”, “each class in its own thread”, etc…) and the time out for each of these thread pools.

Another popular feature of TestNG is data providers. Let’s add two methods and two data providers to the test class above:

public Object[][] dp1() {
return new Object[][] {
new Object[] { 1 },
new Object[] { 2 },
new Object[] { 3 },
new Object[] { 4 },
@Test(dataProvider = "dp1")
public void f1(Integer n) {
log("f1", n);
public Object[][] dp2() {
return new Object[][] {
new Object[] { 11 },
new Object[] { 12 },
new Object[] { 13 },
new Object[] { 14 },
@Test(dataProvider = "dp2")
public void f2(Integer n) {
log("f2", n);

f1() will be invoked with 1, 2, 3 and 4 while t2() will receive 11, 12, 13 and 14.

Here is the output (each color represents a different kind of test method: one for the four methods that don’t use any data provider, one for f1() and one for f2():

Thread:9 g4()
Thread:8 g3()
Thread:9 g2()
Thread:8 f1(1)
Thread:9 f2(11)
Thread:9 f2(12)
Thread:9 f2(13)
Thread:8 f1(2)
Thread:9 f2(14)
Thread:8 f1(3)
Thread:9 g1()
Thread:8 f1(4)

Everything is still running on a thread pool of size 2, but you will also notice that the two methods using data providers (f1() and f2()) are invoked in sequence on the same thread. In other words, f1() is invoked on one thread an then it remains on that same thread until it has received all the values from its data provider (1, 2, 3 and then 4). Same thing for f2() and the values 11, 12, 13 and 14.

Extending multithreading to data providers has been one of the most requested features for TestNG, and I’m happy to announce that it’s now implemented and it will be part of the next release of TestNG.

In order to make a data provider run in a pool of threads, you use the new annotation parallel:

@DataProvider(parallel = true)
public Object[][] dp2() {

Data Providers are run in their own thread pool, which is different from the thread pool used for test methods. Let’s run the example above again with a test thread pool size of 2 and a data provider thread pool of 3:

Thread:9 g4()
Thread:8 g3()
Thread:8 g2()
Thread:9 f1(1)
Thread:10 f2(11)
Thread:11 f2(12)
Thread:12 f2(13)
Thread:9 f1(2)
Thread:12 f2(14)
Thread:9 f1(3)
Thread:9 f1(4)
Thread:8 g1()

In this run, both the g methods and f1() are running on the test thread pool (remember that even though f1() is using a data provider, it’s not using parallel=true, so it’s using the test thread pool). The novelty here is that the four invocations of f2() are now happening on three different threads (10, 11 and 12). These three threads are part of the data provider thread pool, which was configured with a size of three.

Let’s now make the other data provider parallel as well:

@DataProvider(parallel = true)
public Object[][] dp1() {

The output:

Thread:9 g4()
Thread:8 g3()
Thread:8 g2()
Thread:10 f2(11)
Thread:11 f1(1)
Thread:12 f1(2)
Thread:10 f1(3)
Thread:11 f1(4)
Thread:12 f2(12)
Thread:11 f2(13)
Thread:10 f2(14)
Thread:9 g1()

This time, only the g() methods are using the test thread pool (threads 8 and 9) while the two methods using a data provider (f1() and f2()) are sharing the data provider thread pool (threads 10, 11 and 12).

With this new feature, TestNG makes it even easier to run your tests in parallel, and tests that are using data providers returning large sets of values are likely to see a significant decrease in running time.

Parallel data providers will be part of TestNG 5.10 but you can already download the beta and try it for yourself.

Why Java doesn’t need operator overloading (and very few languages do, really)

Operator overloading is a topic that never fails to generate very passionate responses, and this monster thread on Artima is no exception.

First of all, I’d like to address a common complaint heard from people who dislike operator overloading, and which usually goes along the lines of:

In Java, when I see a + b, I know exactly what is going on. If Java supported operator overloading, I would have no idea what + means.

I have very little sympathy for this argument because this example is hardly different from a method call such as o.init(). When you read such code in Java, you don’t necessarily assume that it’s the method init on o‘s class that is being invoked. You know that you will have to look at how this object is created in order to determine its effective type before you can find out which method init is being invoked.

Operator overloading is no different, and if you knew that the language that you are reading supports it, you are just going to extend this mental path to operations that involve overridable operators.

Very often, I find that operator overloading is being demonized because its uses in other languages has led to excesses. I have found that many people who dislike operator overloading can trace it back to some personal experiences where they once had to deal with code that was clearly abusing this feature. This is unfortunate, but I still think that Java is doing fine without it and that overall, it only leads to clearer code in very few and uncommon cases.

Here are a couple of objections I have with operator overloading:

  • The number of operators that you can overload is very small and each of them is attached to very specific semantics that makes little sense outside the realm of scalars and of a few other specialized mathematical concepts (e.g. matrices).

  • In exchange for infix notation, operator overloading severely restricts my freedom to name my methods in a meaningful way.

The most common operators that people want to overload are + and -, and they are also the operators that probably make the most sense outside simple arithmetics. I can see these two operators make sense for types that represent a collection of elements (java.util.Collection obviously, but also strings or streams). However, even in this ideal case in favor of operator overloading, the metaphor quickly breaks down when you realize that you need to supplement these two methods with ones that have slightly different meanings, such as addAll(). Would you rather have an interface that contains add() and addAll() or +() and addAll()?

Of course, there are more operators that can be overloaded, such as * or /, but I find that even strong advocates of operator overloading quickly run short of arguments when trying to justify why one would even want to overload them outside of specific algebras.

I am a big fan of clear names, and I dislike the name + for the simple reason that it doesn’t carry enough meaning to explain to me what is going on. This operator is very clearly defined when it is used in a mathmatical sense but it becomes very vague when you start to extend it to concepts that supports the idea of addition in a more vague sense. For example, adding to a List is very different from adding to a Map since the latter can produce an object that’s equal to the one you started with, even though the object you added is not zero. And what is zero in the sense of a collection, anyway? Doesn’t it make much more sense to use add() for a List and put() for a Map?

Overall, I find that code that makes heavy use of operator overloading is harder to read and harder to maintain because it is severely restricted syntactically (forced to use specific names taken from a very small pool) and semantically (since you are using symbols that have very precise mathematical definitions, it is very unlikely that your usage will match the mental model that people who read your code have of their meaning).

I would love to see some good examples of operator overloading, if you have any to share…

Announcing TestNG 5.9

I’m happy to announce the release of TestNG 5.9. A lot of bug fixes went into this release, and a few noteworthy new features:

  • Parallelism can now be specified on classes. In the previous versions, you could specify either a setting of "methods" (every test method will be run in its own thread) and "tests" (all the methods in a given <test> tag will be run in the same thread but each <test> tag will run in its own thread). The new "classes" setting extends this functionality to test classes: all the test methods in the same class will run in the same thread but each class will have its own thread. Of course, the thread pool size and time out for these three settings are configurable.
  • You can now specify a time-out on @Test methods that use the invocationCount attribute. This way, you can specify that a test method should be invoked fifty times but that the maximal time allowed for these fifty invocations is ten seconds. If the invocations don’t complete in time, the test will be marked as failed.

  • A few new interfaces to assist developers who use the TestNG API: IAnnotationTransformer2 extends the annotation modification capabilities to more than just @Test and IInvokedMethodListener, which gives you additional monitoring on what test method is being invoked and when.

A lot of people have contributed patches, fixed and improvements for this release, and their names can be found in the full release notes below. Thank you all!

Release notes for TestNG 5.9

  • Added: New ant task boolean flag: delegateCommandSystemProperties (Justin)
  • Added: skipfailedinvocations under <suite> in testng-1.0.dtd (Gael Marziou / Stevo Slavic)
  • Added: -testrunfactory on the command line and in the ant task (Vitalyi Pamajonkov)
  • Added: TESTNG-298: parallel="classes", which allows entire classes to be run in the same thread
  • Added: @BeforeMethod can now declare Object[] as a parameter, which will be filled by the parameters of the test method
  • Added: IAnnotationTransformer2
  • Added: @Test(invocationTimeOut), which lets you set a time out for the total time taken by invocationCount
  • Added: IInvokedMethodListener
  • Added: -testjar supports jar file with no testng.xml file
  • Fixed: IInvokedMethodListener wasn’t properly recognized from the command line (Leonardo Rafaeli)
  • Fixed: TESTNG-309 Illegal default value for attribute in DTD file
  • Fixed: TESTNG-192: JUnit XML output includes wrong tests (Aleksandar Borojevic)
  • Fixed: Set a generated suite to default to non-parallel (Mark Derricutt)
  • Fixed: -testJar command line parsing bug
  • Fixed: testng-failed.xml didn’t include the listeners
  • Fixed: annotation transformers were not run when specified in testng.xml
  • Fixed: TESTNG-192: JUnit XML output includes wrong tests (Borojevic)
  • Fixed: @Parameters was not working correctly on @BeforeMethods with @DataProvider used on @Test methods
  • Fixed: testng-failed.xml was sometimes incorrectly generated (Borojevic)
  • Fixed: TESTNG-228: Assert.assertEqualsNoOrder
  • Fixed: TESTNG-229: Assert.assertEquals does not behave properly when arguments are sets
  • Fixed: TESTNG-36: assertEquals(Collection actual, Collection expected, String message) may have bug
  • Fixed: TESTNG-296: Malformed jar URLs breaking -testJar
  • Fixed: TESTNG-297: TestNG seemingly never stops running while building failed test suite (Gregg Yost)
  • Fixed: TESTNG-285: @Test(sequential=true) works incorrectly for classes with inheritance
  • Fixed: TESTNG-254: XmlSuite toXML() ignores listeners
  • Fixed: TESTNG-276: Thread safety problem in Reporter class
  • Fixed: TESTNG-277: Make Reporter.getCurrentTestResult() public
  • Fixed: Potential NPE in XmlTest#getVerbose (Ryan Morgan)
  • Fixed: EmailableReporter only displayed the first group for each test method
  • Fixed: time-outs were not working in <test> and <suite>
  • Fixed: @BeforeTest failing in a base class would not cause subsequent test methods to be skipped
  • Fixed: TESTNG-195: @AfterMethod has no way of knowing if the current test failed
  • Fixed: TESTNG-249: Overridden test methods were shadowing each other if specified with <include>
  • Fixed: DataProviders from @Factory-created tests were all invoked from the same instance
  • Fixed: enabled was not working on configuration methods
  • Fixed: IIinvokedMethodListener was not correctly added in TestNG
  • Fixed: NPE in XmlSuite#toXml
  • Fixed: TESTNG-231: NullPointerException thrown converting a suite to XML (Mark)

You can’t ignore the types

The thread called “Getting Dynamic Productivity in a Static Language” on Artima has generated a lot of very interesting comments.

In particular, the following statement made me react:

So that was the history, and so I was asking him what it was that made him feel so productive in Smalltalk, and one of the things he said is he didn’t have to waste time thinking about types.

This is a fundamental mistake that a lot of dynamic enthusiasts (short for “dynamically typed language enthusiasts”) keep making over and over.

The fact is: you always have to think about types when you write code, period.

Dynamic enthusiasts are convinced that they can ignore this aspect of development altogether, but it always comes back to bite you, either when

  • you write tests for your code
  • you try to refactor it
  • somebody else needs to modify it
  • or simply when someone needs to use your code

Eventually, you look at an object and you have to figure out what methods or messages it responds to, and I find that this problem is much easier to solve when the answer is in the source code in a form that can be enforced by the compiler.

If you are interested in this topic, read the entire thread, it’s worth it.