I took advantage of the holiday break to completely revamp TestNG’s HTML reports, something I’ve been meaning to do for a very long time but never found the time to put at the top of my TODO list. I wrote the original reports pretty much with the very first version of TestNG, around 2004, and I hardly touched them since then. My intention with this rewrite was not just to revamp them visually but also technically, so that I could give myself as much freedom as possible to improve them from this point on.

Here are some notes I took during this process.

Content and appearance

First of all, here are the new reports. They show all the suites on the same page with a banner at the top, a navigator pane on the left hand side (which always stays visible) and a main panel on the right hand side, which shows all the information you requested depending on which item you clicked in the navigator. Pretty straightforward.

If you take a look at the source, you will see that it hardly contains any HTML: it’s mostly made of <div> and <span>. I didn’t really set out to do this initially, it’s just that whenever I used HTML elements, I inherited undesirable CSS attributes (margins, paddings, etc…) which I ended up resetting manually, so after a while, it just seemed much easier to use divs and spans knowing that they start with a clean CSS slate.

Since there is so much generated text, I pondered using a templating library to make my life easier. The first question to settle was: client side or server side templates? In this case, there is not really a “server” side, so the question is more “Java” or “Javascript” templates. I quickly rejected the client side approach which, while well adapted to serve pages over a network, doesn’t seem to have much benefit in this particular case since the page is generated locally. As for the Java side, the two standard options for templates (Freemarker and Velocity) seemed quite overkill for what I was trying to do, so I ended up writing my own tiny implementation of mustache.js (which I’ve used in the past and liked quite a bit)..

Once I had the templating code working, I started converting the Java code over to it but… something felt wrong. I quickly gained the impression that this was actually a step backward compared to doing the generation 100% from Java, and the reason quickly became apparent to me: the Java compiler.

The problem with using templates is that you get very little help from the tools. Some template libraries allow you to include other templates, which can reduce the repetition, but it’s still very easy to make simple typos or being forced to copy/paste a lot. In contrast, my Java hierarchies makes it quite trivial to, say, add a new panel on the right hand side with a link to the left. Implement the right class, declare it and it will automatically work, with the right defaults (CSS selectors) and a lot of the logic validated by the compiler.

This comfort was too good to pass up so I scrapped my template idea (my mustache.js implementation is still around, though, and I might use it some time in the future) and I stuck to 100% Java generation.

Javascript and development tools

Picking jQuery was a no-brainer, it’s hard for me to imagine doing anything in Javascript that manipulates the DOM without using JQuery. The logic in the new reports is fairly simple and is just above one hundred lines of Javascript.

Chrome’s development environment is also quite superb (I’m sure other browsers’ is just as good). In a nutshell, you have access to pretty much the same range of support that Eclipse or IDEA provides in Java: breakpoints, inspection and modification of variables, and even CSS debugging, which I’ve found invaluable to track down unexpected CSS behaviors (I came across quite a few).

If you haven’t kept up with what you can do with Chrome/Javascript/JQuery these days, try the following:

• Open http://testng.org/new in Chrome.
• Right click anywhere on the page and select "Inspect element". This will open the document explorer at the bottom.
• With the focus in the document explorer, type ESC, which will open the REPL.
• Type $('.navigator-suite-content').hide().
• This will collapse the content under both suites.

You can see how easy it is to debug this kind of code.

Other libraries

If you select the "Times" link of the first test suite in the Navigator pane, you will see the following table:

Click to enlarge

I am using Google’s Chart Tools to display this table and it was quite straightforward to add. The only tricky part was how to generate the data in Javascript form and make sure that the table gets drawn only after this data has been initialized, which required using a few Javascript tricks to make sure the initialization order is correct. The Google Chart Tools are very powerful and I will probably use more of their API to display additional graphs in the TestNG reports (pie charts, etc…).

Conclusions

Here are my main take away points:

• CSS still feels like black magic to me. The theory is trivial on paper but in reality, I often come across results that just don’t make sense. With the help of modern CSS debuggers, it’s a little bit easier to find out what is happening and why (I especially like Chrome’s “Computed style” which gives you a list of all the attributes that were derived from .css files), but there are still times where I feel completely helpless trying to find out why something is not how it should be.
• I’m a bit concerned with the size of the reports: I was quite surprised to find out that the reports you’re looking at is one meg in size. Admittedly, it shows over five hundred test methods, but the fact that it contains all the panels, even those the user might not be interested in, is a potential place for improvement. I might decide to put each individual pane in its own file.
• Javascript’s rubbery type system is handy for this kind of task, but I’m still a bit unsure how well it scales to hundreds of thousands of lines of code. For example, I really enjoy being able to say if (v) regardless of what type v is and have Javascript’s truthiness values do the right thing. Similarly, it’s nice to be able to add a parameter to a function and not having to update one single call site (if you call a Javascript method with less parameters than it expects, the extra parameters simply receive the Undefined value. Obviously you need to test for this in said function).
• JQuery is great and it’s probably making Javascript even more popular than it already is. I’m hoping the Dart team is taking notes and making sure that a similarly powerful and elegant framework will be available in Dart.

Finally, a call for help: I welcome any feedback on how to improve the CSS of these new reports, so if you are so inclined, feel free to improve the look of these new reports and share your improvements with me. I will be very grateful.

Happy new year!

I recently received an alarming bug report about the TestNG Eclipse plug-in:

I’m talking about TestNG plugin for eclipse and its performance. Since some time this plugin contains new feature – searching already finished tests. (just “Search: ” with a field).
Performing this search makes that my eclipse does not response for minutes. After all I can see as a result – all historical run tests (according to my search request).

A bug that would cause Eclipse to be unresponsive for minutes is certainly unacceptable, so I tried to dig into this problem quickly. However, I was still not quite sure what this user was doing, and since I use and I work on the plug-in on a regular basis, the problem have been happening under some very specific and unusual circumstances. A few emails later, I received the following clarification:

Yes, I’m talking about the Search box and filtering.
‘historical data” means all collected test results. In case I don’t restart eclipse for weeks and start million tests (via providers) performance plays here important role.
In fact I don’t need the search box feature. Simply today I filled it up by mistake and lost half an hour… First to find, second to remove from the box…

I have to say I laughed when I read “I filled it by mistake and lost a half hour”. Okay, maybe not very funny from this user’s stand point, but funny because I can see how having a million test instances could cause the plug-in to become unresponsive.

The problem is caused by the “Search filter”, a feature that appeared in the plug-in a few months ago:

When you type characters in this text box, the results displayed in the tree get filtered and only the nodes that match the text are shown, a functionality that is very convenient when you want to inspect the results of a specific test.

This poor user created a test run with one million test instances, accidentally typed a letter in the search box and then the plug-in diligently went through the million nodes, retaining only those that contain this letter. Obviously, deleting this character will cause the exact same thing to happen, except that the entirety of the test suite will be restored in the tree view.

This code is fairly naive and not really optimized to account for the creation of a million TreeItems, so I wasn’t really surprised to hear that doing so would cause Eclipse to become unresponsive for a while. After all, the addition of these SWT objects has to be made on the Event Dispatch Thread at some point, and whether you add them one by one or in bulk (which is what the plug-in is doing), it’s pretty much guaranteed that the dispatch thread will get severely hogged.

It’s a pretty simple problem with a few obvious solutions, the hard part is finding out the best course of action.

My first thought at trimming down the solution space was to decide that a test result featuring millions of objects was an unusual case and that optimizing this part of the code should therefore not be my priority. But just for the sake of argument, I explored several ways of doing so and I came up with various ideas, among which doing some precaching of subwords (for example, so I can match three letter words to nodes very quickly) or by virtualizing the tree. I’m sure there are plenty of other techniques available and I’m definitely interested in hearing about them.

But for now, I decided to take a lighter approach, so I made two changes:

1) When I detect that the tree contains more than a certain number of nodes, I configure the text box not to start filtering until it has at least three characters. This addresses the problem of accidentally typing a letter in the box, and it also guarantees that when the filtering is triggered, it will match fewer nodes (since these will have to match three letters instead of just one). Obviously, the code sill has to go through the million nodes.

This looks simple enough but I couldn’t help trying to devise clever ways of actually calibrating these numbers: when should this behavior be triggered? 1000 nodes? 10,000 nodes? Also, should the minimum number of characters be a function of that number? For example, require two characters for 1000 nodes but three for 10,000 nodes? How about using a log in base 1000 to create these pairs of values? Or should the base of the log be 50?

2) I added a new “Clear results” icon to the toolbar:

Clicking it wipes the results displayed in all the panes, which guarantees that typing text in the search filter will do nothing (the search text box actually gets disabled).

The only concern I have with this approach is that users might get confused to see that sometimes, the search filter will activate with one character and other times, it requires three. They might even think that the search filter is broken if nothing happens after typing two characters.

You can work around this problem by providing a tool tip to the text box or, better, display a helpful text tip in a greyed out font inside the text box itself, saying something like “[Type at least three characters to search]”.

I found it interesting how such a simple problem can offer so many different avenues to solve it and how each one comes with benefits and costs that need to be carefully weighed. I’m hoping to have struck a correct balance with the current approach, one which solves the problem at hand without impacting most of the regular users too adversely.

I’m happy to announce the release of TestNG 6.0. A lot of changes have gone into this release, which have slowly accumulated in the 5.14 line over the past few months. I’ll go over the most important features and bug fixes in this entry and I’m including the full change log at the bottom of this post.

If I had to pick the most prominent features for TestNG 6.0, they would be:

• YAML.
• Guice.
• Improvements to the Eclipse plug-in.

YAML

As most of you know, TestNG uses an XML file to capture the entire description of your test suite, and while I think the schema for this file has managed to remain fairly simple over the years, the XML aspect of it can sometimes get in the way of making quick changes. I considered supporting an additional format for a while and I ended up narrowing my choices down to JSON and YAML. In the end, YAML won on tiny details such as not requiring the constant use of double quotes and the fact that JSON seems to be more optimized for computer than human consumption. Here is a short TestNG YAML example:

name: SingleSuite
threadCount: 4
parameters: { n: 42 }
tests:
  - name: Regression2
    parameters: { count: 10 }
    excludedGroups: [ broken ]
    classes:
      - test.listeners.ResultEndMillisTest
      - test.listeners.TimeOutTest

Unfortunately, YAML and JSON don’t have as much tooling support as XML especially when it comes to editing and validation, but one thing that I liked in YAML is that it’s much more “copy/paste friendly” than XML. In XML, it’s rarely possible to just cut a line and paste it somewhere else in the file: you usually end up having to add surrounding tags or move closing tags. In YAML, you can usually just cut/paste lines right away. It’s also much easier to comment out regions than XML’s awkward “<!--” and “-->” delimiters.

I discussed my choice of YAML in this blog entry, and here is the official TestNG documentation.

Guice

From the very early days (around 2006), TestNG has made it possible for users to take control of the instantiation of their test classes thanks to the very useful IObjectFactory interface. Developers in need of creating the instances of their test classes themselves in order to prepopulate them could use this factory to perform whatever operations they needed and then return these instances back to TestNG, which would then use these to run the tests.

The release of Guice made this interface even more useful: instead of instantiating their test objects, users were now able to simply ask Guice to hand them a fully injected instance. This worked great but considering the number of discussions and requests that we received on the mailing-list on this topic, I started wondering if we couldn’t provide an even better way to support Guice directly with TestNG.

A few hours later, TestNG’s official Guice support was born:

@Guice(modules = GuiceExampleModule.class)
public class GuiceTest {
  @Inject
  ISingleton m_singleton;
  @Test
  public void singletonShouldWork() {
    m_singleton.doSomething();
  }
}

In this example, TestNG will use the GuiceExampleModule module to retrieve an instance of the GuiceTest class and then use that instance to run the tests.

You no longer need to use the object factory, you can now directly tell TestNG which Guice module(s) should be used to create the instance of a given test. Here is the original blog post that started the discussion (note that the syntax changed slightly since then) and the direct link to the documentation.

Improved Eclipse plug-in

I have been adding a lot of features to the Eclipse plug-ins over the past months, among which:

• A summary tab that allows you to browse the results very easily (shown partially above).
• A search functionality for whenever you need to look for the result of a specific test among hundreds.
• Vastly improved automatic conversions from JUnit 3 and JUnit 4 to TestNG.
• A revamped display view that matches the testng.xml format more closely.

Here is a quick rundown of the latest features and the full documentation.

Miscellaneous

These are the big items, here is the full change log, along with the contributors:

Core

• Added: @Guice(moduleFactory) and IModuleFactory
• Added: @Guice(module)
• Added: timeOut for configuration methods
• Added: -randomizesuites (Nalin Makar)
• Added: IConfigurable
• Fixed: @Test(priority) was not being honored in parallel mode
• Fixed: @Test(timeOut) was causing threadPoolSize to be ignored
• Fixed: TESTNG-468: Listeners defined in suite XML file are ignored (Michael Benz)
• Fixed: TESTNG-465: Guice modules are bound individually to an injector meaning that multiple modules can’t be effectively used (Danny Thomas)
• Fixed: Method selectors from suites were not properly initialized (toddq)
• Fixed: Throw an error when two data providers have the same name
• Fixed: Better handling of classes that don’t have any TestNG annotations
• Fixed: XmlTest#toXml wasn’t displaying the thread-count attribute
• Fixed: TESTNG-415: Regression in 5.14.1: JUnit Test Execution no longer working
• Fixed: TESTNG-436: Deep Map comparison for assertEquals() (Nikolay Metchev)
• Fixed: Skipped tests were not always counted.
• Fixed: test listeners that throw were not reporting correctly (ansgarkonermann)
• Fixed: wasn’t working.
• Fixed: In parallel “methods” mode, method interceptors that remove methods would cause a lock up
• Fixed: EmailableReporter now sorts methods chronologically
• Fixed: TESTNG-411: Throw exception on mismatch of parameter values (via DP and/or Inject) and test parameters
• Fixed: IDEA-59073: exceptions that don’t match don’t have stack trace printed in console (Anna Kozlova)
• Fixed: IDEA’s plug-in was not honoring ITest (fixed in TestResultMessage)
• Fixed: Methods depending on a group they belong were skipped instead of throwing a cycle exception
• Fixed: TESTNG-401: ClassCastException when using a listener from Maven
• Fixed: TESTNG-186: Rename IWorkerApadter to IWorkerAdapter (Tomás Pollak)
• Fixed: TESTNG-415: Assert.assertEquals() for sets and maps fails with ‘null’ as arguments
• Fixed: typo -testRunFactory
• Fixed: NPE while printing results for an empty suite (Nalin Makar)
• Fixed: Invoke IInvokedMethodListener.afterInvocation after fixing results for tests expecting exceptions (Nalin Makar)
• Fixed: TESTNG-441: NPE in SuiteHTMLReporter#generateMethodsChronologically caused by a race condition (Slawomir Ginter)

Eclipse

• Added: Convert to YAML
• Added: New global preference: JVM args
• Added: Eclipse can now monitor a test-output/ directory and update the view when a new result is created
• Added: Right clicking on a class/package/project now offers a menu “TestNG/Convert to TestNG”
• Added: Excluded methods are now listed in the Summary tab
• Added: “Description” column in the excluded methods table
• Added: Dialog box when the plug-in can’t contact RemoteTestNG
• Added: Double clicking on an excluded method in the Summary tab will take you to its definition
• Added: If you select a package before invoking the “New TestNG class” wizard, the source and package text boxes will be auto-filled
• Added: When an item is selected in a tab, the same item will be selected when switching tabs
• Added: A new “Summary” tab that allows the user to see a summary of the tests, sort them by time, name, etc…
• Added: It’s now possible “Run/Debug As” with a right click from pretty much any element that makes sense in the tree.
• Added: JUnit conversion: correctly replaces assertNull and assertNotNull
• Added: JUnit conversion: removes super.setUp() and super.tearDown()
• Added: JUnit conversion: removes @Override
• Added: JUnit conversion: replaces @Test(timeout) with @Test(timeOut) (5.14.2.4)
• Added: JUnit conversion: replaces @Test(expected) with @Test(expectedExceptions) (5.14.2.4)
• Added: JUnit conversion: replaces fail() with AssertJUnit.fail() (5.14.2.2)
• Added: JUnit conversion: replaces Assert with AssertJUnit (5.14.2.1)
• Added: The progress bar is now orange if the suite contained skipped tests and no failures
• Added: Skipped test and suite icons are now orange (previously: blue)
• Added: New method shortcuts: “Alt+Shift+X N”, “Alt+Shift+D N” (Sven Johansson)
• Added: “Create TestNG class” context menu
• Added: When generating a new class, handle overridden methods by generating mangled test method names
• Fixed: Green nodes could override red parent nodes back to green
• Fixed: Was trying to load the classes found in the XML template file
• Fixed: Stack traces of skipped tests were not showing in the Exception view
• Fixed: XML files should be run in place and not copied.
• Fixed: NPE when you select a passed test and click on the Compare Result icon (Mohamed Mansour)
• Fixed: When the run is over, the plug-in will no longer force the focus back to the Console view
• Fixed: The counter in the progress bar sometimes went over the total number of test methods (5.14.2.9)
• Fixed: org.eclipse.ui.internal.ErrorViewPart cannot be cast to org.testng.eclipse.ui.TestRunnerViewPart (5.14.2.9)
• Fixed: Workspace preferences now offer the “XML template” option as well as the project specific preferences (Asiel Brumfield)
• Fixed: TESTNG-418: Only last suite-file in testng.xml run by Eclipse plugin

Documentation

• Added: Section on Selenium (Felipe Knorr Kuhn)
• Added: Link to an article on TestNG, Mockito and Emma in the Misc section

Upgrading to TestNG 6.0

Ant users can download TestNG 6.0 directly from the web site while Maven users only need to specify the following dependency:

    
      org.testng
      testng
      6.0
    

Don’t forget to update your Eclipse plug-in as well.

If converting your tests to TestNG is one of your new year’s resolution, you are in luck.

Introducing the improved JUnit to TestNG converter.

A couple of months ago, I gave a preview of the new features in the TestNG Eclipse plug-in and I observed that more and more people were converting their tests from JUnit 3 and JUnit 4 to TestNG. This latter was a surprise to me since I never really expected anyone would want to move away from JUnit 4 once they have migrated to it.

TestNG has long supported individual class conversions in the form of Quick Fixes:

I recently expanded this support and turned it into a full refactoring, which means that you can now apply it to entire packages, source folders or even projects.

To use it, install the latest TestNG Eclipse plug-in, open your Package Explorer and right click on either a package, a source folder or a project:

The refactoring wizard contains two pages.

The first one lets you customize the testng.xml that’s about to be generated (you can also choose not to generate one):

The next page shows you a view with all the changes that are about to be made to your source files. These changes are similar to the ones made with the Quick Fix except that they now apply to multiple files:

On this page, you can exclude certain files from being converted, and when you’re happy, press the Finish button.

Like all refactorings in Eclipse, you can undo your changes in one click if you change your mind:

In a next post, I’ll show how you can use this new functionality to help you check that your unit tests are as isolated as you think they are.

Now go ahead and convert your tests, that’s one less new year’s resolution you have to worry about!