I was chatting with Hani last night and two interesting ideas came up:

  • Repeating failed tests.

    When a test run shows several failed tests, the first thing that
    developers want to do is rerun only those tests that failed, and not the
    entire suite.  An easy way to implement this with TestNG is to generate
    a testng-failed.xml file that contains all the methods that failed. 
    If you need to rerun these failed tests, just invoke TestNG with this XML
    file and you are done.

    The only challenge here is that this file also needs to include all the
    methods that the failed tests depend upon, but that’s pretty easy since
    TestNG already supports this.

    This feature also illustrates the importance of separating your test static
    model (business logic) from the runtime model (which tests are run). 
    If you wanted to provide a similar feature with JUnit, you would need to
    generate some Java code creating a TestSuite containing all the failed tests
    (making sure you get the imports right, etc…) and compile this suite, or
    by generating an ant file with the correct decorator.
     

  • Concurrent testing.

    The original idea was to be have the testing framework invoke a test method
    several times with concurrent threads, but I’m thinking that this should be
    extended to entire groups. You put all your methods in a certain group and
    you specify that this group should be run by a thread pool a certain number
    of times.

    I think the methods being invoked concurrently would not
    necessary contain the asserts and that the real testing would be done by a
    method later (which, obviously, will depend on that group).

    Another thing is that this specification is a runtime concern, so I think it
    belongs in testng.xml, not in the annotations.

Any thoughts?