Is it possible to have 100% unit test coverage and yet have an application that fails?


Consider the following code:

public class Main {
public void openFile() {}
public void filterFile() {}
public static void main(String[] argv) {

We have tests for openFile() and filterFile() and they all pass, however, since the main method should be calling openFile() first, our application is broken.

The problem here is that we only have unit tests but no functional (or “end-to-end”) tests. In this case, only one functional tests would suffice (it would invoke main() and make sure that the application does what it’s expected to do). Note also that just like testing openFile() and filterFile(), having one test that invokes main() will result in 100% coverage as well, but it should be clear to everyone now that 100% coverage doesn’t always mean that the code you are testing actually works.

Does this mean that unit tests are useless and that we should only write functional tests?

Not exactly, but I definitely want to make something very clear: while unit tests have been receiving a lot of exposure these past years, functional tests are ultimately the only way to guarantee that your application works the way your customers expect.

Let’s write a few tests for the code above to make things clearer:

public class MainTest {
@Test(groups = { "unit", "fast" })
public void openFileShouldWork() { ... }
@Test(groups = { "unit", "slow" })
public void filterFileShouldWork() { ... }
@Test(groups = { "functional", "slow"})
public void mainShouldWork() { ... }

For illustration purposes, I’m assuming that filterFile() is slower to run than openFile(), so I put this method in the “slow” group. This test class if fairly extensive: it covers unit and functional tests, and achieves more than 100% coverage.

More than 100% coverage? Is this even possible? I know it sounds a bit strange, but it simply means that running all the tests in this class will result in certain portions of our main application to be run several times, which is not always a bad thing. Another way to look at it is that running only the group “functional” will result in 100% coverage (as will running only the group “unit”).

Assume that we only run the group “functional” and the test fails. We now know that our application is broken, but we don’t know exactly where: main() invokes openFile() and filterFile(), and either could be broken. We won’t find out until we start testing them individually.

If we run the entire class, not only will the functional test fail, but at least one of the unit tests will break, therefore pointing us directly to what’s wrong in our application. The virtue of unit tests is therefore to help us, the programmers, pinpoint errors when they occur. But keep in mind that traditionally, unit tests are of no interest to users: they are just here to help developers.

Ideally, you want your application to be covered by both unit and functional tests, but if you are pressed for time and you need to choose between implementing a unit test or a functional test, I therefore recommend going for the latter (because it serves your users) and then down the road, implement more unit tests as you see fit (for your own comfort).

Note: this topic (and many others) is covered in greater details in our upcoming book.