Negative empiricism; testing

Suppose I want to test the Reverse LINQ method, to ensure it always returns the reverse of a given list. I might write a test like below:

public void Test()
{
    var list = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 };

    list.Reverse().Reverse().SequenceEqual(list);
}

The test, unsurprisingly, passes. What does this tell me? It does not tell me the Reverse method functions correctly all the time. It tells me that the Reverse method works as expected for the given list only. It does not tell me the Reverse method is devoid of bugs and works for any given sequence, we merely assume this is the case; inducing it from the limited test above.¹

Now, the above example is perhaps obviously going to work every time, because the functionality to reverse a collection is easily made universal, and one may place high trust in the LINQ methods. However, it is easy to imagine business code for which a test is written that covers only a subset of the possible inputs, as above, and yet is treated as though it proves the functionality works for all. I know I have written plenty of such tests. I think we have all known insufficient happy-path testing come back to bite us.

This isn’t a newly discovered limitation of positive unit testing; orthogonal testing has been used to test on the boundaries of wide input variations for decades. For example, tests to list an item for sale at -$1.00, $0.00, $1.00. These values are important because they sit at the boundaries of validity; $1 is as valid as $200, and -$1 is as invalid as -$5000, but the assumption still remains that what does not work for -$1.00 will also not work for -$300, and -$10,000. The inference is just that, inference. It’s certainly useful information, but it can’t be treated as exhaustive proof.

More recently, libraries like QuickCheck (and FsCheck) have tried to resolve the issue by leveraging the processor to perform exhaustive testing. The limitation is though, that for more complex tests than the above, the developer is required to build models for the test inputs, and what isn’t modelled, or is insufficiently modelled, won’t be caught. Sufficiently complex business cases require equally complex models as test inputs, increasing the likelihood of the model mismatching reality and hiding faults. There is an inescapable limit to confirmation testing in this way; you can never be sure you’ve covered everything.

So what does all this tell us? Well, firstly it tells us unit tests aren’t everything. Unfortunately, they are sometimes sold as panacea for instability issues, and when taken in this way, they can make situations worse by introducing a false sense of confidence. Unit testing is a work tool and a necessity when pursuing software development from an engineering perspective, but unit tests are limited in their ability to “prove” working software. We should not blindly take a passing unit test to prove the code works, it is far more nuanced than that.²

Another more subtle thing negative empiricism tells us about our code is that in order to move our tests closer to the rare realm of high-confidence knowledge, we must limit the complexity and functionality they are testing over. TDD guides us to write in this way, by getting us to ask what minimum unit of code would cause our test to pass. The results tend to be smaller, more composable, testable units; easier to reason with, safer to trust the inference of confirmation tests, because the range of behaviour in a single method is so limited. If my functionality reverses a collection, and only reverses a collection, then I am far safer building some confidence from confirmation tests than if the method contains a multitude of side effects that are not consistently triggered. This isn’t deductive knowledge, but it is “better” inductive knowledge, because the scope of the inference is narrower.

Finally, we should hold in higher esteem deductive over inductive tests. Deductive tests are rarely seen in the main source control branches, because they would break a CI pipeline. These are the tests you write that fail, because they’ve exposed behaviour in the code you were not expecting, be that a defect or no. In TDD, deductive tests are the tests we first write to prove our functionality fails and to prove our test is effective. The test falsifies the assumption that the functionality works for the given inputs, we then modify the system under test (SUT) to meet the criteria of the failing test.

The tragedy of these deductive tests is that the moment we change the SUT to pass the test, the test becomes inductive again. For the time it is proving our code works, it is inductive, but it becomes deductive when we receive a notification that functionality is failing our assumptions. As we must have all tests passing to integrate our changes, so all deductive tests must end up inductive, until the time comes for them to fail again, protecting another developer from an unknown defect.