The science of testing (and debugging)

I was amazed (but after thinking about it, it seemed pretty obvious, how much the scientific method, and testing, have in common. Testing, and once something goes wrong, debugging, are naturally scientific ways of observing a system. Taking the steps directly from Wikipedia, one can observe how similar the steps are for testing.

1. Define the question
2. Gather information and resources (observe)
3. Form hypothesis
4. Perform experiment and collect data
5. Analyze data
6. Interpret data and draw conclusions that serve as a starting point for new hypothesis
7. Publish results
8. Retest (frequently done by other scientists)

First the question needs to be defined. In our case this means that during testing, be it Black box or White box, one needs to know what is going on. What am I looking at? What is it supposed to be doing? Is it doing what it is supposed to be doing? A system can very easily do the wrong thing in a very good manner. In such a case, the testing is said to have failed nonetheless.


The obervation part is the actual testing. Here yo uare expected to interact and observe the system itself, noting down what is happening, measuring the results, and all this without inflencing the actual execution.


Then the hypothesis comes. From the previous data, you see whether the test has failed or not. If it succeeds, you understand why. If it fails, you need to understand why too. You might need to rerun some tests.

The fourth step refers to performing the experiments and collecting the data. Try toggling some setting. See when the results are the same, and what is common in all. This is like detective work, and this is where bright mings normally start to shine and be seen as the geniuses they are.

Now we need to analyze the data. When many programmers, QA people, testers, or any other person, looks at the facts, they all see the same data. But the information is not the same to all. To some the problem is hard to find, and they are as short sighted as bats. While some other person would scream "Eureka", and go rummaging through some forgotten folder, tweak with some random setting that is forgotten by all, and smile at seeing the bug solved, while others just stare at his face asking what the hell just happened.
This contains all the last steps. One needs to come up with a hypothesis, try to dis/prove it, and act on the results.

As usual, science helps us in al situations. =)