Thaddeus L Olczyk wrote:

> In all that time, I have never once been able to produce a system of tests that meets three criteria: the tests run automatically, the tests are comprehensive, and the tests are easily maintainable. At this point I am strongly beginning to wonder whether such a system is possible.

0.
I'm with you Thaddeus, and I think you raise worthwhile questions, plus several alternative answers, about what I perceive as the over-emphasis on unit tests. (Where's the integration part ?)

For instance, I've heard the claim that "We don't need interface specifications in Smalltalk - because unit tests guarantee the quality of my code".  I don't buy this, and thus welcome your skepticism.

The questions that trouble me are: "When can we say we have enough unit tests? How can we know our unit tests are 'correctly written' ? ".

To take your concrete example, how can we know there are an adequate number of unit tests in the Ruby implementation, and how can we know that the ones in there are 'correct'  ? How can we know we have 95% of the needed unit tests in place ? Or is it more like 30%  ?

1.
If we start system building from tests first,  we could declare that the tests *are*  the "Specification" of the system. Then the question turns into: "How can we know all assumptions other components make about my component are expressed by my unit tests?".  I.e. how can we know nobody is violating the contract/specification expressed by my tests-code?

But trying to reason about violation of contracts between components, we have already stepped outside the realm of unit tests, into integration testing. So I think unit testing is over-emphasized by some methodologies, since  we have no "S-Integration".

2.
If on the other hand unit tests are *not* taken as the 'specification' of a component,  we should ask: "How can we know the existing unit tests prove all specified requirements are being fulfilled?"  And if  we can't prove this, how can we have any qualitative assurance about  the 'correctness' or even 'quality' of our code ?

3.
I believe (automated) unit tests are a good technique for discovering bugs due to changes made to the system, early on. But it is misleading to imply that since you have written and run *some* tests, your system now works "correctly". Yet this is what  a consultant might tell a customer: "Look, we have all these unit tests in place, and we run them all without errors! We don't need interface contracts. We don't need specifications. Our code is simply the best since it passes all unit tests we have written!"

-Panu Viljamaa
P.S.
Wouldn't it intuitively make sense to have another set of people writing the tests, than the ones writing the code which must pass those tests?