Keith Ray <k1e2i3t4h5r6a7y / 1m2a3c4.5c6o7m> wrote in message news:<k1e2i3t4h5r6a7y-7C2620.18082110122001 / news.attbi.com>... > In article <ded6b237.0112100935.57024f11 / posting.google.com>, > rbinder / rbsc.com (Bob Binder) wrote: > > > BTW, the apparent meaning of "adequate testing" in this thread is not > > consistent with definition used in the testing community for over 15 > > years: adequate testing must *at least* achieve statement coverage. > > There are many other possible definitions of "adequate testing", but > > if your tests don't at least reach all the code in the IUT, you can't > > claim to be doing adequate testing. Although any definition of > > "adequate testing" which attempts a more stringent criteria can be > > falsified, many unambiguous non-subjective criteria for adequacy have > > been useful in practice, but they don't include "I say so". > > Didn't the tests in the examples by Kent and Uncle Bob have 100% > statement coverage, so by your definition, they are "adequate"? There is no evidence that this has (or would) have happened for an actual implementation. Measuring coverage requires using a coverage analyzer or equivalent instrumentation. Simply calling every method in a (sub)class interface does not guarantee coverage of the statements in the method implementations. Excluding toy problems, it is completely impractical to attempt to assess coverage without using an automated coverage analyzer. Lest there be any confusion, I should also say that achieving statement coverage is "adequate" testing *only* in a very narrow technical sense. It is usually easy to achieve statement coverage and miss lots of nasty bugs. But it is also very easy to incorrectly conclude that a "black box" (behaviorally-based) test suite has reached all statements. In fact, repeated experiments have shown that without the feedback of a coverage analyzer, you rarely get to more than 2/3s of the code using only a "black-box" testing approach. The narrow technical sense of "adequate" is based on the fact that you *must* at least exercise buggy code to reveal a bug. You have zero chance of revealing a bug if you don't exercise buggy code. On the other hand, exercising buggy code does not equal a 100% chance of revealing a bug. So, if you can't demonstrate by coverage analysis that your test suite has reached all the code in the scope of the test, then you have almost certainly not gotten to everything. This untouched code is often *more* likely to be buggy. You've probably tried all the obvious stuff, so what remains can only be reached under unusual, non-obvious conditions, and you may not have completely worked out how your app should deal with these conditions (if you had, you'd probably have run a test to show it off.) The chance of lurking bugs in un-exercised segments usually increases for mystery code that has been "reused" or maintained for a long time. The fact that it may be infeasible or impractical to reach some code segments, meaning that you can't always get 100% coverage, doesn't diminish this basic result. Once you get the hang of it, running a coverage analyzer is about as hard as running a spell checker -- it is not the god-awful monster that many people seem to think it is. But just like a document with no spelling errors is not necessarily a good document, getting 100% statement coverage with a coverage analyzer doesn't mean you've necessarily done a good job of testing. Achieving 100% branch coverage is one of the exit criteria in almost all of the test design patterns presented in my book. The additional exit criteria are pattern-specific, and determined by the test model and probable bugs of the implementation under test. Achieving these exit criteria are what I consider to be adequate testing. _________________________________________________________________ Bob Binder http://www.rbsc.com RBSC Corporation rbinder@ r b s c.com Advanced Test Automation 3 First National Plz 312 214-3280 tel Process Improvement Suite 1400 312 214-3110 fax Test Outsourcing Chicago, IL 60602