Keith Ray <k1e2i3t4h5r6a7y / 1m2a3c4.5c6o7m> wrote in message news:<k1e2i3t4h5r6a7y-7C2620.18082110122001 / news.attbi.com>...
> In article <ded6b237.0112100935.57024f11 / posting.google.com>,
>  rbinder / rbsc.com (Bob Binder) wrote:
> 
> > BTW, the apparent meaning of "adequate testing" in this thread is not
> > consistent with definition used in the testing community for over 15
> > years: adequate testing must *at least* achieve statement coverage.
> > There are many other possible definitions of "adequate testing", but
> > if your tests don't at least reach all the code in the IUT, you can't
> > claim to be doing adequate testing. Although any definition of
> > "adequate testing" which attempts a more stringent criteria can be
> > falsified, many unambiguous non-subjective criteria for adequacy have
> > been useful in practice, but they don't include "I say so".
> 
> Didn't the tests in the examples by Kent and Uncle Bob have 100% 
> statement coverage, so by your definition, they are "adequate"?

There is no evidence that this has (or would) have happened for an
actual implementation. Measuring coverage requires using a coverage
analyzer or equivalent instrumentation. Simply calling every method in
a (sub)class interface does not guarantee coverage of the statements
in the method implementations. Excluding toy problems, it is
completely impractical to attempt to assess coverage without using an
automated coverage analyzer.

Lest there be any confusion, I should also say that achieving
statement coverage is "adequate" testing *only* in a very narrow
technical sense. It is usually easy to achieve statement coverage and
miss lots of nasty bugs.  But it is also very easy to incorrectly
conclude that a "black box" (behaviorally-based) test suite has
reached all statements.  In fact, repeated experiments have shown that
without the feedback of a coverage analyzer, you rarely get to more
than 2/3s of the code using only a "black-box" testing approach.

The narrow technical sense of "adequate" is based on the fact that you
*must* at least exercise buggy code to reveal a bug. You have zero
chance of revealing a bug if you don't exercise buggy code. On the
other hand, exercising buggy code does not equal a 100% chance of
revealing a bug.

So, if you can't demonstrate by coverage analysis that your test suite
has reached all the code in the scope of the test, then you have
almost certainly not gotten to everything. This untouched code is
often *more* likely to be buggy.  You've probably tried all the
obvious stuff, so what remains can only be reached under unusual,
non-obvious conditions, and you may not have completely worked out how
your app should deal with these conditions (if you had, you'd probably
have run a test to show it off.) The chance of lurking bugs in
un-exercised segments usually increases for mystery code that has been
"reused" or maintained for a long time.

The fact that it may be infeasible or impractical to reach some code
segments, meaning that you can't always get 100% coverage, doesn't
diminish this basic result.

Once you get the hang of it, running a coverage analyzer is about as
hard as running a spell checker -- it is not the god-awful monster
that many people seem to think it is. But just like a document with no
spelling errors is not necessarily a good document, getting 100%
statement coverage with a coverage analyzer doesn't mean you've
necessarily done a good job of testing.

Achieving 100% branch coverage is one of the exit criteria in almost
all of the test design patterns presented in my book.  The additional
exit criteria are pattern-specific, and determined by the test model
and probable bugs of the implementation under test.  Achieving these
exit criteria are what I consider to be adequate testing.

_________________________________________________________________
Bob Binder           http://www.rbsc.com      RBSC Corporation
rbinder@ r b s c.com Advanced Test Automation 3 First National Plz 
312 214-3280  tel    Process Improvement      Suite 1400
312 214-3110  fax    Test Outsourcing         Chicago, IL 60602