Apparently, Ben Giddings recently wrote:
> Wesley J. Landaker wrote:
>> What about some other extreme cases? Like, a class that plays a sound on
>> a
>> speaker? How do you unit test that? Or, likewise, any class that
>> interfaces with the "world" "outside" your control? What if they are so
>> complex that you can't easily or acurately model them? What about output
>> that "needs" human interaction?
>
> I also regularly deal with cases that involve interfacing to external
> systems.
>
> If I want to test something that grabs a web page, what happens if due
> to network issues, my access times out?  If what you're really
> interested in is not related to the network, then you can either test to
> see if you have the right URL and headers to fetch the web page, or fake
> the fetch and see if your code handles the data properly, but what if
> the web access is essential?

Just to add some more fuel to the discussion: another real general place
where this comes up a lot is graphics applications.

From a real raytracer project I did: How do I know that my raytracer
rendered a sphere correctly? Well, I look at the output image and say
"hmmm... yeah, it looks like a 3d sphere..." -- there were a lot of things
like that. I could unit test the guts, but often writing the tests
themselves required first writing a little program to compute what the
results should be... ;)

Suffice to say, on that particular project, I ended up just unit testing
some core pieces (where, arguably it was simple enough that I didn't
really need the tests) and just doing ad-hoc testing on the larger pieces
that did complex math or actual rendering (but, ironically, this is where
I would have benefitted the most from test-first unit-tests, because it
was so hard to get right!)

> An example would be a bit of code that tries to use some data to figure
> out what URL to use to get what it needs.  The only way to test to see
> if that URL is correct is to see if the web page it fetches contains the
> data it expects.  Because of the nature of the problem (the changing
> web), you can't tell by looking at the URL if it is correct, you need to
> see what it fetches.  You could always retry if a network error occurs,
> but what if the error is a 500 series HTTP error (i.e. your URL was
> correct but the server is temporarily broken)?
>
> Are there general unit testing strategies for dealing with things
> outside the system you have control over?

I think it's pretty clear that there is set of "un-unit-testable" things.
I wonder, though, if anyone has some brilliant ideas on how to mitigate
this class of problems. So far what I have in my head is:

 1) Don't bother unit testing (yikes!).
 2) Do regression tests with "golden" results (i.e. compare against
previous output -- this assumes no test-first, and that you at some point
got it right and verified by hand--this, IMO, is a lot to assume)
 3) Make use of mock-objects to simulate an external interface (only works
if it's low-enough complexity for you to model)
 4) ???
 5) PROFIT!! (Okay, maybe not these last two... ;)

Wes