In article <5.1.0.14.0.20011110113326.030f3ca0 / mail.webservepro.com>,
Brian Marick  <marick / visibleworkings.com> wrote:
>
>I find I don't have many ideas for general-purpose testing tools. So here's 
>the scenario I envision:
>
>A. Tester thinks, "It would be really nice if we could put up test results 
>on an internal web page. The first page would have the high-level summary. 
>You could click to drill down to details on particular suites, or examine 
>the failures one by one, or whatever. Something like Mozilla's Tinderbox 
><http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey>, but more 
>aimed at testing." Then she finds someone's Ruby code that does that. 
>She'll have to recode the way it connects to test results, and maybe tweak 
>what it displays, but that kind of thing is easy because several people 
>have done it before, so those pieces are nicely isolated and some friendly 
>person has written documentation about how to tinker with the package.

I like this.  My goal for dTest is that it should be highly configurable 
so that you could specify different types of output 'filters' through 
either extending classes or inheriting from them(if the filter didn't 
exist already).  One of these could be 
an html output filter. (actually, you could have differnt formats (html, 
txt, roll your own) and differnt methods of posting results (file copy, 
ftp, etc.)  BTW: it is the same for accessing testcases - you should be 
able to find your testcases: 1) locally (and then you would have 
different access methods - copy, cvs) or 2) on the 
'net (and again you would specify how it should be accessed: ftp, http, 
scp, cvs...).

>
>The hard part is getting the first person to write one in Ruby (and then 
>getting that person's company to let it be released).
>
>That said, here are some ideas, general or no:
>
>- a tool to distribute tests to set up remote machines, distribute tests 
>there,
>   run the tests, collect the results. (This is what Phil's article talks 
>about.)
>
Yes, this is the idea of dTest.

>- tools to summarize status and produce reports. The test-results viewer
>   above is one example. Another would work with bug reports. Another
>   would track "churn" - find files that have been changed often, or
>   changed by many people. Those files likely have more bugs.

I've often thought about some way of interfacing with the bug system (and 
thereby incorporating the bug cases into the pool of testcases) the only 
problem is that everybody uses different bug reporting systems.  Maybe we 
could try integrating with one of the free (open source) bug systems 
(bugzilla?).


>- A Ruby version of TCL's expect (http://expect.nist.gov/), which is a
>   package for talking to apps with a command-line interface, like
>   modem dialers or FTP commands.
>

There is supposed to be some kind of expect.rb module written in Ruby out 
there, bit I don't see it on the RAA.

>- a tool to reserve machines, hardware boards, or other scarce things
>   needed for tests. This wheel keeps being reinvented.
>
>- maybe a stress test driver (something that forks off a bunch of tests all
>   at once).
>
>- a tool to populate databases with test data, maybe randomly, maybe
>   with variation.
>
>- a tool to combine lists of test data such that every pairwise combination
>   of values is present. (That is, if A=[on, off] and B=[red, yellow, green],
>   the result is [on, red], [on, yellow], etc. It gets more complicated if you
>   have more variables and you want a minimal list.)
>

This is sort of the idea I have for using Ruby Templates.

Say you have a template file that you will use to generate testcases, 
like:

begin
A = #{width}
B = #{length}
C = #{depth}
end

Now let's say you want to generate testcases with the following length for 
width, length, depth:

width = (0..10)
length = [5,7,9,11,42]
depth = (4..8)

Then you could iterate through each of the ranges/arrays to create a 
catesian product and plug the values in as you use the TemplateFile class.
(I don't have all the details here now, but it looks like you could use 
the enumerable tools module to do something like:

for width,length,depth product do 
   tf = TemplateFile.new("template",binding)
   tf.transform #and you could write the resulting string to a file
                #and call it a testcase
end

This way you create one testcase template to generate multiple testcases.  
I found this pretty useful for the type of testing I was doing.

Phil