On 12 May 2005, at 08:59, Daniel Berger wrote:

> All,
>
> This is a long post, and I hate to sound like I'm lecturing, but...
>
> Matz's announcement of 1.8.3 beta 1 reminds me of something.  We  
> need a test suite.  A large, comprehensive test suite.  Written  
> using test-unit.  And we need it NOW.
>
> I know that Chad Fowler started something along those lines (the  
> "Rubicon" project on RubyForge).  But, that was mainly a  
> translation of RubyUnit to TestUnit.  I don't think those tests  
> were sufficient.  They were also written in the 1.6.x days, making  
> them potentially obsolete.

They're not too bad, actually.

Plus the framework handles both 1.6 and 1.8 and changes between minor/ 
major revs quite handily.

They *do* need some loving to get up to 1.8, though.

Ryan and I are using Rubicon's Array tests for implementing MetaRuby,  
and have noted missing tests in the suite, these include the block  
form of Array#fill and no tests were present for Array#zip.  I don't  
know if Ryan has checked in those patches yet.  Its not as if the  
entire core library was changed when we went from 1.6 to 1.8.

While not perfect, reinventing Rubicon because Array was missing a  
handful of tests would be quite silly.

> Another thing that concerns me is that some packages in the  
> standard library don't appear to have any test suite.  Or, at  
> least. they aren't in the 'test' directory.
>
> So, we need to fix this.  Here's a few suggestions I have:
>
> * Each class in the core library should have it's own directory  
> under 'test' in the Ruby distro.  Then each method should have its  
> own test case.  That's right - each *METHOD* (for the core Ruby  
> classes).

[snip tree]

> As things stand now we've got a single file for each core class,  
> but if we write comprehensive test cases for each method, putting  
> every test in a single file would become unwieldy.  Hence the  
> suggestion to have each method in its own file.

I don't think so:

$ ls *Array* *String* *Hash* | xargs wc -l | sort -r
     3267 total
     1363 StringBase.rb
     1224 ArrayBase.rb
      634 HashBase.rb
       16 TestString.rb
       16 TestArray.rb
       14 TestHash.rb

A 1500 line ruby file is far from unmanageable.  This also reduces  
the benefit for people making their own Ruby implementation.  Instead  
of looking through one file when implementing Array, they now have to  
look through over 100.

> As an aside, I would like to see Rake included in 1.8.3 and beyond  
> because I think it could help with automating some of these things.

Rubicon uses make and some custom C utilities.  Using rake would  
reduce the utility for people using the test suite for alternative  
Ruby implementations.  (JRuby uses this test suite, and so does  
MetaRuby.)

> Having said all this, this project is just too big for one person  
> to do in their free time.  Short of someone offering me a full time  
> salary to pursue my goal, this effort will need many volunteers.
>
> So, here's what I think we need to do:
>
> * Register a project on RubyForge (I'll handle that, depending on  
> the response I get).

I think you'd be better off working on Rubicon.  It has 75% of the  
work done for you already, has nice reports, and has many of the  
platform/version differences spelled out.

You'll simply be reinventing the wheel by writing your own brand-new  
test suite for Ruby.

> * Canvas for volunteers - one person per one or two core classes/ 
> modules (ideally).  See below.
> * Volunteers would be responsible for writing test cases for their  
> assigned classes/modules.

Please, don't reinvent the wheel.

> * The authors of the various stdlib packages would be responsible  
> for maintaining their own test suites, or for finding someone to  
> handle that task for them.

I think this should be broken apart from the core libraries.  It is  
of lesser, but equally important, utility to Ruby implementors.

> * One or two people would write the online database that would  
> handle test reports.  (Perhaps a Rails or Nitro app).
>
> Hopefully, this rant hasn't come off as condescending or rude.  I'm  
> just a little frustrated with the current state of the test suite,  
> especially when I see just how well done the Perl test suite is.   
> It was also motivated by the desire to submit core changes without  
> having to provide a custom test case every time.  I could submit a  
> diff, perhaps a bug fix, and say with confidence, "Yup, still  
> passes the test suite", to say nothing of actually finding  
> potential bugs in the already existant code, or potentially useless  
> code.
>
> Anyway, if you would like to volunteer, please respond to this post  
> or send me a personal email at djberg96 at yahoo dot com.  Indicate  
> which classes or modules you would like to be responsible for,  
> though please limit yourself to two, max, at least initially.
>
> If anyone feels that I am out of line, or that this is a waste of  
> time, I'm listening.

I think you should take a serious look at Rubicon before rolling your  
own.  Rubicon already has at least 75%, possibly 90% of the work you  
need already done.  Save yourself lots of time and give it a little  
love, and you'll have something much, much better with much, much  
less effort.

-- 
Eric Hodel - drbrain / segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E  7C11 332A 551C 796C 9F04