On Saturday 09 August 2008 23:26:28 Ryan Davis wrote:
> 
> On Aug 9, 2008, at 06:42 , Farrel Lifson wrote:
> 
> > Rubinius and JRuby use a shared RSpec suite which they use to test
> > compatiability. It's not the Official Spec(tm) but I think it's the
> > closest to one.
> 
> that is so far from an actual specification it isn't funny.

Erm, how? Is there something particularly deficient about the actual specs?
Or are you just uncomfortable with having a test suite be a specification?

If it's the latter, read on.


My thoughts: If you've ever filed a decent bug report, you've already done 
this. Even for an "enhancement", even for older trackers like Bugzilla, 
there's an expectation that you'll write something like this:


Steps to reproduce:
1. Log in
2. Go to your profile
3. Click on 'foo'

Expected result:
It should 'bar' when I click 'foo'.

Actual result:
I got an "Internal Server Error: 500" error. Log pasted below.


Now, actually parse that out. Sounds exactly like a regression test, to 
me. "Steps to reproduce" are steps to setup the test. "Expected result" is 
your assertions. 

Again, who says it has to be a bug? Test-driven development suggests that you 
should code by writing the test first, then fix it. Shouldn't the same hold 
true for enhancements, also? All that would change is that "Expected result" 
would be some feature that doesn't exist yet.

Behavior-driven development (behaviour-driven.org) and supporting software 
(rspec.info) only make this relationship more explicit. Given a 
properly-formatted feature specification (similar to above), you can actually 
generate skeleton integration tests (called "stories").

It works the other way around, too -- Rspec will generate English-language 
versions of its tests, which can be used as a specification.

When you think about it, any open standard should have at least an open 
implementation. If it is meant to allow multiple implementations, it should 
also have an automated test suite -- for the Web, we have validator.w3.org, 
as well as the Acid2 (and now Acid3) tests, as an easy way to measure a 
particular implementation's adherence to the standard.

We all agree that a suite of compatibility tests is a good thing, right?



Here are those links again, in case your client didn't make them into links:

http://behavior-driven.org
http://rspec.info
http://validator.w3.org