Vladimir Sizikov wrote:
> Hi,
>
> A bit of clarifcation:
>
> On Fri, Jul 4, 2008 at 7:39 PM, Igal Koshevoy <igal / pragmaticraft.com> wrote:
>   
>> For example, I run the checker against my patched version of p111 and then
>> again with your latest SVN release. When the checks finish, I use gvimdiff
>> (or meld, kdiff3, etc) to visually compare the reports (e.g., output of the
>> RSpec test suite) to determine if the new version if failing tests that
>> worked with the previous version, and then review these tests to see why
>> they're failing.
>>     
>
> RubySpec suite uses "ruby_bug" guards to mark some tests that fail
> with MRI, and those guards can have the version info (including
> patchlevel) to determine whether the test should be filtered out or
> not.
>
> So, if MRI 1.8.6 pl 114 produces clean results, but the very latest
> from 1_8_6 branch shows some errors, that does not necessary means
> that there is a new regression, it also might mean that ruby_bug
> guards are not updated to the latest patchlevel version and allow to
> run the spec that still fails.
>
> E.g.:
>
> ruby_bug "#bug_id", "1.8.6.114"
>
> will filter out the test for pl 114, but will allow to run it in pl 256.
>
> It is good to be aware of that.
>
> Also, if you'd like to run ALL specs, even those filtered out by
> ruby_bug guards, provide the additional command line parameters: -n
> blah  (that will fake the ruby *name* to be 'blah', and ruby_guard
> will not recognize MRI as MRI thus running all the specs).
>
> If you run like that, then you could for sure use the "diff" comarison
> method for failing specs to find out the new regressions. But be
> aware, that the spec run might not even finish if you have -n blah
> option (for example, due to MRI bugs like hangups and crashes that are
> typically guarded by the ruby_bug guards).
>   
Thanks for writing that up, Vladimir. Sorry for making too broad a 
generalization.

Everyone that's starting with RubySpec, please spend the time to read 
and understand what Vladimir just explained.

When I was began work with RubySpec and the new releases, I didn't 
understand the guard conditions and inadvertently reported a number of 
things that I thought were new bugs or regressions. However, these were 
merely guard conditions in the RubySpec code that were written with the 
incorrect assumption that particular Ruby bugs existing only up to p114 
and were fixed in all later revisions, while in fact those bugs are 
still present. Tanaka Akira quickly jumped in and help explain the issue 
to me (thanks!) so we could identify the portions of the code that were 
really new bugs and regressions. Since then, Federico, Arthur, Vladamir 
and others have reworked most of those problematic guard conditions, but 
it's still necessary to read beyond the diffs and understand what the 
passing/failings specs actually mean.

-igal