On 2015/01/06 07:18, Charles Oliver Nutter wrote:
> I'll try to be brief so we can discuss all this. tl;dr: RubySpec is
> valuable, MRI tests are valuable, we need to better utilize both of
> them.

I agree. I think that one problem with test frameworks in general is 
that they are too easy to write, and people have too many opinions about 
them, so there are too many variations.

Ruby always has had an inclusive rather than prescriptive culture (e.g. 
when compared to Python), so the fact that there are two test suites may 
not be that bad. Also, mixing and matching things, and combining stuff 
from different sources, is something that is very frequent in Japanese 
culture.

> As you may have heard, Brian Shirai officially ended the RubySpec
> project last week:
> http://rubini.us/2014/12/31/matz-s-ruby-developers-don-t-use-rubyspec/

I have read it. As usual, Brian is very good at stating his view :-(.


> I believe this is a good opportunity for us to address concerns about
> how Ruby behavior is being tested over time.
>
> I think we can agree on the following facts:
>
> * ruby-core devs *have* been using RubySpec on and off, and at least
> some of us see value in continuing to do so. naruse already maintains
> a fork at github/naruse/rubyspec.

As far as I have heard from Yui, he has good reasons why it's forked.

I have tried to run it recently, on Cygwin with cruby trunk. My first 
comment would be that I got some 'F's, and after about 5 lines of 
....F.... output, it got stuck (not even responding to Ctrl-C). I had no 
idea how far it got, or what went wrong, so I abandoned it for the 
moment. But this is just my personal experience.


I think contributions to both MRI's test suite and to RubySpec are very 
welcome. We are getting lots of small but valuable contributions to the 
documentation from what I understand are people who use this as a way to 
study Ruby. Maybe something similar could work for tests; it would be 
somewhat more difficult than working on documentation but may have a 
similar educational effect and likewise produce very valuable improvements.

Regards,    Martin.


> * MRI's suite is not nearly as bad as the post claims. I have done
> some coverage measurements that show it doing at least as well as
> RubySpec for core classes, and significantly better for standard
> libraries.
>
> * MRI's tests are often hard to read, too large, undercommented.
> There's a lot of room for improvement, but this is obviously an
> organic codebase that has had many contributors.
>
> * Ugly tests are harder for users (and contributors) to grok. Ugly
> tests are less attractive targets for contribution.
>
> * RubySpec represents a valuable set of tests for Ruby behavior. There
> will be a large overlap with MRI's tests, but RubySpec does things
> that MRI's suite does not (and vice versa).
>
> * RubySpec is not gone; Rubinius will still need a suite to verify
> Ruby compatibility. RubySpec lives in the Rubinius repository now.
>
> There's a number of paths forward for us right now.
>
> * Do nothing. Run the last public RubySpec dump periodically to ensure
> we don't regress, as has been done with naruse's fork. New tests
> continue to go into MRI suite.
>
> This is the status quo, and I don't think anyone benefits from it
> other than folks who dislike change.
>
> * Begin an effort to bring missing tests from RubySpec into MRI's
> suite and to clean up and improve MRI's suite. Abandon RubySpec at
> some point.
>
> I think *some* effort should be made to improve MRI's suite in the
> short term, since we obviously aren't going to throw it away. I don't
> like the idea of spending a lot of time porting tests one way or the
> other...at least until we've picked a winner. I don't like the idea of
> abandoning RubySpec, since it has many valuable features.
>
> * Take over RubySpec as a fork and begin to maintain it as a secondary
> suite for MRI. Encourage contributors and committers to write specs
> rather than tests.
>
> I think this is possible. The barrier to contributing to RubySpec is
> much lower if we're maintaining a fork of it, and I think that
> eliminates many obstacles. I also feel like the future needs to be in
> either RubySpec or a much cleaner version of MRI's suite, and there's
> less work to start using the former now.
>
> ...
>
> So that's what I have for now. Give me your thoughts...anything goes.
>