2015-01-06 7:18 GMT+09:00 Charles Oliver Nutter <headius / headius.com>:
> I'll try to be brief so we can discuss all this. tl;dr: RubySpec is
> valuable, MRI tests are valuable, we need to better utilize both of
> them.

Other points.

It is difficult to distinguish between specification and non-specification.
If the distinction is required before writing a test, it discourage
writing a test.
Also, non-specification behavior can be de facto standard eventually and
finaly be a specification.
I think the distinction should be easily changed after a test is written.

It is not appropriate to block a release arbitrary from outside of project.
If a specification is changed too fast or changed just before a release,
it is nearly impossible to release a project without test failures.
So what tests to be applied should be choosen by each project.

Specification is version dependent.
We change specification to improve language and library.
However the improved specification is not applicable for older implementations.
RubySpec is interesting here but even RubySpec doesn't support too old Ruby.
RubySpec drops Ruby 1.9.x at 2013-10-21 (commit
0e4e64cfd431f6f7aec0171af80065154b377561)
while Ruby 1.9.3 is still supported.
(The end of security fix for Ruby 1.9.3 is 2015-02-23.  It is still alive.)
https://www.ruby-lang.org/en/news/2014/01/10/ruby-1-9-3-will-end-on-2015/
This may be similar to above: what tests to be applied should be
choosen by each version.

I feel "excludes" feature recently added by [Feature #10682] is a good approach.
https://bugs.ruby-lang.org/issues/10682
It makes easy to choose what tests to be applied.

Of course, tests for mature behavior (such as behaviors specified in
ISO spec.) can
be extracted to choose them easily.
-- 
Tanaka Akira