Thank you Shyouhei-san for your reply.

On Wed, Jan 4, 2017 at 2:52 AM, Urabe Shyouhei <shyouhei / ruby-lang.org>
wrote:
>
> I once was a SET (Software Engineer in Test) at a company of
> 1,000+ employee.  My role there included writing tests of someone
> else's products, much like the situation of rubyspec.  Why that
> company had dedicated test-automation engineers was because
> writing non-garbage acceptance tests in general requires a
> different skill set from writing a production code.  Of course,
> people wrote their own unit tests and that's definitely a good
> thing.  But they did so to _boost_ their development, not to be
> conservative about their specifications.
>

I think the difference is much smaller between ruby/spec and test/ruby,
essentially it's the same code as unit tests, but in smaller chunks,
each chunk with a plain English description. For me, the "spec" in the name
is
mostly to refer to these descriptions and the fact it tries to explain the
behavior.
It is not a formal specification of the Ruby language, it just intends to
test the Ruby language,
its core and standard libraries and also describes the reason behind the
behavior.

I think the person implementing a new feature is usually the most
knowledgable
about the feature and implementation, and therefore can give relevant tests
and
write helpful descriptions. Nothing is perfect, so there are many cases
which are
not covered or sometimes even misleading, but I think ruby/spec is a very
valuable
effort overall.

So when it comes to ruby.  I don't think test/ruby is the ideal
> solution for everyone.  I also admit it's ugly, and needs
> improvements.  But at least I can say it's a wrong idea to
> abondon it.  Doing so is much like to throw away unit tests and
> let Selenium do everything.  Just nonsense.  Unit test is not a
> spec.  They exist separately because they have different
> purposes.  Merging them hurts both sides.  The (failed) past
> attempts to migrate rubyspec had problems on this point, seems to
> me at least.
>

I do not propose to abandon test/ruby neither to merge it with ruby/spec
as that seems unrealistic and of limited value. But, I would like to
encourage
using ruby/spec instead (or in complement) of test/ruby. I think it is
quite similar
to write ruby/specs or test/ruby tests, but I think ruby/spec tests
have many advantages as described above.

Maybe some people dislike or do not want to write ruby/spec, and that is
fine.
But if a few MRI committers contribute more actively to ruby/spec,
it would already be a huge step towards testing and understanding Ruby
better.
The whole Ruby ecosystem would benefit as alternative implementations would
be able to follow changes more efficiently and contribute back specs for
additional cases.


> That said.  I also have to note that tests without specs is a
> poor solution.  The problems pointed by @eregon in this thread is
> worth listening to.  The company I worked before solved this
> problem by hiring someone to write specs.  That's not doable by
> us (no such budget) but someone has to do that job somehow.  Are
> we going to let the core devs do that?  Then I think we need to
> motivate them.  "Spec is a good thing you should do" is a true
> assertion but sounds much like a homily.  Instead show them it
> benefits.  For instance, find bugs using spec and show them how
> it helps their developments.
>

Indeed, I found many bugs using ruby/spec.
Testing from the perspective of trying to explain the behavior can discover
many interesting cases.

I reported 70 bugs on the Ruby tracker in 7 years.
The majority of them where discovered by specs.

Here are a few examples:
* Inconsistent Float#round on Windows
* Various difference of behavior on Windows
* Coverage is leaking in global state
* ObjectSpace.each_object exposes internal objects
* Deadlock in #autoload
* Incorrect encoding for Time#zone
* Bugs in keyword arguments, there were many of them back then
* Various methods changed the exception they throw in error cases

Alternative implementations also have their own test suites.
They could each have a different one and duplicate the efforts,
or each use the same test suite known as ruby/spec to share the benefits.
(supressed text/html)
Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>