On Sun, Oct 07, 2007 at 03:02:15AM +0900, M. Edward (Ed) Borasky wrote:
> Chad Perrin wrote:
> >
> >I don't think that's entirely the case (or even close-to-entirely).  I
> >don't just participate in the Ruby community -- I'm as much an on-again,
> >off-again member of a couple other language communities as I am of the
> >Ruby community here at ruby-talk (if not more so).  One in particular is
> >the Perl community, and just to make a point I'll compare the two on this
> >subject.
> >
> >When people in each community look at comparative benchmarks for Ruby and
> >Perl, and when they see that Perl fairly consistently kicks the crap out
> >of Ruby for execution speed, I see that in *both* communities they tend
> >to follow that up with something like ". . . but micro-benchmarks are
> >pretty much useless.  Only real-world performance testing matters, and
> >usually even that is just a case of premature optimization."  In other
> >words, considering that even the usual "winner" in such a "contest"
> >dismisses those benchmarks as mostly useless in that respect, I don't
> >think the problem people in the "losing" camp have with the way people
> >tend to want to use benchmarks is primarily one of wanting to ignore what
> >shines unfavorable light in their direction.
> 
> That's an easy way to dismiss the results, but you're assuming
> 
> 1. That micro-benchmarks are pretty much useless. I don't agree. 
> Processors, compilers, interpreters, virtual machines and languages can 
> only be measured and optimized using micro-benchmarks.

No, I'm not assuming that.  I'm pointing out that objections to using
those micro-benchmarks to choose a language to use for general purpose
programming are common, and that I find them agreeable.  Micro-benchmarks
are *very* useful for determining problem areas in execution performance,
if you happen to be on the core implementation maitenance team, but
that's not what I was talking about when I said that as an overall
performance measure for purposes of choosing a language they aren't so
useful.

In other words, the problem is that you're assuming that I'm assuming
something that I am not, in fact, assuming.  Put otherwise, I basically
said that for the purpose to which most people seem tempted to put those
micro-benchmarks when they drag them out while trolling, they're useless
in any practical sense -- and you generalized from there to assume that I
was saying they're never useful for anything.


> 
> 2. That competition doesn't matter. Again, I don't agree. If it doesn't 
> matter, why do we do it? We long ago stopped racing horses because it 
> "improved the breed" and continue to race horses only because it is a 
> source of entertainment, and of tax revenue in some jurisdictions. But 
> we race computer chips and languages and algorithms because it *does* 
> improve the breed.

Again, you're putting words in my mouth.  I didn't say competition
doesn't matter.  What I suggested is that "my interpreter is faster than
yours" competition based on micro-benchmarks is pointless for languages
in the same general neighborhood of execution performance, and that
usually other concerns are *far* more important within a given
neighborhood of performance than execution speed.  If you want faster
execution speed, you should be in a completely different performance
neighborhood, not quibbling over whether Ruby or Python is faster for an
arbitrary set of micro-benchmarks, in other words.


> 
> To sum up, as a practicing performance engineer, I can say that there 
> are ways to go from micro-benchmark results to "good enough but not 
> perfect" predictions of real-world performance. That's why we do it and 
> why it matters in a nutshell.

Yes, we can -- but the margin for error is so great that, again, petty
microsecond disputes between Perl, Python, and Ruby are just that: petty.
I say this despite the fact that, overall, Perl kicks the tuckus of both
these other close competitors -- and Perl is one of my favorite
languages.  I would have said roughly the same thing even before I
started using Ruby.


> >
> >I see the same things being said by Perlists and Rubyists: that
> >programmer productivity, flexibility, "real-world" performance,
> >maintainability, and other factors are more important.  In either camp,
> >people then tend to provide evidence that their chosen language is the
> >best for these factors in some way -- at least for the purposes of the
> >person talking.  As someone who likes and uses both Perl and Ruby, I find
> >that both "sides" are right, too.  They're just right for differet use
> >cases.
> 
> Well ... in the specific case of Perl vs. Ruby, I'm not by any means an 
> expert on the use cases that differentiate between the two. Ruby is by 
> design a *successor* to Perl, IIRC. So I would expect Ruby to provide a 
> better "user experience" than Perl for a lot of use cases.

I don't know if it was intended that way, but I certainly don't view it
that way.  They are sufficiently different that they are simply
*different* languages, not *successive* languages, in practice.  Granted,
I'd definitely be willing to use Ruby as Perl's successor within the
specific niche of "object oriented programming", since Perl's OO syntax
is so onerous to deal with (by comparison), but OOP is not the be-all and
end-all of programming (the last fifteen years of Java industry
domination notwithstanding).


> 
> But I don't see how it's possible that language/implementation A can be 
> significantly faster than language/implementation B on a varied set of 
> micro-benchmarks and yet language/implementation B can be significantly 
> faster than language/implementation A on more than, say, *one*, 
> real-world application. Computers and mathematics just don't work that way.

Who said anything about "significantly faster"?  I'm just talking about
the fact that Perl consistently kicking Ruby's tail in performance
benchmarks doesn't mean a whole lot when choosing languages for projects,
generally speaking, since if performance is enough of a concern to
override the other factors involved you should be using something like C,
Objective Caml, or at least compiled Lisp instead.  In other words, if
you're trying to decide between Perl and Ruby based on performance
micro-benchmarks, you might as well be choosing between two paint shops
for your car's new coat of paint based on which one has a better record
for using paint that provides a finish with a lower drag coefficient.
Obviously, it's such a miniscule difference in the grand scheme of things
that you're better of deciding based on whether you like the color,
whether a given shop's paint jobs tend to last longer, how much it'll
cost you at one shop vs. the other, and so on.

-- 
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
They always say that when life gives you lemons you should make lemonade. 
I always wonder -- isn't the lemonade going to suck if life doesn't give
you any sugar?