On Feb 23, 2010, at 10:00 AM, J=F6rg W Mittag wrote:

> Chuck Remes wrote:
>> On Feb 23, 2010, at 7:50 AM, J=F6rg W Mittag wrote:
>>> Alexander Jesner wrote:
>>>> On 02/22/2010 20:30, Benedikt M=FCller wrote:
>>>>> Ruby is not the fastest interpreted language out there.
>>>> If you have not already done so, switch to Ruby 1.9.
>>> I have seen this claim that Ruby 1.9 is somehow faster than Ruby 1.8
>>> repeated over and over again, but I have *never* seen any credible
>>> evidence for that, neither in my own benchmarks nor in Antonio
>>> Cangiano's (or any other, for that matter). Does anyone have any
>>> evidence that this is actually the case? I would be very interested =
in
>>> that.
>> Are you looking at the same benchmarks and charts that I am?
>>=20
>> http://antoniocangiano.com/category/ruby-benchmark-suite/
>=20
> Yes, I am.

Oh, boy, here we go. By the tone of your response, I'm not sure if =
you're a jerk or I'm in a bad mood. I'll assume you are *not* and that I =
*am*. But just in case you are this guy...

http://xkcd.com/386/


>> This clearly shows that 1.9.1 is consistently and measurably faster
>> than 1.8 in nearly every benchmark.
>=20
> No, it doesn't. It shows that 1.9.1 running on one VM is faster than
> 1.8 on a *totally different* VM. That's shoddy benchmarking. If you
> want to benchmark 1.8 vs. 1.9, you benchmark 1.8 vs. 1.9 and not 1.8
> and some other things vs. 1.9 and some totally unrelated, completely
> different other things.

I don't understand your point here. How can I benchmark 1.9.1 against =
1.8.7 without the VMs being different? The whole point of the move to =
1.9.1 was YARV and its new interpreter. The VMs are different and unless =
someone undertakes the effort to move the 1.8 syntax on top of YARV, =
that won't change. You mentioned this in two separate replies to this =
thread so I know it is an important point for you. You *cannot* bench =
1.8 versus 1.9 without YARV entering the equation.

I suppose you are saying that there is no valid comparison between 1.8.x =
and 1.9.x because of this difference? If not, explain yourself better.

If you *do* believe that there is no valid comparison because the VMs =
are different, then please explain how 1.8.x and 1.9.x could be =
rigorously compared.

I don't know about you, but I run my ruby code on MRI, YARV, Java6 and =
LLVM (Rubinius and MacRuby). The VM doesn't matter to me as long as the =
results are the same and each platform gives me the performance I =
require.=20

I don't know how you run your code without a VM; sounds neat.

> This is statistics 101, and is usually called "controlling your
> variables", although the great Zed Shaw in his inimitable style said
> it much better: "if you want to measure shit, don't measure other
> shit."
>=20
> You claim that the performance increase is due to Ruby 1.9. But how do
> you know that? How do you know it is not due to YARV? Or different
> compiler options? Or different ./configure options? After all, it
> wouldn't be the first time that Antonio had to withdraw or correct his
> benchmark results.

You're looking for a level of rigor that just doesn't exist in software =
benchmarking and particularly for ruby open source. I ran these =
benchmarks on my system. I can give you the basic details but my machine =
likely performs differently from the *exact* same machine with the same =
software load on it. There are dozens of Linux distros all with =
differing patch levels; same goes for BSD, Windows and OSX. Maybe my =
machine runs a little hotter (therefore slower) than an exact clone; I'm =
not about to get a temperature controlled room to bench this language. =
All of these things are likely to effect the bench results.

I say, "So what." Unless there is a bug in a major system component =
(e.g. libc, pthreads, etc), the compiler toolchain, or a hardware =
defect, the results across comparable systems are pretty close. That is =
good enough.

Results from lots of independent sources show that 1.9 is faster than =
1.8. Why? It's worthy of investigation by someone other than me. :)

>> My own experience proves this out too.
>=20
> Then show your results! As I wrote before: I would love to see them!

I emailed my results to you using Cogniano's benchmark suite available =
from github. I used the source available via the "rvm" utility and =
benched both 1.8.7-p248 and 1.9.1-p376 built on my system (OSX Intel, =
10.6.2, dual quad-core, blah blah blah).

Ruby 1.9.1 was faster in 60 (of 100+) tests, the same in 2 or 3, and =
slower in about 20. These are against the "micro" benchmarks in the test =
suite. In my world, the usual scripts I run are all noticeably faster =
with 1.9.1 versus any 1.8.x MRI branch.

cr