On Sun, Oct 07, 2007 at 01:23:15PM +0900, M. Edward (Ed) Borasky wrote:
> Chad Perrin wrote:
> >On Sun, Oct 07, 2007 at 09:58:02AM +0900, M. Edward (Ed) Borasky wrote:
> >>M. Edward (Ed) Borasky wrote:
> >>
> >>OK ... here's the analysis. The attached PDF is what's known as a box 
> >>and whisker plot, usually shortened to "boxplot". The raw numbers that 
> >>went into this are from the Alioth shootout page, and what I *haven't* 
> >>done is checked out which versions of Perl, Python, YARV, Ruby, jRuby 
> >>and PHP these tests used. They could be years old or they could have 
> >>been run yesterday morning. :) I discarded all the tests for which there 
> >>were any missing values.
> >
> >Nice.  I wasn't expecting anything as effective for a quick, by-the-pants
> >analysis as a box plot.  What did you use to develop the graphic?  I'm
> >curious. . . .
> 
> R ... and no, I haven't ported the benchmarks to R ... and yes, I should. :)
> 
> I also suppose I should post the R code to RubyForge. :)

Then maybe we can turn a translation of that R code into Ruby into a Ruby
Quiz with some kind of additional requirements . . .

. . . or not.


> 
> >Correct me if I'm wrong, but . . . won't doing this as a gcc-comparative
> >ratio bias slower languages toward an exaggerated upper bound on the box?
> >
> >I admit I haven't done this sort of thing in a while, so my sorta
> >heuristic analysis of what I see may be prone to minor glitches.
> 
> Well, actually, yes, the gcc "box" is a meaningless point. Also, the 
> *log* of the ratio is better behaved than the ratio itself. It's more 
> like a Gaussian, which is why the whole geometric mean thing works.

Good to know I wasn't getting high on the smell of dishwasher detergent
when I questioned the choice of a ratio to gcc performance.


> 
> >Both yarv and ruby appear to be prone to low-end statistical outliers, or
> >perhaps to a "long tail" on the slow side of things.
> 
> Yeah ... again, it looks better with the log of the ratios.
> 
> >Actually, the ruby and php medians (using lower-case to denote
> >implementations) are almost identical, and perl's isn't far off,
> >according to this.  It appears that only in instances approaching
> >worst-case does ruby really begin to suffer.  Also, if I'm not entirely
> >missing the implications of a gcc-ratio comparison, I'm not entirely sure
> >that we can trust the huge visual differences in the height of the boxes
> >to indicate significant performance problem areas.
> 
> OK ... you've convinced me ... I actually computed the logs, so I might 
> as well make the boxplots of them too. :)

Thanks!

These results look a lot more like what I expected, especially with the
high-performance tail weighting for main Perl and PHP implementations.
I'm still pretty impressed with Python's median performance here, as
contrasted with what I expected -- unless those are bytecode performance
benchmarks, in which case it's not quite as surprising.  Also, since the
kind of work I tend to do doesn't really lend itself to persistent
bytecode compilation, I'd be less inclined to consider those benchmarks
(assuming I thought performance benchmarking in this class of languages
was likely to benefit me much in language choice at all).

Also . . . the log of ratio box plots don't make JRuby look like such a
dog in comparison with the rest, which is nice to see, even though it
still lags overall here.

By the way, reading this sentence of yours was the point at which I
thought "Y'know, I like disagreeing with this guy more than I like
agreeing with most other people I encounter in the Internet."  A
disagreement with you seems to be going quite well -- polite,
informative, and educational.  I'm learning while I'm stubbornly
disputing you, in other words.  Thanks.


> 
> >I'm also not sure the jruby implementation's numbers here are
> >representative of JRuby's strengths.  Does JRuby benefit from the
> >benefits of Java's optimizing VM for long-running processes?  I know it's
> >probably suffering, in micro-benchmarks, from increased start-up times as
> >it loads the VM.
> 
> I think it's more a case of this being an old jRuby -- Charlie tunes it 
> more or less daily and I've seen some benchmarks where it beats MRI.

I'm curious about both newer numbers and the effects of VM startup and VM
optimization on performance here -- though not quite curious enough to
bother installing it myself.


> 
> >I'm surprised to see python's median so low.  Are these benchmarks heavy
> >on bytecode-optimized Python, or do they tend to use standard interpreted
> >Python?
> 
> I'll have to check that ... I assumed it was bytecode.
> 
> Here's the boxplot of the log (base 10) ratios. *Very* interesting -- 
> Perl has a heavier tail on the *fast* side than it does on the slow side.

I've known for a long time that Perl does some impressive things with
execution performance.  The JIT parse tree compilation it does is a big
piece of that -- as is the fine-tuning the JIT compilation process has
gotten over the years.

-- 
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
W. Somerset Maugham: "The ability to quote is a serviceable substitute for
wit."