As I was investigating the performance of Eric Wong's Sleepy GC patches (
https://bugs.ruby-lang.org/issues/14723) I saw some regression in early
April Ruby head-of-master. I mentioned that I'd seen it, and Koichi
suggested I email Ruby Core.

The speed regression I saw was pretty small, and I re-ran the benchmarks to
make sure. A large parallel run of Discourse can have very noisy data. I've
seen consistent results in this case with around 100 batches of 10k HTTP
requests/batch, so those are what I show below.

For small regressions, what I often see is slightly lower throughput, but
higher variance - that means a few runs are slow, but most runs are the
same speed. High variance isn't always a problem, and it changes noticeably
from run to run. But high variance and low speed normally means multiple
extra-slow runs bringing the mean and median down.

In each case I'm just checking a date - I look at version.h and use the Git
SHA of the revision for that date.

Here are the results I'm seeing for several revisions of Ruby:

April 1st: slight regression, high variance
Median Throughput: 171.9 reqs/second, variance 9.3
Commit: trunk@63052 / abb19b13ee7c966c3153ec0cb09f5489efa3d2b6

April 15th: regression is gone, variance is normal
Median Throughput: 177.7 reqs/second, variance: 3.7
Commit: trunk@63153 / 136643a8764115620c0950c1493c8416b69791b6

May 1st: No result, benchmark does not complete; this Ruby was broken
Commit: trunk@63308 / 5e5b2fe69b3d5ac7cf6a8eedf4355bc8d1acc97c

May 3rd: Speed looks fine, variance is high but I think within normal
margin of error
Median Throughput: 177.2 reqs/second, variance: 5.9
Commit: trunk@63324 / 41dc02c6360cbcaad726b1d0fda5539d13caed68

May 16th: Variance looks good, I think speed is within margin of error
Median Throughput: 174.5, variance: 2.8
Commit: trunk@63439 / a930a064c015b4182ad244c19608cd996bab9347

I haven't tracked down the April 1st regression. But if anybody has a
particular commit around that time that they're worried about, I can check
it - each of these trials for a specific Ruby version takes around 3 hours.
That's not the end of the world, but it's slow enough that I'm only
checking specific versions, not every version.

(Larger changes can be checked much faster - these only take so long
because I'm running many batches to detect a small effect, around 2%-4%
performance change.)

If anybody would like me to check other patches, or other Ruby versions, or
specific SHAs, let me know. I can also give you a copy of any of this data.
I save per-request timing data. I'm happy to make it public. But it's far
too large to send to all of Ruby Core.

I'm going to try to re-run RRB more often so that, if a regression like
April 1st happens again, I'll see it earlier.
(supressed text/html)
Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>