On Sep 4, 2006, at 8:06 PM, Rick DeNatale wrote:

> So, I've corrected the benchmark.  I've also added another method
> using String#split as Eric suggested.
>
> Here's the code being benchmarked:
> rick@frodo:~/rubyscripts$ cat stringsplit.rb

Putting the code here involves an extra method call.  While they are  
all wrapped in a method call equally, doing less is always better.

> class String
> [methods]
> end
>
> Note that I made tested three different "pre-compiled" regex's one
> with a Regex.new('.'), one with Regex.new(/./), and one a constant
> with the literal regex /./.

You don't need three different benchmarks for this.  Its easy to  
determine that these are the same with irb.

> Now here's the benchmark:
> rick@frodo:~/rubyscripts$ cat benchstringsplit.rb
> require 'benchmark'
> include Benchmark
> load 'stringsplit.rb'
>
> iters = 100

100 iterations is never enough.  GC behavior, other processes waking  
up, etc. will all cause fluctuations in the benchmark.

Use 100_000 or 1_000_000.

> [benchmark code]

Here's a more-meaningful version of yours:

$ cat stringsbench.rb
require 'benchmark'

N = 100_000
RE = /./
str = "abcdefghijklmnopqrstuvwxyz" * 5

Benchmark.bmbm do | x |
   x.report 'empty'    do N.times do end end
   x.report 'str'      do N.times do str end end
   x.report 'unpack'   do N.times do str.unpack('a' * str.length) end  
end
   x.report 'scan RE'  do N.times do str.scan RE end end
   x.report 'scan /./' do N.times do str.scan /./ end end
   x.report 'split //' do N.times do str.split // end end
end

$ ruby stringsbench.rb
Rehearsal --------------------------------------------
empty      0.030000   0.000000   0.030000 (  0.088456)
str        0.050000   0.000000   0.050000 (  0.064515)
unpack    14.370000   0.210000  14.580000 ( 20.264274)
scan RE   24.240000   0.380000  24.620000 ( 36.296017)
scan /./  24.100000   0.310000  24.410000 ( 32.661832)
split //  29.660000   0.360000  30.020000 ( 40.596177)
---------------------------------- total: 93.710000sec

                user     system      total        real
empty      0.030000   0.000000   0.030000 (  0.045199)
str        0.050000   0.010000   0.060000 (  0.065739)
unpack    14.500000   0.190000  14.690000 ( 19.813677)
scan RE   23.850000   0.350000  24.200000 ( 33.028376)
scan /./  23.910000   0.320000  24.230000 ( 35.219831)
split //  29.720000   0.470000  30.190000 ( 42.712222)

with 'abc...xyz' * 25:

Rehearsal --------------------------------------------
empty      0.050000   0.000000   0.050000 (  0.217969)
str        0.050000   0.000000   0.050000 (  0.087123)
unpack    65.370000   0.710000  66.080000 ( 80.341765)
scan RE  107.840000   1.030000 108.870000 (129.723449)
scan /./ 110.320000   1.250000 111.570000 (142.540348)
split // 135.220000   1.330000 136.550000 (164.418453)
--------------------------------- total: 423.170000sec

                user     system      total        real
empty      0.030000   0.000000   0.030000 (  0.040596)
str        0.050000   0.000000   0.050000 (  0.066506)
unpack    62.130000   0.590000  62.720000 ( 70.183396)
scan RE  107.730000   0.940000 108.670000 (125.308283)
scan /./ 107.550000   0.950000 108.500000 (127.365719)
split // 133.700000   1.020000 134.720000 (150.798611)

> So, at least from this benchmark it doesn't seem that in-line literal
> regular expressions are faster than pre-compiled ones.

inline regular expressions are no less "pre-compiled" than regular  
expressions in a variable or constant.

$ ruby -e '2.times do puts /./.object_id end'
938970
938970

('=~ /./' is faster than '=~ var' is faster than 'match anything',  
but for other reasons)

If you want to test inline vs "pre-compiled" regular expressions you  
need to throw away all the parts that are the same and focus on what  
is different.  In your benchmark it was what was on the right hand  
side of String#scan.  Since the string was the same and the scan was  
the same, just throw those away.

You end up with a benchmark like this:

$ cat vs.rb
require 'benchmark'

N = 100_000_000

RE = /./
re = /./
$re = /./
@re = /./
@@re = /./

Benchmark.bmbm do |bm|
   bm.report 'empty'    do N.times do end end
   bm.report 'lit'      do N.times do /./ end end
   bm.report 'local'    do N.times do re end end
   bm.report 'global'   do N.times do $re end end
   bm.report 'instance' do N.times do @re end end
   bm.report 'class'    do N.times do @@re end end
   bm.report 'constant' do N.times do RE end end
end

$ ruby vs.rb
Rehearsal --------------------------------------------
empty     27.930000   0.120000  28.050000 ( 31.550709)
lit       48.240000   0.320000  48.560000 ( 57.422011)
local     48.820000   0.290000  49.110000 ( 60.650110)
global    49.920000   0.420000  50.340000 ( 62.156780)
instance  55.740000   0.180000  55.920000 ( 61.027820)
class     57.800000   0.190000  57.990000 ( 63.733099)
constant  59.240000   0.330000  59.570000 (119.118487)
--------------------------------- total: 349.540000sec

                user     system      total        real
empty     27.820000   0.160000  27.980000 ( 58.584698)
lit       48.170000   0.170000  48.340000 ( 59.557450)
local     48.720000   0.170000  48.890000 ( 53.775692)
global    49.830000   0.180000  50.010000 ( 55.743586)
instance  55.880000   0.370000  56.250000 ( 72.011443)
class     57.910000   0.390000  58.300000 ( 67.037158)
constant  59.230000   0.410000  59.640000 ( 77.993887)

So literal regular expressions are "faster" by about 56% (after  
discounting loop overhead) (when you're performing 100 million  
retrievals only) (actual speedup in real code will probably be  
swallowed elsewhere or completely irrelevant).

This follows from reading eval.c.  Constant lookup involves a bunch  
of C function calls, but literal lookup just returns an Object stored  
in the parse tree.

> In fact they never came in first, although PCR3 which was a  
> constant set to a
> literal regex did win 4 times.

In fact, when you adjust the stringsbench.rb you'll see /./ winning  
over RE.  Set N to 10_000_000 (or more) and str to "abcde" (or just N  
high enough).

> Is this significant? Who knows, and with a new RegExp engine coming
> the numbers will be different in future.

The difference was probably due to processes waking up, garbage  
collection and similar unevenly distributed events.  With more  
iterations and more focused benchmarks you'll get better results.

But ultimately, these types of microbenchmarks are not very useful.   
Yes, you will get a speedup using /./ over RE, but will your program  
really run long enough where it matters to make a difference?  I bet  
not.

> So as usual the right approach is test, profile to find out what needs
> improvement, and benchmark.

Be careful with your benchmarks, they are most useful when they are  
as small and simple as possible.  Be sure to throw away all the  
irrelevant parts.

Be careful with your benchmarks, they are most useless when they are  
as small and simple as possible.  Be sure to understand how little  
speedup you'll get.

-- 
Eric Hodel - drbrain / segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com