Issue #5985 has been updated by Eric Wong.


 ko1 / atdot.net wrote:
 > > +BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common
 > > $(RUN_OPTS)
 > 
 > I'm not sure BENCHRUBY works with libruby in builddir (don't use
 > installed libruby?).
 
 Good catch, I tested out-of-tree build, but not --enable-shared.
 I can force the use of STATIC_RUBY, it looks like.
 I have this working both with and without --enable-shared.
 
 http://bogomips.org/ruby.git/patch/?id=7495b0fdc7
 
 --- a/common.mk
 +++ b/common.mk
 @@ -1001,14 +1001,14 @@ COMPARE_RUBY = $(BASERUBY)
  ITEM =
  OPTS =
  
 -BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)
 +BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)
  
 -benchmark: $(PROGRAM) PHONY
 +benchmark: $(PROGRAM) PHONY $(STATIC_RUBY)
  	$(BASERUBY) $(srcdir)/benchmark/driver.rb -v \
  	            --executables="$(COMPARE_RUBY); built-ruby::$(BENCHRUBY)" \
  	            --pattern='bm_' --directory=$(srcdir)/benchmark $(OPTS)
  
 -benchmark-each: $(PROGRAM) PHONY
 +benchmark-each: $(PROGRAM) PHONY $(STATIC_RUBY)
  	$(BASERUBY) $(srcdir)/benchmark/driver.rb -v \
  	            --executables="$(COMPARE_RUBY); built-ruby::$(BENCHRUBY)" \
  	            --pattern=$(ITEM) --directory=$(srcdir)/benchmark $(OPTS)

----------------------------------------
Bug #5985: miniruby skews "make benchmark" results
https://bugs.ruby-lang.org/issues/5985#change-44839

* Author: Eric Wong
* Status: Closed
* Priority: Low
* Assignee: Koichi Sasada
* Category: build
* Target version: 2.0.0
* ruby -v: -
* Backport: 
----------------------------------------
miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3_gc
ruby ../benchmark/driver.rb -v \
	            --executables="ruby; ./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems" \
	            --pattern=vm3_gc --directory=../benchmark 
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]

-----------------------------------------------------------
vm3_gc

#! /usr/bin/ruby
5000.times do
  100.times do
    {"xxxx"=>"yyyy"}
  end
  GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	.2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	.1.857623815536499

-----------------------------------------------------------
raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)
-----------------------------------------------------------
benchmark results:
name	ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	average difference
vm3_gc	2.752	1.858	-0.895
-----------------------------------------------------------
average total difference is -0.894646406173706




-- 
http://bugs.ruby-lang.org/