On 3/6/07, Phrogz <gavin / refinery.com> wrote:
> When performing a benchmark comparison of several techniques, this
> idiom is very common for me:
>
>   N = 1_000_000
>   Benchmark.bmbm{ |x|
>     x.report( 'foo' ){
>       N.times{
>         # foo code
>       }
>     }
>     x.report( 'bar' ){
>       N.times{
>         # bar code
>       }
>     }
>     x.report( 'jim' ){
>       N.times{
>         # jim code
>       }
>     }
>   }
>
> Observation #1: It's annoying to have to type "N.times{ }" for each
> report. I also want the exact same value, and I always want some
> iterations.
>
> Consequent Desire #1: It'd be nice if I could write the above as:
>   Benchmark.bmbm( 1_000_000 ){ |x|
>     x.report( 'foo' ){
>         # foo code
>     }
>     x.report( 'bar' ){
>         # bar code
>     }
>     x.report( 'jim' ){
>         # jim code
>     }
>   }
>
> Observation #2: I'll frequently start with a 'guess' value of
> N=1_000_000, but if it's taking too long, I'll start playing with
> values of N until I get something that takes 1-5 seconds for the first
> compared value.
>
> Consequent Desire #2: It'd be cool if I could write the above as:
>   Benchmark.autobm{ |x|
>     x.report( 'foo' ){
>         # foo code
>     }
>     x.report( 'bar' ){
>         # bar code
>     }
>     x.report( 'jim' ){
>         # jim code
>     }
>   }
> ...and it would figure out an appropriate number of iterations for me.
> (One approach would be to time one iteration, figure out an
> appropriate multiple, try benchmarking that number of iterations, and
> adjust the number iterations if the result is outside some desired
> range. Rinse/repeat.)
>

Mauricio Fernandez wrote a blog post about just this and created
AdaptiveBenchmark.

http://eigenclass.org/hiki.rb?cmd=view&p=adaptative+benchmark&key=TDD

Blessings,
TwP