On 6/15/05, Ralph PJPizza Siegler <pjpizza / rsiegler.org> wrote:
> On Wed, Jun 15, 2005 at 01:55:58PM +0900, Steven Jenkins wrote:
>> In this essay I'm going to attempt, one final time, to
>> demonstrate that it is possible to have a useful benchmark. I'm
>> going to do so by telling about a real benchmark that we used to
>> solve a real problem in the Deep Space Network. But first, some
>> clarifications:
> Steve, that was a wonderful and fascinating account, thanks very
> much.

Yes, it was. And, given the broadness of the definition that Steven
used, I agree: it was done with benchmarks. However, I also agree
with you (Ralph) that this is what I would consider a performance
simulation -- a prototype, if you will -- and not a benchmark *as
such*. At least, not in *common* parlance, although the proper
definition does include this.

> One observation I would make would be that you set up
> benchmark/test/simulation that was very relevant to your problem
> domain, you didn't use some industry standard MIPS or TPH or such.
> That is the real problem with the type of benchmarks such as
> spawned the debate here, such things are interesting and I like to
> look at them, but that's as far as their usefulness go.

Right. I *personally* wouldn't consider what Steven has described in
his fascinating story as a benchmark. He may have *benchmarked* some
performance with his simulation, and used that as a baseline for
further performance tests, but the marketers have taken over the
word by and large. The alioth "benchmarks" don't do what Steven did;
things like MIPS, TPS, SpecMARK, FLOPS, and other benchmark values
are pure marketing speak -- just like the alioth shootout.

Thank you, Steven.

-austin
-- 
Austin Ziegler * halostatue / gmail.com
               * Alternate: austin / halostatue.ca