Ralph "PJPizza" Siegler wrote:
> One observation I would make would be that you set up
> benchmark/test/simulation that was very relevant to your problem domain,
> you didn't use some industry standard MIPS or TPH or such. That is the
> real problem with the type of benchmarks such as spawned the debate
> here, such things are interesting and I like to look at them, but that's
> as far as their usefulness go.

I said I had other examples :-).

It's been a long time since I was involved in one, but I'm reasonably 
confident that we use "standard" benchmarks for large procurements. When 
you spend US Government money, you have to jump through a lot of hoops 
to ensure a level competitive playing field. A protest from a losing 
bidder can tie you up for a long time, so you try to avoid that. Using 
your own benchmarks for procurement qualification invites protest.

Nobody wins just because their TPC-A or whatever is highest. A Request 
for Proposal may give a particular performance threshold, and the 
proposing vendors use that to decide which of their products to propose. 
They don't want to propose anything more expensive than they have to, 
because they're in a cost competition. It's a rough and imperfect but 
vendor-neutral way to talk about classes of performance. The real key is 
that, if the vendors buy into it, they can't protest on that point.

Obviously, if one vendor claims dramatically better performance than 
another in the same price class, that might be worth looking to. For the 
most part, however, the benchmarks just establish who's in the game, and 
most of the competition is on cost.

Steve