On Mon, 23 Oct 2006, snacktime wrote:

> I've tested out a couple of ways of storing a queue structure and
> wondered if others had some better ideas.  I tried a berkeleydb based
> queue using it's native queue structure, which gives the best
> performance so far but it's for unix only.  I also have a file based
> queue where every message is just stored in a sequentially numbered
> file.  Almost as fast as berkeleydb and works anywhere.  Just for
> giggles I tried sqlite, and then immediately deleted it.  6-8 tps with
> sqlite compared to around 1000 with berkeleydb/flat files.
>
> Another alternative I was thinking of is a single file storage of some
> type, maybe fixed length or even variable length records.  Something
> that's portable.  But I'm thinking that could get complicated fairly
> quickly.
>
> Any other ideas?

why not:

   harp:~ > cat a.rb
   require 'dbm'

   n = 10000
   db = DBM.open 'db'
   at_exit{ db.close }

   kvs = Array.new(n).map{ [ k = rand(n).to_s, v = rand(n).to_s ] }

   a = Time.now.to_f;  kvs.each{|k,v| db[k] = v};  b = Time.now.to_f
   elapsed = b - a

   puts("elapsed : #{ elapsed }")
   puts("tps : #{ n / elapsed }")



   harp:~ > ruby a.rb
   elapsed : 0.220124006271362
   tps : 45428.9387576941


that's damn fast: it runs at nearly the exact same speed on my windows box too.
btw, dbm will in fact be backed by bdb on any newish *nix dist, and ndbm for
the windows one-click installer - either way though, you get it in the standard
lib and, even if you decide to marshal objects it's still extremely fast, easy
to use, and disk based.

it should be very robust, esp in the case of a bdb backed impl since bdb is acid.

regards.

-a
-- 
my religion is very simple.  my religion is kindness. -- the dalai lama