You could probably just use memcached. It's optimized for efficient
memory allocation of hash based data stores.

On Mon, Sep 14, 2009 at 5:15 AM, Carsten Gehling <carsten / sarum.dk> wrote:
> I've created a small daemon, that serves certain data very fast to our
> remaining web-application. It does so by pre-loading a lot of data from
> our database into a special structure in memory. Boiled down, the
> pre-loading procedure looks like this:
>
> @data_hash = {}
> DBI.connect(dns, user, pass) do |dbh|
>  ¨Βτθ δβθ®εψεγυτε¨Άσονε­σρμ­ςετυςξιξΎ³°°®°°° ςοχσΆ©
>  ¨Βθιμε ςοστθ®ζετγθίθασθ
>  ¨ΒασθίλεΌγαμγυματεδ­ζςον­ςοχ­δαταΎ
>
>  ¨ΒοχξεςίςεματιοξσΫθασθίλεωόόΫέ
>  ¨ΒοχξεςίςεματιοξσΫθασθίλεωΌΌ ςο>  ¨Βξδ
>  ¨Βτθ®ζιξισ> end
>
> It gives a @data_hash with a structure like this:
>
> @data_hash = {
> key1' => [row1, row2, row3, ...]
> key2' => [row4, row5, ...]
> ..
> }
>
> I have a problem with the speed of the pre-loading though. I am loading
> ~360.000 rows into memory (about 500MB). The first 50% of the rows are
> read and placed in the hash pretty quick. But after that it slows down.
>
> I suspect that Ruby's dynamic memory allocation for the Hash is to
> blaim. Can I somehow pre-allocate 500MB memory for the hash? Or tweak
> the way Ruby allocates memory?
>
> Thanks in advance!
>
> - Carsten
> --
> Posted via http://www.ruby-forum.com/.
>
>