On Thu, May 17, 2007 at 05:50:11PM +0900, Bill Kelly wrote:
> It just seemed to me that using memory-mapped I/O, it might be
> possible to get maybe an order of magnitude faster performance
> than sqlite3 for raw key/value style fetches and stores, as long
> as lots of the relevant pages were being cached by the VMM.
> 
> Certainly if the VMM were able to cache *all* the pages representing
> the hash database file, I'd expect performance could approach an
> in-memory hash--which appears to be around two orders of magnitude
> faster than sqlite3.  Although, once the hash database file 
> significantly exceeds RAM size, one would expect performance to
> degrade to I/O speeds given totally random access patterns; however
> if just a subset of the total keys were being read repeatedly, I'd
> expect performance to stay high.  (The latter being a case I'm
> interested in.)
> 
> So I figured there might conceivably already be several existing
> implementations of such a thing.  (But had trouble finding them
> with web searches, if they exist--apart from BDB.)

Well, there's ruby-mmap. Looking at the docs, this effectively gives you a
very large string, which is not what you're looking for, but maybe you could
layer something on top of this.

If your keys are small but your objects are large, you could keep all the
keys in RAM (e.g. using something like Madeleine) and just store pointers to
the data. Then you only(*) have to worry about fragmentation and garbage
collection within your mmap'd file.

Or you could just keep keys in RAM and use separate disk files for each
object.

Regards,

Brian.

(*) :-)