On Sat, 23 Sep 2006, ben / somethingmodern.com wrote:

> Does anyone have experience with using Ruby for analysis (*lots* of
> maths), on a machine with a ridiculous amount of RAM? For example, a
> hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.

Believe it or not, make or Rake or something like that is your friend.
(I tend to roll a few lines of Ruby to do the heart of it)

Break your computation into a pipeline of processes and store
intermediate results on disk.

Since Life's a Bitch, your program will have bugs / crash / wrong data /
....

So fix the appropriate input, the Makefile or whatever knows the
dependency net and recomputes only the steps needed. (You did say Big
didn't you? That in my experience means lots and lots of wall clock time
for each run. This way decreases your run,debug,fix cycle time hugely)

Since you have multiple processes, the problems you mention fade away.

If you have multiple CPU's or machines, distributing the load becomes
easy.




John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : john.carter / tait.co.nz
New Zealand

"We have more to fear from
  The Bungling of the Incompetent
  Than from the Machinations of the Wicked." (source unknown)