Robert Klemme wrote:
> On 09.04.2008 00:30, Michael Linfield wrote:
>> That would work, but even with marshal dumping the data set is just too 
>> large for memory to handle quickly.
> 
> Which data set - the range definitions or the output?  I thought this is
> a one off process that transforms a large input file into a large output
> file.
> 
>> I think I'm going to move the 
>> project over to PostgreSQL and see if that doesn't speed things up a 
>> considerable amount, Thanks Robert.
> 
> That's of course an option.  But I still feel kind of at a loss about
> what exactly you are doing.  Is this just a single processing step in a
> much larger application?
> 
> Cheers
> 
>   robert

The dump would be to the pre-defined hash, to hence retrieve the 
information faster.

To answer your 2nd question yes this is just a single step in a very 
large 12 step application. I'm hoping to condense it down to about 8 
steps when I finish.  This step alone involves transforming a large 
dataset into a smaller dataset.

I'm trying to extract all the numbers between ranges and push the keys 
of the hash results into a file. This file will then be opened by 
another part of the step process to be analyzed.

IE:
if the transformation involved the file of:
12345
67423
97567
45345
ect.
I would want to pull all of those numbers and get the keys for those 
hash ranges
IE:
12000..15000 => 100
60000..70000 => 250
ect.

So 12345 would fall in the range of 12000.15000 so the output file would 
get 100 added to it. Then the next step would be analyzing the results 
(IE: 100).
Hope this explains things a bit better.

Thanks,

- Mac
-- 
Posted via http://www.ruby-forum.com/.