On Apr 2, 2008, at 3:08 AM, Michael Linfield wrote:

> Recently I've attempted to push a huge csv into arrays via code that
> looks along the lines of this:
>
> csvFile = FasterCSV.read('data.csv', :headers => true)
>
> array = []
>
> csvFile.each do |row|
> array << row['column_name']
> end

That code is pretty inefficient, since it reads the entire file into  
memory only to walk over it row by row.  Let's just read it row by  
row, instead.

   column = [ ]
   FCSV.foreach('data.csv', :headers => true) do |row|
     column << row['column_name']
   end

> The problem arises when the csv file is someodd 2 million lines or  
> more.
> Normally I would comment about how long it took but I decided to  
> call it
> quits after 9 hours of waiting lol.

FasterCSV could be choking on the CSV data, if it's not valid.  It  
sometimes has to read to the end of the document to know that, which  
could take a long while with that much data.

James Edward Gray II