On Sep 13, 2008, at 12:45 AM, Brad Volz wrote:

> Hello,
>
> I seem to have an issue with file descriptors that aren't being  
> closed when I attempt to put some parallelization into one of my  
> scripts.

you can list them all with something like

   limit = 8192
   files = Array.new(limit){|i| IO.for_fd(i) rescue nil}.compact.map{| 
io| io.fileno}
   p files


>  I am trying to make use of the forkoff gem, but I guess that I am  
> not using it correctly.
>
> If it's useful, here is what the code looks like currently:
>
>    def measure_n ( start_time, stop_time, direction, pattern,  
> records )
>      counters = [ :flows, :packets, :octets ]
>
>      # single process
>      stats = Hash.new
>      counters.each { |c| stats[c] = 0 }
>
>      puts "starting single process execution"
>      single_start = Time.now.to_f
>      pattern.each do |p|
>        r = self.measure_1( start_time, stop_time, direction,  
> "#{direction} AS #{p.to_s}", records)
>        counters.each { |c| stats[c] += r[c] }
>      end
>      single_stop = Time.now.to_f
>      puts stats.inspect
>      puts "single exeution time: #{single_stop - single_start}"
>
>      # multiple processes
>      stats = Hash.new
>      counters.each { |c| stats[c] = 0 }
>
>      puts "starting multi process execution"
>      multi_start = Time.now.to_f
>      asn_stats = pattern.forkoff! :processes => 4 do |asn|
>        a = Netflow::Nfdump.new

this probably opens a file or pipe

>
>        a.measure_1(start_time,stop_time,direction,"#{direction} AS  
> #{asn}",records)

this too possibly

>
>      end
>      asn_stats.each do |asn|
>        counters.each { |c| stats[c] += asn[c] }
>      end
>
>      multi_stop = Time.now.to_f
>      puts stats.inspect
>      puts "multi execution time: #{multi_stop - multi_start}"
>
>      return stats
>    end
>
>
> The part that I find really odd, is that I can replace the block  
> that I am attempting to parallelize with a simple:-
>
> 	puts "#{asn}"
>
> And the script dies in the same place -- at the 255'th element in  
> the array, which corresponds well to the number of file descriptors  
> that I can use:-
>
> 	bradv:bvolz:$ ulimit -a | grep files
> 	open files                      (-n) 256
>
> Since I see this problem with a simple 'puts' does that mean that  
> the issue is not in my code, and perhaps lies elsewhere?  Or have I  
> misunderstood how to make use of the forkoff gem?  In either case,  
> how can I figure out what these open file descriptors are?
>
> Thanks,
>
> Brad
>

you are using the gem properly.  you should attempt to track down the  
open files with something like this in the forkoff block

     ios = Array.new(8192){|i| IO.for_fd(i) rescue nil}.compact
     filenos = ios.map{|io| io.fileno}
     paths = ios.map{|io| io.path rescue nil}

     STDERR.puts "child   : #{ Process.pid }"
     STDERR.puts "filenos : #{ filenos.inspect }"
     STDERR.puts "paths   : #{ paths.inspect }"
     STDIN.gets

i just releases version 0.0.4 of forkoff.  should not affect your  
issue, but might want to install anyhow (just pushed to rubyforge -  
might have to wait for it propagate or grab the gem from there manually)

cheers.




a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama