Mark Probert <probertm / acm.org> writes:

> This is getting tough to track down!  
>
> The basics are that my script is multi-threaded (potentially 20-30 threads), 
> running remote data collection using telnet, for up to 300 nodes.  Some state 
> information (basic info about the remote node) is stored using hashes within 
> the class though all of the "real" data is spooled to disk for later 
> processing.
>
> Here are some facts, not necessarily all directly related to the problems I am 
> seeing:
>
>
> 1.  I am using -lots- of memory under Solaris, without any exceptions being 
> thrown (approx 200Mb per-thread reported for the full application).
> When I trim everything down to do just login with no state saving, I
> am using about 12Mb per thread.  This seems to be very high, though
> I am not really sure whether that amount of usage is or not.
> Perhaps it is in keeping with Ruby's thread model.

I once had a server program that grew continuously to the point of
memory exhaustion until I wrote my own tempfile.rb.  Somehow
tempfile's use of delegate.rb caused the script to use gobs of memory.
I never tracked that down, but at the time I remember something about
some internal Ruby data structures never being collected (symbols come
to mind).


> 2.  When I use a fork() model, the amount of memory used per-process is around 
> 20Mb.  Again, this seems to be quite heavy, however, memory usage doubles to 
> 40Mb when I capture data in the hashes!  Now, I am -not- gathering a lot of 
> data here:
>
>       ### process card data
>       if line =~ /^\s+([0-9]+)\s+(.*)$/ then
>         @cards[$1] = $2
>       end
>
> as an example.  There are at a maximum 14 entries like this.  20Mb+ seems a 
> lot of memory to use!

Very odd.  Can you isolate this expensive use of memory into a stand
alone script?


> 3.  I don't have, yet, a copy of Purify to check for leaks under Solaris, so, 
> I am using valgrind under FreeBSD.  valgrind is crashing a lot due
> to the threads and sockets, but that is another issue :-(

[...]

> 4.  Before valgrind dies, it is reporting the following (many of these errors, 
> I am just picking out the unique ones):

[...]

Purify/Valgrind is not so useful for Ruby, since Ruby's GC confuses them.



> 5.  There doesn't appear to be an easy way of tracking memory usage / 
> allocations within ruby.  Are there any plans on adding something that might 
> help?  Or is it all too system dependant?

[...]

Telling more about your ruby version, and isolating the problem down
to the smallest script that exhibits the problem are both useful.


-- 
matt