I'm trying to work out ways to reduce the memory use of one of my
projects, but I don't know what methods are available to the Ruby
programmer for profiling memory use and tracking down garbage
collection problems.

The short version:

I have a project where processing a file can consume dozens of
megabytes of memory; if I process many files in a single run then
total memory usage can reach hundreds of megabytes or more than a gig.
I would expect garbage collection to kick in along the way but it
doesn't seem to be happening, memory usage grows and grows, and I
don't know where to start zeroing in on the problem.

The long version:

I've written an object-oriented templating system[1] that incorporates
a memoizing packrat parser. As each file is parsed the parser
"memoizes" the partial results for speed. In a lengthy file the size
of the memoizing cache can grow quite large (dozens of megabytes). But
I would expect the entire contents of the memoization cache to get
garbage collected when I move on to the next file; the cache itself
has definitely fallen out of scope by that time. But garbage
collection doesn't seem to be happen, as memory use grows linearly as
I batch process input files.

As this is a largish, complicated project I don't even know where to
begin to start investigating this. So really, I am looking for general
information on techniques for measuring and exploring memory use and
garbage collection in Ruby.

Thanks in advance for the advice!
Wincent

[1] http://walrus.wincent.com/