unknown wrote:
> 
> Do you have situations where you end up with a lot of threads?  Do you
> then have a lot of threads all using the same Mutex?  

Up to 15 threads work the queue. One threads add work items, the others 
pull items out of the queue. All threads access the queue via this 
object. Here is the latest code (the work queue is initialized once and 
is accessible to all threads):

class StationWorkQueue


  def initialize
    @mutex = Mutex.new
    @stations = []
  end

  def add_station(station)
    @mutex.synchronize do
    @stations.clear if @stations.size == 0
    @stations.unshift(station)
    end
        station = nil
    return nil
  end

  def get_station
    return @stations.pop
  end

  def size
    @mutex.synchronize do
        return @stations.size
    end
  end
end

By the way, my worker threads don't die. Is there any kind of memory 
issue with that? They run in a loop. They keep calling 
StationWorkQueue.get_station, perform work,  then sleep for a short 
while so the CPU doesn't spike, and then go looking for more work. They 
drain the queue pretty fast too.

unshift/pop doesn't seem to stop the memory consumption, but it's been 
slowed down a great deal. It's only gained a few MB over the last 30 
minutes. I'll let this run for a few hours and see what happens. I'll 
also try fastthread.

I may also try replacing my simple work queue with reliable-msg. That 
was my original plan, so I could scale this across multiple processes 
and/or machines but I just wanted the satisfaction of seeing something 
done fast so I took at stab at it this way.

I also created a second daemon to load the YAML files, look at the data 
and decide which database it should get plopped into. That one leaks 
memory like crazy, and it's simpler than the first daemon. So ya....I'll 
take that other suggestion and try ODBC over ADO.

Fun fun!


-- 
Posted via http://www.ruby-forum.com/.