I'm making the back end for a feed reader.  I plan to have a daemon 
which periodically checks feeds for updates.  The daemon will have one 
refill thread responsible for maintaining a list of feeds to be updated. 
It will also have n worker threads who pop feeds from the list and 
process them.  When the list is empty, the refill thread will sleep for 
some amount of time (no need to update the feeds constantly), refill the 
list of feeds, and signal to the workers to start up again.

Here's a crawler class as I have it so far.  I've removed some of the 
nonrelevant parts.

class Crawler
  def initialize(minutes_sleep)
    @minutes_sleep = minutes_sleep
    @first_run = true

    @feeds = Array.new

    @feeds_lock = Mutex.new
    @empty = ConditionVariable.new
    @filled = ConditionVariable.new
  end

  def fill
    @feeds_lock.synchronize do
      @empty.wait(@feeds_lock)
      if @first_run
        @first_run = false
      else
        sleep(60 * @minutes_sleep)
      end
      @feeds = Feed.find(:all, :conditions=>'active = 1')
      @filled.broadcast(@feeds_lock)
    end
  end

  def crawl(worker_number)
    @feeds_lock.synchronize do
      if @feeds.size == 0
        @empty.broadcast(@feeds_lock)
        @filled.wait(@feeds_lock)
      end

      feed = @feeds.pop
      # Do some processing
    end
  end
end

Then I kick off with another script that looks like this:

crawler = Crawler.new(minutes_sleep)
filler = Thread.new {crawler.fill}
workers = (1..num_workers).map do |i|
  Thread.new {crawler.crawl(i)}
end

filler.join
workers.each{|thread| thread.join}

This code results in a deadlock.  It seems like the refill thread is not 
waking up, doing its work, and signaling to the workers.  But I have no 
idea - I am stumped.  Any pointers?

I did try to use the Queue class, but it seemed a little magical and I 
couldn't quite figure out how to use it for my needs.

-- 
Posted via http://www.ruby-forum.com/.