On 30 Aug 2005, at 16:40, Townley, Andrew wrote:

> On Tue, 2005-08-30 at 19:05, Eric Hodel wrote:
> [lots of really useful feedback deleted]
>
> Thanks for the feedback, Eric.  I'd actually made some other  
> changes to
> it prior to getting your comments.  I have to say, looking at the
> difference, I can see some value in the ThreadGroup, but I don't  
> really
> get why you're so excited about it... this is for another day, I  
> think.
>
> In other news, I found the mystery multiplying threads and have a
> problem that I'm not quite sure how to solve with the tools at hand...
> Maybe someone out there can help, but maybe the answer is to just roll
> my own something-or-other again.
>
> The problem came from two seemingly benign blocks of code:
>
> $ cat fu.rb
> require 'thread'
> require 'timeout'
>
> @queue = Queue.new
>
> def read(timeout)
>   begin
>     Timeout::timeout(timeout) do
>       puts("READ:   #{@queue.length} elements; # 
> {@queue.num_waiting} threads waiting.")
>       return @queue.deq
>     end
>   rescue Timeout::Error
>     puts("TIMEOUT:  #{@queue.length} elements; # 
> {@queue.num_waiting} threads waiting.")
>   end
> end
>
> and this (/usr/lib/ruby/1.8/thread.rb):
>
>     280       @waiting.push Thread.current
>
> because of this (/usr/lib/ruby/1.8/timeout.rb):
>
>      40       y = Thread.start {
>      41         sleep sec
>      42         x.raise exception, "execution expired" if x.alive?
>      43       }
>      46     ensure
>      47       y.kill if y and y.alive?
>
> The key line is 280.  The problem seems to come from @waiting being an
> array which holds on to references to the threads created in line 40
> (which is also why I couldn't find it because I was tracing for
> Thread#new and not Thread#start doh!!).  Even though the thread  
> seems to
> definitely be killed in line 47, the array still holds a reference to
> it, so I'm guessing that like Java, this prevents garbage  
> collection for
> a while.  Therefore, when I was looking at the list, the threads  
> created
> here were still in it.

Yes.  A ThreadGroup doesn't have the GC problem, but would not be  
suitable here.  (A Thread can only belong to one ThreadGroup, and  
Queue shouldn't reorganize threads it didn't create.)

> I'm a bit worried about this block in thread.rb, though:
>
>     257       t = @waiting.shift
>     258       t.wakeup if t
>
> because I really don't like what I've observed happening in #257/8
> here.  Based on what I'm doing, the waiting array will eventually get
> huge... and I mean, HUGE.  Also, based on further experiments, adding
> items to the queue will reduce @waiting.length by n, however there  
> will
> be a lot more attempted reads than there will attempted writes to the
> queue.

More job runners than jobs?

> I'm open to suggestions, but I can't just do it without the timer
> because I need to get control back every n seconds so I can do things
> like graceful shutdown, etc.

If you need safe concurrent access like a Queue and timeouts you may  
find rinda/tuplespace.rb useful.  Somewhere around here I have a  
stream implementation for it (but its not terribly difficult to write  
from scratch).  I think it can be modified to have timeouts on pop.

(I have RDoc patches for rinda out for review.)

> Here's the full test program (not out to win any style awards with the
> calls to read, btw) :)

Give me a bit and I think I can make your test program with with a  
TupleSpace streams.

-- 
Eric Hodel - drbrain / segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E  7C11 332A 551C 796C 9F04