Zed Shaw wrote:
> * The threads create a ton of objects which can make the GC cause
> large pauses.  I've tried a group of threads waiting on a queue of
> requests, but that's not much faster or better.  So far the fastest
> is using IO.select (see below).

Have you checked to see if your C extension is "leaking" memory by 
virtue of Ruby not correctly handling it? This happened to me recently 
with a similarly purposed C extension, so much so that I had to do it in 
pure C and simply fork/pipe in Ruby to use it. The problem was that my 
extension was using ALLOC() and friends for allocation, but Ruby didn't 
understand that it could release that memory, even after the usage of 
the process was 3GB+. I moved on, but I will eventually get back there 
to find out why that was happening...

> Yes, that's still true since Ruby and libevent don't know about the
> other.  They fight like twenty rabid cats in a pillow case.  The main
> difference is that IO.select knows about Ruby's threads, so it's
> supposed to be safe to use.

As far as I understand, at the base of it, IO::select's C handler, 
rb_f_select() calls rb_thread_select() to do the actual select'ing. It 
appears that there are more functions on top of the rb_thread_select() 
when coming at it from the co-op thread scheduling callchain, however. 
This would be in line with what you were saying.

>> 2. What are these "weird ways" you keep referring to? I got the
>> select-hogging-the-event-party thing, but what else?
>>
> Basically select hogs the party, threads just kind of stop for no
> reason, select just stops, etc.  I really which they'd just use pth
> so I could get on with my life. :-)   I've been playing with it, and
> I think I have something that might work.

Does Ruby spawn Thread objects even when not requested by the 
programmer? I seem to remember that GC was in a Thread? Is that right?

If not, can you just not spawn any and avoid these issues altogether 
(perhaps alias Thread's new method to raise an exception to make sure it 
doesn't happen?)

> You know, having tried this, I have to say you'll be fighting a
> losing battle.  Ruby's thread implementation just isn't able to work
> with external multiplexing methods.  I couldn't figure it out, so if
> you do then let me know.

I'm not at all put off by simply replacing select(2) in the Ruby core 
with something else, just so I can get what I need, 
[porta|releasa]bility be damned. I know this is not the best solution, 
but it might be the fastest. I would really like something I could 
gem-ify, though, if at all possible. I thought about trying to work this 
into YARV and just use that, but that's nigh-on-unusable at the moment 
for other reasons.

-- 
Posted via http://www.ruby-forum.com/.