Well, I found something on sun's website about the fcntl "compatibility" 
implementation of flock.

    Locking a segment that is already locked by the calling process
    causes the old lock type to be removed and the new lock type to
    take effect.

So, I guessed that repeated lock calls by a process that already has a 
lock will briefly release the lock before reacquiring it (even for the 
same lock type), and that this would potentially allow a writer process 
to get the lock before the original reader was done with it, because 
another reader had tried to access it. Repeated lock calls happen 
because of the existence of several threads in the process (that's why 
the problem only happened with thread_count > 1 and process_count > 1).

Since I already had a reader count (to prevent closing too early with 
multiple readers), it was simple to use it to prevent multiple readers 
from trying to get the lock.

For the curious, the fixed version is at

   http://path.berkeley.edu/~vjoel/ruby/solaris-bug-fixed.rb

This was the last major bug with my software, so I will be releasing it 
soon....