On Sun, May 27, 2007 at 12:45:12AM +0900, Hans Fugal wrote:
> >If you read your input in 4096 byte blocks, reading a new block when your
> >buffer is less than 4096 bytes full, then you'll have somewhere between 4K
> >and 8K of lookahead. You'd do that for efficiency anyway, I'd hope.
> 
> When was the last time you typed 4k faster than the computer could 
> process it? The biggest reason for stream parsing is when there's a 
> human on the other end.

OK. So it sounds like the problem is that you *need* partial matching,
because you need to handle parse-as-you-type, rather than parse an existing
file presented as a stream.

In which case the conclusions seem clear:

(1) Find an existing regular expression engine or lexical analyser engine
which does what you need. There are several which have been ported to Ruby
which you could try.

or:

(2) Write your own Regular Expression engine.

Sorry for stating the obvious. But I can't really see what else you want
from this thread. Commiseration that Ruby's built-in regexp engine isn't
suitable for your requirements? :-)

Regards,

Brian.