On Wed, Sep 24, 2008 at 4:18 PM, Michael Furr <furr / cs.umd.edu> wrote:

> Ruby's syntax can be disambiguated using just static information, but its
> rather complex.  The lexer/parser uses a state machine to determine how to
> lex the next character.  Here is my understanding of how this situation
> (lexing '/') is resolved.  There are two primary things that influence the
> state: whitespace and the location of the token in an expression (i.e., the
> beginning/middle/end).

Michael, thank you, you've really answered the question in the way I needed.

You've confirmed that whitespace is significant to the parser, and
explained how.

You've confirmed that the parser is in fact doing some lightweight
interpretation (or at least "assumption").

Now I know what I need to do in my tokenizer.

Thank you, everybody who responded, for your help with this.

Seth