vikkous wrote:

>>Hm, why is that? Could it not use the rules it uses for parsing
>>for one-token-at-a-time-ahead lexing?
> 
> I just can't see this. The lexer rules' input is the source file, but
> the parser's is the parse stack -- which comes from the lexer's output
> ultimately.... this can be a very powerful way to compose pattern
> matchers, but in the end different rule sets are used with 2 different
> inputs.
> 
> The lexer and parser can run interleaved, and the lexer can get
> information from the parser to help interpret things (this is sometimes
> called "cheating", but it isn't; it's often the easiest way). But
> there's still the two rule sets. I don't know if it's possible to have
> 1 rule set do both at once, but the idea is intruiging.

Hm, this might be related to me thinking pretty much in Regexps as that 
has turned out to be quite simple. Is it not possible to apply your 
extended expressions to Strings? Perhaps by .scan(/./)?

>>I'm not sure whether not having lexing and parsing more
>>unified has benefits or downsides with your approach. I
>>guess I will just have to write a Joy interpreter using all this.
>>Do you think that that can already be done or is there
>>features missing that would make it wise to delay this further?
> 
> I took a little look a joy. Hoo-boy. I'm guessing this language is
> pretty easy to parse. I would say reg is not ready for anything
> significant until it has backreferences and substitutions. At that
> point, it's got match-and-replace, and retrieval of arbitrary match
> subexpressions. If you think you can live without those, I'd say go for
> it. There are some problems with the backtracking engine, but so far as
> I can see, only a whole lot of ambiguity causes the problems, so it's
> _probably_  ok for most things.

Yup, it ought to be relatively simple to parse, though I still don't 
like lexing it as you don't want to handle spaces specially in Strings 
and so on.

I'm not even sure if I will need non-trivial backtracking or 
substitutions which is probably a sign I will need them.

>>Now I actually do wonder if using CRLF instead of LF does
>>anything special to newline-delimited literals on any
>>platforms.
> 
> Sure enough, I translated to unix format and the problems disappeared.

Was this the only problem? I think that my usage of here-docs might turn 
out to be quite exotic as well.