------art_39600_17936071.1147495147309
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Might be fun to integrate it with racc.

On 5/13/06, Meador Inge <meadori / gmail.com> wrote:
>
> > Are you going to give this a try?
> I think I just may. I will be sure to post to ruby-talk if I come up with
> anything.
>
> On 5/12/06, Francis Cianfrocca <garbagecat10 / gmail.com> wrote:
> >
> > Well, yes, that's what I meant by "not much more." In my experience
> > (writing
> > scanners for programming languages) the start states and greedy matching
> > were the most important features beyond the basic NFA. Unshifting input
> > seems like more of a job for the parser than the scanner, but different
> > strokes, I guess. Are you going to give this a try?
> >
> > On 5/13/06, Meador Inge <meadori / gmail.com> wrote:
> > >
> > > > If you look closely at what traditional tools like lex and flex do,
> > you
> > > > could almost take the input to lex and execute it directly in Ruby!
> > > They do look very similar.
> > >
> > > > It's not much more than a case statement from regular expressions.
> > > It is a bit more that a case statement on regular expressions:). You
> > have
> > > to
> > > do a bit of work to manage the input buffer and ensure that the
> longest
> > > match is made (e.g. greedy) for each token. Also, if you want some of
> > the
> > > other traditional features of lex and flex you have to add features
> such
> > > as
> > > start states, putting characters back into the input stream, reject
> > > matches,
> > > etc...
> > >
> > > > I'm not sure what you mean by "lightweight"
> > > I more or less meant simple and easy to use (maybe I should have just
> > said
> > > so:)) without the full range of features of a typical lex\flex
> knockoff.
> > >
> > >
> > > On 5/12/06, Francis Cianfrocca <garbagecat10 / gmail.com> wrote:
> > > >
> > > > If you look closely at what traditional tools like lex and flex do,
> > you
> > > > could almost take the input to lex and execute it directly in Ruby!
> > It's
> > > > not
> > > > much more than a case statement from regular expressions. I'm not
> sure
> > > > what
> > > > you mean by "lightweight" but if you mean "fast": a lot of people
> who
> > > > wrote
> > > > language compilers back in the day (I'm one of them) would often
> write
> > > > scanners in hand-tuned C rather than using lex.
> > > >
> > > > On 5/12/06, Meador Inge <meadori / gmail.com> wrote:
> > > > >
> > > > > Is there lightweight lexer for Ruby? I have seen a few lexer
> > > > > generators out there, but I want something simple that can be used
> > > > > without generating the lexer first. I really just need the basic
> > > > > functionaliy of matching a pattern and associating an action with
> > that
> > > > > pattern. For example, I was thinking something like:
> > > > >     class MyLexer < Lexer
> > > > >        rule /a/ do
> > > > >           # action for pattern a
> > > > >        end
> > > > >        rule /b/ do
> > > > >           # action for pattern b
> > > > >        end
> > > > >        rule /c/ do
> > > > >           # action for pattern c
> > > > >        end
> > > > >     end
> > > > >
> > > > >     l = MyLexer.new(input_stream)
> > > > >     l.each_token do |token|
> > > > >        # do something with token
> > > > >     end
> > > > >
> > > > > I have something similar to the above that I have created.
> However,
> > I
> > > > > wanted to check and see what is out there first before I invested
> > too
> > > > > much time in it. If no one knows of anything else and\or is
> > interested
> > > > > in what I have described let me know and I will clean it up and
> > > > > publish it.
> > > > >
> > > > > --Meador
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------art_39600_17936071.1147495147309--