[prog] lex and yacc

Jenn Vesperman jenn at anthill.echidna.id.au
Fri Jan 23 19:54:56 EST 2004


On Fri, 2004-01-23 at 00:09, Sue Stones wrote:

> Yes I am using that book, and still feeling somewhat befuddled after a few
> days.  Which probably says a fair bit about my state of mind.  Concentration
> is not really happening!  Sorry I should have said that in my original post.
> the fact that the first example is anlaysing English sentences is adding to
> the confusion, because I can't see how that generalises to programming
> languages.

AHA! This, I can help with.

Let's take an example sentence (such as 'I can help with this'), and an
example fragment of a program (such as 'printf("Hello, world")').

WARNING: this is from memory, and memory that's (ack!) more than fifteen
years old. If I have details wrong, please forgive me.


Lexical analysis of the program would show this:

printf            (             "Hello, world"           )
  ^               ^                   ^                  ^
  |               |                   |                  |
function call   argument boundary  string literal    arg. bound.


And of the sentence:

I              can help       with            this
^                ^              ^              ^
|                |              |              |  
subject    (complex) verb    preposition    object


(Actually, lex may be even more basic than that. If your book says it's
at the level of 'pronoun', 'verb', 'preposition', 'pronoun', then trust
the book.)


Regardless of what is being split up, the process of splitting it up is
called lexical analysis, and is what lex does. Lex doesn't care if
you're splitting a sentence into subject, verb, object, or a statement
into function calls and arguments. It only wants to know what rules
you're using to split the text stream.

The generalisation is 'lexical analysis is how we split the text stream
up into its component pieces (tokens)'.



Once you have the text stream split up, THEN you turn around and use
yacc (or some other parser) to say 'ok, so how do I put these tokens
together into a coherent form'.




Jenn V.
-- 
    "Do you ever wonder if there's a whole section of geek culture 
        	you miss out on by being a geek?" - Dancer.
   My book 'Essential CVS': published by O'Reilly in June 2003.
jenn at anthill.echidna.id.au     http://anthill.echidna.id.au/~jenn/




More information about the Programming mailing list