A token is a syntax element which defines in-order patterns of text. The result of a token on compilation is a deterministic finite-state automation. Tokens can reference other tokens; however, they cannot be recursive. A token which references another token, upon compilation, the reference to the token is embedded into the definition of the original token. This ensures that the automation is not dependent upon another automation.

Put simply the following token definitions:

A := B | C;
B := 'b';
C := 'c';

Would be rewritten as:

A := 'b' | 'c';

If B or C are not referenced by any production rules then they are removed from the resulted compile since their automations will never be used.

Last edited Aug 19, 2009 at 12:37 PM by AlexanderMorou, version 2


No comments yet.