[/==============================================================================
    Copyright (C) 2001-2010 Hartmut Kaiser
    Copyright (C) 2001-2010 Joel de Guzman

    Distributed under the Boost Software License, Version 1.0. (See accompanying
    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

[section:lexer_concepts Lexer Concepts]

__lex__ components fall into a couple of generalized __concepts__. The
/Lexer/ is the most fundamental concept. All __lex__ components are
models of the /Lexer/ concept. /PrimitiveLexer/, /UnaryLexer/,
and /NaryLexer/ are all refinements of the /Lexer/ concept.

The following sections provide details on these concepts.

[/////////////////////////////////////////////////////////////////////////////]
[section Lexer]

[heading Description]

The /Lexer/ is the most fundamental concept. A Lexer has a member
function, `collect`, that accepts a token definition container `Def`, and a
the name of the lexer state the token definitions of the lexer component need 
to be added to (a string). It doesn't return anything (return type is `void`). 
Each Lexer can represent a specific pattern or algorithm, or it 
can be a more complex lexer component formed as a composition of other Lexer's.
Additionally, a Lexer exposes a member `add_actions`, that accepts the token 
definition container `Def`, while returning nothing (again, the returned type
is `void`).

[variablelist Notation
    [[`l`]              [A `Lexer`.]]
    [[`L`]              [A `Lexer` type.]]
    [[`Def`]            [A token definition container type.]]
    [[`State`]          [A type used to represent lexer state names.]]
]

[heading Valid Expressions]

In the expressions below, the behavior of the lexer component, `l`, is left 
unspecified in the base `Lexer` concept. These are specified in subsequent, 
more refined concepts and by the actual models thereof.

For any Lexer the following expressions must be valid:

[table
    [[Expression]              [Semantics]                       [Return type]]
    [[`l.collect(def, state)`] [Add all token definitions provided 
                                by this Lexer instance to the lexer
                                state `state` of the token definition 
                                container `def`.]                [`void`]]
    [[`l.add_actions(def)`]    [Add all semantic actions provided
                                by this Lexer instance to the token
                                definition container `def`.]     [`void`]]
]

[heading Type Expressions]

[table
    [[Expression]                                   [Description]]
    [[`traits::is_lexer<L>::type`]                  [Metafunction that evaluates to `mpl::true_` if 
                                                     a certain type, `L` is a Lexer, `mpl::false_`  
                                                     otherwise (See __mpl_boolean_constant__).]]
]

[heading Postcondition]

Upon return from `l.collect` the following post conditions should hold:

* On return, `def` holds all token definitions defined in the Lexer, `l`. This 
  includes all Lexer's contained inside `l`.

Upon return from `l.add_actions` the following post conditions should hold:

* On return, `def` holds all semantic actions correctly associated with the 
  corresponding token definitions as defined in the Lexer, `l`. This 
  includes all semantic actions defined by the Lexer's contained inside `l`.

[heading Models]

All lexer components in __lex__ are models of the /Lexer/ concept.

[endsect] [/ Lexer Concept]

[/////////////////////////////////////////////////////////////////////////////]
[section PrimitiveLexer]

[heading Description]

/PrimitiveLexer/ is the most basic building block that the client uses
to build more complex lexer components. 

[heading Refinement of]

[:__lexer_concept__]

[heading Type Expressions]

[table
    [[Expression]                                   [Description]]
    [[`traits::is_primitive_lexer<L>::type`]        [Metafunction that evaluates to `mpl::true_` if 
                                                    a certain type, `L`, is a PrimitiveLexer, `mpl::false_`  
                                                    otherwise (See __mpl_boolean_constant__).]]
]

[heading Models] 

The following lexer components conform to this model: 

* character literals (i.e. `'x'`), `char_`,
* string literals (`"abc"`), `std::basic_string<>`, `string`

__fixme__ Add more links to /PrimitiveLexer/ models here.

[endsect] [/ PrimitiveLexer Concept]

[/////////////////////////////////////////////////////////////////////////////]
[section UnaryLexer]

[heading Description]

/UnaryLexer/ is a composite lexer component that has a single subject. The
UnaryLexer may change the behavior of its subject following the
__delegate_pattern__. 

[heading Refinement of]

[:__lexer_concept__]

[variablelist Notation
    [[`l`]              [A UnaryLexer.]]
    [[`L`]              [A UnaryLexer type.]]
]

[heading Valid Expressions]

In addition to the requirements defined in __lexer_concept__, for any
UnaryLexer the following must be met:

[table
    [[Expression]       [Semantics]                 [Return type]]
    [[`l.subject`]      [Subject lexer component.]  [__lexer_concept__]]
]

[heading Type Expressions]

[table
    [[Expression]           [Description]]
    [[`L::subject_type`]    [The subject lexer component type.]]
    [[`traits::is_unary_lexer<L>::type`]       [Metafunction that evaluates to `mpl::true_` if 
                                                a certain type, `L` is a UnaryLexer, `mpl::false_`  
                                                otherwise (See __mpl_boolean_constant__).]]
]

[heading Invariants]

For any UnaryLexer, `L`, the following invariant always holds:

* `traits::is_lexer<L::subject_type>::type` evaluates to `mpl::true_`

[heading Models]

The following lexer components conform to this model: 

* action lexer component (allowing to attach semantic actions) 

__fixme__ Add more links to models of UnaryLexer concept

[endsect] [/ UnaryLexer Concept]

[/////////////////////////////////////////////////////////////////////////////]
[section NaryLexer]

[heading Description]

/NaryLexer/ is a composite lexer component that has one or more subjects. The
NaryLexer allows its subjects to be treated in the same way as a single
instance of a __lexer_concept__ following the __composite_pattern__. 

[heading Refinement of] 

[:__lexer_concept__]

[variablelist Notation
    [[`l`]              [A NaryLexer.]]
    [[`L`]              [A NaryLexer type.]]
]

[heading Valid Expressions]

In addition to the requirements defined in __lexer_concept__, for any
NaryLexer the following must be met:

[table
    [[Expression]       [Semantics]                 [Return type]]
    [[`l.elements`]     [The tuple of elements.]    [A __fusion__ Sequence of __lexer_concept__ types.]]
]

[heading Type Expressions]

[table
    [[Expression]           [Description]]
    [[`l.elements_type`]    [Elements tuple type.]]
    [[`traits::is_nary_lexer<L>::type`]             [Metafunction that evaluates to `mpl::true_` if 
                                                     a certain type, `L` is a NaryLexer, `mpl::false_`  
                                                     otherwise (See __mpl_boolean_constant__).]]
]

[heading Invariants]

For each element, `E`, in any NaryLexer, `L`, the following
invariant always holds: 
  
* `traits::is_lexer<E>::type` evaluates to `mpl::true_`

[heading Models]

The following lexer components conform to this model: 

* lexer sequence component

__fixme__ Add more links to models of NaryLexer concept

[endsect] [/ NaryLexer Concept]

[endsect]