[/==============================================================================
    Copyright (C) 2001-2010 Joel de Guzman
    Copyright (C) 2001-2010 Hartmut Kaiser

    Distributed under the Boost Software License, Version 1.0. (See accompanying
    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

[section:lexer_token_values About Tokens and Token Values]

As already discussed, lexical scanning is the process of analyzing the stream 
of input characters and separating it into strings called tokens, most of the 
time separated by whitespace. The different token types recognized by a lexical 
analyzer often get assigned unique integer token identifiers (token ids). These 
token ids are normally used by the parser to identify the current token without 
having to look at the matched string again. The __lex__ library is not 
different with respect to this, as it uses the token ids as the main means of 
identification of the different token types defined for a particular lexical 
analyzer. However, it is different from commonly used lexical analyzers in the 
sense that it returns (references to) instances of a (user defined) token class 
to the user. The only limitation of this token class is that it must carry at 
least the token id of the token it represents. For more information about the 
interface a user defined token type has to expose please look at the 
__sec_ref_lex_token__ reference. The library provides a default 
token type based on the __lexertl__ library which should be sufficient in most 
cases: the __class_lexertl_token__ type. This section focusses on the 
description of general features a token class may implement and how this 
integrates with the other parts of the __lex__ library.

[heading The Anatomy of a Token]

It is very important to understand the difference between a token definition 
(represented by the __class_token_def__ template) and a token itself (for 
instance represented by the __class_lexertl_token__ template). 

The token definition is used to describe the main features of a particular 
token type, especially:

* to simplify the definition of a token type using a regular expression pattern 
  applied while matching this token type,
* to associate a token type with a particular lexer state,
* to optionally assign a token id to a token type,
* to optionally associate some code to execute whenever an instance of this 
  token type has been matched,
* and to optionally specify the attribute type of the token value.

The token itself is a data structure returned by the lexer iterators. 
Dereferencing a lexer iterator returns a reference to the last matched token 
instance. It encapsulates the part of the underlying input sequence matched by
the regular expression used during the definition of this token type. 
Incrementing the lexer iterator invokes the lexical analyzer to
match the next token by advancing the underlying input stream. The token data 
structure contains at least the token id of the matched token type, 
allowing to identify the matched character sequence. Optionally, the token 
instance may contain a token value and/or the lexer state this token instance
was matched in. The following [link spirit.lex.tokenstructure figure] shows the 
schematic structure of a token.

[fig tokenstructure.png..The structure of a token..spirit.lex.tokenstructure]

The token value and the lexer state the token has been recognized in may be 
omitted for optimization reasons, thus avoiding the need for the token to carry 
more data than actually required. This configuration can be achieved by supplying 
appropriate template parameters for the 
__class_lexertl_token__ template while defining the token type. 

The lexer iterator returns the same token type for each of the different 
matched token definitions. To accommodate for the possible different token 
/value/ types exposed by the various token types (token definitions), the 
general type of the token value is a __boost_variant__. At a minimum (for the 
default configuration) this token value variant will be configured to always 
hold a __boost_iterator_range__ containing the pair of iterators pointing to 
the matched input sequence for this token instance.

[note If the lexical analyzer is used in conjunction with a __qi__ parser, the 
      stored __boost_iterator_range__ token value will be converted to the 
      requested token type (parser attribute) exactly once. This happens at the 
      time of the first access to the token value requiring the  
      corresponding type conversion. The converted token value will be stored 
      in the __boost_variant__ replacing the initially stored iterator range.
      This avoids having to convert the input sequence to the token value more 
      than once, thus optimizing the integration of the lexer with __qi__, even 
      during parser backtracking.
]

Here is the template prototype of the __class_lexertl_token__ template:

    template <
        typename Iterator = char const*, 
        typename AttributeTypes = mpl::vector0<>, 
        typename HasState = mpl::true_
    >
    struct lexertl_token;

[variablelist where:
    [[Iterator]       [This is the type of the iterator used to access the 
                       underlying input stream. It defaults to a plain 
                       `char const*`.]]
    [[AttributeTypes] [This is either a mpl sequence containing all 
                       attribute types used for the token definitions or the
                       type `omit`. If the mpl sequence is empty (which is 
                       the default), all token instances will store a 
                       __boost_iterator_range__`<Iterator>` pointing to the start
                       and the end of the matched section in the input stream.
                       If the type is `omit`, the generated tokens will
                       contain no token value (attribute) at all.]]
    [[HasState]       [This is either `mpl::true_` or `mpl::false_`, allowing
                       control as to whether the generated token instances will
                       contain the lexer state they were generated in. The 
                       default is mpl::true_, so all token instances will 
                       contain the lexer state.]]
]

Normally, during construction, a token instance always holds the 
__boost_iterator_range__ as its token value, unless it has been defined 
using the `omit` token value type. This iterator range then is 
converted in place to the requested token value type (attribute) when it is
requested for the first time.


[heading The Physiognomy of a Token Definition]

The token definitions (represented by the __class_token_def__ template) are 
normally used as part of the definition of the lexical analyzer. At the same 
time a token definition instance may be used as a parser component in __qi__.

The template prototype of this class is shown here:

    template<
        typename Attribute = unused_type, 
        typename Char = char
    >
    class token_def;

[variablelist where:
    [[Attribute]      [This is the type of the token value (attribute) 
                       supported by token instances representing this token 
                       type. This attribute type is exposed to the __qi__ 
                       library, whenever this token definition is used as a 
                       parser component. The default attribute type is 
                       `unused_type`, which means the token instance holds a
                       __boost_iterator_range__ pointing to the start
                       and the end of the matched section in the input stream.
                       If the attribute is `omit` the token instance will
                       expose no token type at all. Any other type will be 
                       used directly as the token value type.]]
    [[Char]           [This is the value type of the iterator for the 
                       underlying input sequence. It defaults to `char`.]]
]

The semantics of the template parameters for the token type and the token 
definition type are very similar and interdependent. As a rule of thumb you can 
think of the token definition type as the means of specifying everything 
related to a single specific token type (such as `identifier` or `integer`).
On the other hand the token type is used to define the general properties of all
token instances generated by the __lex__ library.

[important If you don't list any token value types in the token type definition 
           declaration (resulting in the usage of the default __boost_iterator_range__ 
           token type) everything will compile and work just fine, just a bit 
           less efficient. This is because the token value will be converted 
           from the matched input sequence every time it is requested. 
           
           But as soon as you specify at least one token value type while 
           defining the token type you'll have to list all value types used for 
           __class_token_def__ declarations in the token definition class, 
           otherwise compilation errors will occur.
]


[heading Examples of using __class_lexertl_token__]

Let's start with some examples. We refer to one of the __lex__ examples (for 
the full source code of this example please see 
[@../../example/lex/example4.cpp example4.cpp]).

[import ../example/lex/example4.cpp]

The first code snippet shows an excerpt of the token definition class, the
definition of a couple of token types. Some of the token types do not expose a
special token value (`if_`, `else_`, and `while_`). Their token value will
always hold the iterator range of the matched input sequence. The token
definitions for the `identifier` and the integer `constant` are specialized 
to expose an explicit token type each: `std::string` and `unsigned int`.

[example4_token_def]

As the parsers generated by __qi__ are fully attributed, any __qi__ parser 
component needs to expose a certain type as its parser attribute. Naturally, 
the __class_token_def__ exposes the token value type as its parser attribute, 
enabling a smooth integration with __qi__.

The next code snippet demonstrates how the required token value types are 
specified while defining the token type to use. All of the token value types
used for at least one of the token definitions have to be re-iterated for the 
token definition as well.

[example4_token]

To avoid the token to have a token value at all, the special tag `omit` can 
be used: `token_def<omit>` and `lexertl_token<base_iterator_type, omit>`.


[endsect]