[/==============================================================================
    Copyright (C) 2001-2010 Joel de Guzman
    Copyright (C) 2001-2010 Hartmut Kaiser

    Distributed under the Boost Software License, Version 1.0. (See accompanying
    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

[section:lexer_tutorials __lex__ Tutorials Overview]

The __lex__ library implements several components on top of possibly different 
lexer generator libraries. It exposes a pair of iterators, which, when 
dereferenced, return a stream of tokens generated from the underlying character 
stream. The generated tokens are based on the token definitions supplied by the 
user.

Currently, __lex__ is built on top of Ben Hanson's excellent __lexertl__ 
library (which is a proposed Boost library). __lexertl__ provides the necessary 
functionality to build state machines based on a set of supplied regular 
expressions. But __lex__ is not restricted to be used with __lexertl__. We 
expect it to be usable in conjunction with any other lexical scanner generator 
library, all what needs to be implemented is a set of wrapper objects exposing a 
well defined interface as described in this documentation.

[note   For the sake of clarity all examples in this documentation assume 
        __lex__ to be used on top of __lexertl__.]

Building a lexer using __lex__ is highly configurable, where most of this 
configuration is done at compile time. Almost all of the configurable 
parameters have generally useful default values, allowing project startup to be
a easy and straightforward task. Here is a (non-complete) list of features you 
can tweak to adjust the generated lexer instance to the actual needs:

* Select and customize the token type to be generated by the lexer instance.
* Select and customize the token value types the generated token instances will 
  be able to hold.
* Select the iterator type of the underlying input stream, which will be used 
  as the source for the character stream to tokenize.
* Customize the iterator type returned by the lexer to enable debug support,
  special handling of certain input sequences, etc.
* Select the /dynamic/ or the /static/ runtime model for the lexical
  analyzer.

Special care has been taken during the development of the library that 
optimal code will be generated regardless of the configuration options 
selected.

The series of tutorial examples of this section will guide you through some
common use cases helping to understand the big picture. The first two quick 
start examples (__sec_lex_quickstart_1__ and __sec_lex_quickstart_2__) 
introduce the __lex__ library while building two stand alone applications, not
being connected to or depending on any other part of __spirit__. The section
__sec_lex_quickstart_3__ demonstrates how to use a lexer in conjunction with a 
parser (where obviously the parser is built using __qi__). 

[endsect]