| <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| <html> |
| <head> |
| <title>The Token Type</title> |
| <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> |
| <link rel="stylesheet" href="theme/style.css" type="text/css"> |
| </head> |
| <body text="#000000" background="theme/bkd.gif"> |
| <table width="100%" border="0" cellspacing="2" background="theme/bkd2.gif"> |
| <tr> |
| <td width="21"> <h1></h1></td> |
| <td width="885"> <font face="Verdana, Arial, Helvetica, sans-serif"><b><font size="6">The |
| Token Type</font></b></font></td> |
| <td width="96"><a href="http://www.boost.org"><img src="theme/wave.gif" width="93" height="68" align="right" border="0"></a></td> |
| </tr> |
| </table> |
| <br> |
| <table border="0"> |
| <tr> |
| <td width="10"></td> |
| <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td> |
| <td width="30"><a href="class_reference_lexer.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td> |
| <td width="30"><a href="token_ids.html"><img src="theme/r_arr.gif" border="0"></a></td> |
| </tr> |
| </table> |
| <blockquote> |
| <p><a href="class_reference_tokentype.html#introduction">Introduction</a><br> |
| <a href="class_reference_tokentype.html#header_synopsis">Header 'wave/context.hpp' |
| synopsis</a><br> |
| <a href="class_reference_tokentype.html#template_parameters">Template parameters</a><br> |
| <a href="class_reference_tokentype.html#public_typedefs">Public Typedefs </a><br> |
| <a href="class_reference_tokentype.html#member_functions">Member functions</a></p> |
| </blockquote> |
| <h2><a name="Introduction"></a>Introduction</h2> |
| <p>The token type in <tt>Wave</tt> is the main carrier of information. It is returned |
| by dereferencing the iterators exposed by the lexing component and the iterator exposed by the preprocessor component. The tokens are originally generated by the |
| lexer ("An entity that lexically transforms the subject of parsing to a |
| sequence of objects (called tokens) more suitable for subsequent parsing."). |
| The Wave library contains two different, interchangable C++ lexers, which may |
| be used as a starting point during developing your own application. The lexer |
| generated tokens are transformed by the preprocessing engine (macro expansion, |
| include file processing etc.) and after this returned to the user of the <tt>Wave</tt> |
| library.</p> |
| <p>You can use arbitrary token types in conjunction with your own lexer as long |
| as these implement some required interface. The required token type interface |
| is described below and is implemented by the <tt>wave::cpplexer::lex_token</tt> |
| template, the required lexer interface is described <a href="class_reference_lexer.html">here</a>. |
| </p> |
| <p>In the following is described the token type predefined inside the <tt>Wave</tt> |
| library, which is used in conjunction with the two predefined C++ lexers contained |
| in the <tt>Wave</tt> library too. If you need to use your own token type, it |
| is required to implement the interafce described below.</p> |
| <h2><b><a name="header_synopsis"></a>Header <a href="https://svn.boost.org/trac/boost/browser/trunk/boost/wave/cpplexer/cpp_lex_token.hpp">wave/cpplexer/cpp_lex_token.hpp</a> |
| synopsis</b></h2> |
| <pre><span class="keyword">namespace</span> boost { |
| <span class="keyword">namespace</span> wave { |
| <span class="keyword">namespace</span> cpplexer { |
| |
| <span class="keyword">template</span> <<span class="keyword">typename</span> Position> |
| <span class="keyword">class</span> lex_token |
| { |
| <span class="keyword">public</span>: |
| <span class="keyword">typedef</span> <span class="keyword">std::basic_string</span><char><char> string_type; |
| <span class="keyword">typedef</span> Position position_type; |
| |
| <a href="class_reference_tokentype.html#constructor">lex_token</a>(); |
| <a href="class_reference_tokentype.html#constructor">lex_token</a>(token_id id, string_type <span class="keyword">const</span> &value, |
| PositionT <span class="keyword">const</span> &pos); |
| |
| // accessors |
| <a href="class_reference_tokentype.html#operator_tokenid">operator token_id</a>() <span class="keyword">const</span>; |
| string_type const &<a href="class_reference_tokentype.html#get_value">get_value</a>() <span class="keyword">const</span>; |
| position_type const &<a href="class_reference_tokentype.html#get_position">get_position</a>() <span class="keyword">const</span>; |
| void <a href="class_reference_tokentype.html#set_token_id">set_token_id</a> (token_id id); |
| void <a href="class_reference_tokentype.html#set_value">set_value</a> (string_type <span class="keyword">const</span> &newval); |
| void <a href="class_reference_tokentype.html#set_position">set_position</a> (position_type <span class="keyword">const</span> &pos); |
| }; |
| |
| } <span class="comment">// namespace cpplexer</span> |
| } <span class="comment">// namespace wave</span> |
| } <span class="comment">// namespace boost</span> |
| </pre> |
| <h2><b><a name="template_parameters" id="template_parameters"></a>Template parameters</b></h2> |
| <p>The predefined token type uses a template parameter <tt>Position</tt>, which |
| allows to specify the type to be used to carry the file position information |
| contained inside the generated tokens. Your own token type do not need to take this <tt>Position</tt> template parameter, but please note, that the token type in any case needs to have an embedded type definition <tt>position_type</tt> (see below) . </p> |
| <h2><a name="public_typedefs" id="public_typedefs"></a>Public Typedefs</h2> |
| <p>The token type needs to define two embedded types: <tt>string_type</tt> and <tt>position_type</tt>. The <tt>string_type</tt> needs to be a type compatible to the <tt>std::basic_string<></tt> class. </p> |
| <p>This type should contain at least the filename, the line number and the column number of the position, where the token was recognized. For the predefined token type it defaults to a simple file_position template class described <a href="class_reference_filepos.html">here</a>. Note, that your own <tt>position_type</tt> should follow the interface described for the file_position template as well. </p> |
| <h2><b><a name="member_functions"></a>Member functions</b></h2> |
| <p><b><a name="constructor" id="constructor"></a>Constructors</b></p> |
| <pre> lex_token(); |
| |
| lex_token(token_id id, |
| string_t <span class="keyword">const</span> &value, |
| PositionT <span class="keyword">const</span> &pos);</pre> |
| <blockquote> |
| <p>The first (default) constructor is for generating an end of stream token, |
| which is used for indicating the end of the underlying input stream.</p> |
| <p>The second constructor initializes the newly created token object with its |
| token id (for a list of valid token id's please look <a href="token_ids.html">here</a>), |
| the string representation of its value and the file position, describing the |
| position inside the input stream , where this token was recognized.</p> |
| </blockquote> |
| <h2>Accessor functions</h2> |
| <p><a name="operator_tokenid"></a><b>operator token_id</b></p> |
| <pre> <span class="keyword">operator</span> token_id() <span class="keyword">const</span>;</pre> |
| <blockquote> |
| <p>Allows the access to the token id of the token. This accessor allows the usage of <tt>Spirit</tt> |
| parsers directly on top |
| |
| |
| top of a token stream generated by Wave. The possible token id's are described |
| |
| <a href="token_ids.html">here</a></a>. </p> |
| <p>This function does not throw in any case.</p> |
| </blockquote> |
| <p><a name="get_value"></a><b>get_value</b></p> |
| <pre> string_type <span class="keyword">const</span> &get_value() <span class="keyword">const</span>;</pre> |
| <blockquote> |
| <p>Returns the value of the token, as it was recognized in the input stream. |
| Even for constant tokens (as keywords or operators etc.) the returned value |
| reflects the character sequence as found in the input stream. </p> |
| <p>This function does not throw in any case.</p> |
| </blockquote> |
| <p><b><a name="get_position"></a>get_position</b></p> |
| <pre> Position <span class="keyword">const</span> &get_position() <span class="keyword">const</span>;</pre> |
| <blockquote> |
| <p>Returns the position of the token in the input stream, where it was recognized. |
| The position contains information about the filename, the line number and |
| the column number of the token. By default the <tt>Wave</tt> library uses |
| a file_position template for this purpose, which is described in more detail |
| <a href="class_reference_filepos.html">here</a>.</p> |
| <p>This function does not throw in any case.</p> |
| </blockquote> |
| <p><a name="set_token_id"></a><b>set_token_id</b></p> |
| <pre> void set_token_id(token_id id);</pre> |
| <blockquote> |
| <p>Changes the token id of the token to the new value. The possible token id's |
| are described <a href="token_ids.html">here</a>. Probably this function is |
| of little value for the library user, but it is required under certain circumstances |
| for correct operation of the preprocessing engine.</p> |
| <p>This function does not throw in any case.</p> |
| </blockquote> |
| <p><a name="set_value"></a><b>set_value</b></p> |
| <pre> void set_value(string_type <span class="keyword">const</span> &newval);</pre> |
| <blockquote> |
| <p>Changes the value stored inside the token to the new value. Propably this |
| function is of little value for the library user, but it is required under |
| certain circumstances for correct operation of the preprocessing engine.</p> |
| </blockquote> |
| <p><a name="set_position"></a><b>set_position</b></p> |
| <pre> void set_position(Position <span class="keyword">const</span> &newpos);</pre> |
| <blockquote> |
| <p>Changes the position stored inside the token to the new value. This is used |
| for instance for implementing the functionality required for to implement |
| the <tt>#line</tt> directive.</p> |
| </blockquote> |
| <table border="0"> |
| <tr> |
| <td width="10"></td> |
| <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td> |
| <td width="30"><a href="class_reference_lexer.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td> |
| <td width="30"><a href="token_ids.html"><img src="theme/r_arr.gif" border="0"></a></td> |
| </tr> |
| </table> |
| <hr size="1"> |
| <p class="copyright">Copyright © 2003-2010 Hartmut Kaiser<br> |
| <br> |
| <font size="2">Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) </font> </p> |
| <span class="updated"></span> |
| <p class="copyright"><span class="updated">Last updated: |
| <!-- #BeginDate format:fcAm1m -->Sunday, October 12, 2008 20:15<!-- #EndDate --> |
| </span></p> |
| </body> |
| </html> |
