| <html> |
| <head> |
| <title>In-depth The Scanner</title> |
| <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> |
| <link rel="stylesheet" href="theme/style.css" type="text/css"> |
| </head> |
| |
| <body> |
| <table width="100%" border="0" background="theme/bkd2.gif" cellspacing="2"> |
| <tr> |
| <td width="10"> |
| </td> |
| <td width="85%"> <font size="6" face="Verdana, Arial, Helvetica, sans-serif"><b>In-depth: |
| The Scanner</b></font> </td> |
| <td width="112"><a href="http://spirit.sf.net"><img src="theme/spirit.gif" width="112" height="48" align="right" border="0"></a></td> |
| </tr> |
| </table> |
| <br> |
| <table border="0"> |
| <tr> |
| <td width="10"></td> |
| <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td> |
| <td width="30"><a href="indepth_the_parser.html"><img src="theme/l_arr.gif" border="0"></a></td> |
| <td width="30"><a href="indepth_the_parser_context.html"><img src="theme/r_arr.gif" border="0"></a></td> |
| </tr> |
| </table> |
| <h2>Basic Scanner API </h2> |
| <table width="90%" border="0" align="center"> |
| <tr> |
| <td class="table_title" colspan="10"> class scanner </td> |
| </tr> |
| <tr> |
| <tr> |
| <td class="table_cells"><code><span class=identifier>value_t</span></code></td> |
| <td class="table_cells">typedef: The value type of the scanner's iterator</td> |
| </tr> |
| <td class="table_cells"><code><span class=identifier>ref_t</span></code></td> |
| <td class="table_cells">typedef: The reference type of the scanner's iterator</td> |
| </tr> |
| <td class="table_cells"><code><span class=keyword>bool </span><span class=identifier>at_end</span><span class=special>() |
| </span><span class=keyword>const</span></code></td> |
| <td class="table_cells">Returns true if the input is exhausted</td> |
| </tr> |
| <td class="table_cells"><code><span class=identifier>value_t </span><span class=keyword>operator</span><span class=special>*() |
| </span><span class=keyword>const</span></code></td> |
| <td class="table_cells">Dereference/get a <code><span class=identifier>value_t</span></code> |
| from the input</td> |
| </tr> |
| <td class="table_cells"><code><span class=keyword> </span><span class=identifier>scanner |
| </span><span class=keyword>const</span><span class=special>& </span><span class=keyword>operator</span><span class=special>++()</span></code></td> |
| <td class="table_cells">move the scanner forward</td> |
| </tr> |
| <tr> |
| <td class="table_cells"><code><span class=identifier>IteratorT& first</span><span class=special></span></code></td> |
| <td class="table_cells">The iterator pointing to the current input position. |
| Held by reference</td> |
| </tr> |
| <tr> |
| <td class="table_cells"><code><span class=identifier>IteratorT </span><span class=keyword>const</span> |
| <span class=identifier>last</span><span class=special></span></code></td> |
| <td class="table_cells">The iterator pointing to the end of the input. Held |
| by value</td> |
| </tr> |
| </table> |
| <p> The basic behavior of the scanner is handled by policies. The actual execution |
| of the scanner's public member functions listed in the table above is implemented |
| by the scanner policies.</p> |
| <p> Three sets of policies govern the behavior of the scanner. These policies |
| make it possible to extend Spirit non-intrusively. The scanner policies allow |
| the core-functionality to be extended without requiring any potentially destabilizing |
| changes to the code. A library writer might provide her own policies that override |
| the ones that are already in place to fine tune the parsing process |
| to fit her own needs. Layers above the core might also want to take advantage |
| of this policy based machanism. Abstract syntax tree generation, debuggers and |
| lexers come to mind.</p> |
| <p> There are three sets of policies that govern:</p> |
| <ul> |
| <li>Iteration and filtering</li> |
| <li>Recognition and matching</li> |
| <li>Handling semantic actions</li> |
| </ul> |
| <a name="iteration_policy"></a> |
| <h2>iteration_policy</h2> |
| <p> Here are the default policies that govern iteration and filtering:</p> |
| <pre> |
| <code><span class=keyword>struct </span><span class=identifier>iteration_policy |
| </span><span class=special>{ |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>> |
| </span><span class=keyword>void |
| </span><span class=identifier>advance</span><span class=special>(</span><span class=identifier>ScannerT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>scan</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=special>++</span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>first</span><span class=special>; </span><span class=special>} |
| |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>> |
| </span><span class=keyword>bool </span><span class=identifier>at_end</span><span class=special>(</span><span class=identifier>ScannerT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>scan</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=keyword>return </span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>first </span><span class=special>== </span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>last</span><span class=special>; </span><span class=special>} |
| |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>T</span><span class=special>> |
| </span><span class=identifier>T </span><span class=identifier>filter</span><span class=special>(</span><span class=identifier>T </span><span class=identifier>ch</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=keyword>return </span><span class=identifier>ch</span><span class=special>; </span><span class=special>} |
| |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>> |
| </span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>::</span><span class=identifier>ref_t |
| </span><span class=identifier>get</span><span class=special>(</span><span class=identifier>ScannerT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>scan</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=keyword>return </span><span class=special>*</span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>first</span><span class=special>; </span><span class=special>} |
| </span><span class=special>};</span></code></pre> |
| <table width="90%" border="0" align="center"> |
| <tr> |
| <td class="table_title" colspan="8"> Iteration and filtering policies </td> |
| </tr> |
| <tr> |
| <tr> |
| <td class="table_cells"><b>advance</b></td> |
| <td class="table_cells">Move the iterator forward</td> |
| </tr> |
| <td class="table_cells"><b>at_end</b></td> |
| <td class="table_cells">Return true if the input is exhausted</td> |
| </tr> |
| <td class="table_cells"><b>filter</b></td> |
| <td class="table_cells">Filter a character read from the input</td> |
| </tr> |
| <td class="table_cells"><b>get</b></td> |
| <td class="table_cells">Read a character from the input</td> |
| </tr> |
| </table> |
| <p> The following code snippet demonstrates a simple policy that converts all |
| characters to lower case:</p> |
| <pre> |
| <code><span class=keyword>struct </span><span class=identifier>inhibit_case_iteration_policy </span><span class=special>: </span><span class=keyword>public </span><span class=identifier>iteration_policy |
| </span><span class=special>{ |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>CharT</span><span class=special>> |
| </span><span class=identifier>CharT filter</span><span class=special>(</span><span class=identifier>CharT ch</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ |
| </span><span class=keyword>return </span>std::<span class=identifier>tolower</span><span class=special>(</span><span class=identifier>ch</span><span class=special>); |
| } |
| };</span></code></pre> |
| <a name="match_policy"></a> |
| <h2>match_policy</h2> |
| <p> Here are the default policies that govern recognition and matching:</p> |
| <pre> |
| <code><span class=keyword>struct </span><span class=identifier>match_policy |
| </span><span class=special>{ |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>T</span><span class=special>> |
| </span><span class=keyword>struct </span><span class=identifier>result </span><span class=special> |
| { |
| </span><span class=keyword>typedef </span><span class=identifier>match</span><span class=special><</span><span class=identifier>T</span><span class=special>> </span><span class=identifier>type</span><span class=special>; </span><span class=special> |
| }; |
| |
| </span><span class=keyword>const </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>> |
| </span><span class=identifier>no_match</span><span class=special>() </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=keyword> |
| return </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>>(); </span><span class=special> |
| } |
| |
| </span><span class=keyword>const </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>> |
| </span><span class=identifier>empty_match</span><span class=special>() </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=keyword> |
| return </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>>(</span><span class=number>0</span><span class=special>, </span><span class=identifier>nil_t</span><span class=special>()); |
| </span><span class=special>} |
| |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>AttrT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>IteratorT</span><span class=special>> |
| </span><span class=identifier>match</span><span class=special><</span><span class=identifier>AttrT</span><span class=special>> |
| </span><span class=identifier>create_match</span><span class=special>( |
| </span><span class=keyword>std::size_t </span><span class=identifier>length</span><span class=special>, |
| </span><span class=identifier>AttrT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>val</span><span class=special>, |
| </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*first*/</span><span class=special>, |
| </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*last*/</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=keyword> |
| return </span><span class=identifier>match</span><span class=special><</span><span class=identifier>AttrT</span><span class=special>>(</span><span class=identifier>length</span><span class=special>, </span><span class=identifier>val</span><span class=special>); </span><span class=special> |
| } |
| |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>MatchT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>IteratorT</span><span class=special>> |
| </span><span class=keyword>void |
| </span><span class=identifier>group_match</span><span class=special>( |
| </span><span class=identifier>MatchT</span><span class=special>& </span><span class=comment>/*m*/</span><span class=special>, |
| </span><span class=identifier>parser_id </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*id*/</span><span class=special>, |
| </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*first*/</span><span class=special>, |
| </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*last*/</span><span class=special>) </span><span class=keyword>const </span><span class=special>{} |
| |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>Match1T</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>Match2T</span><span class=special>> |
| </span><span class=keyword>void |
| </span><span class=identifier>concat_match</span><span class=special>(</span><span class=identifier>Match1T</span><span class=special>& </span><span class=identifier>l</span><span class=special>, </span><span class=identifier>Match2T </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>r</span><span class=special>) </span><span class=keyword>const |
| </span><span class=special>{ </span><span class=identifier> |
| l</span><span class=special>.</span><span class=identifier>concat</span><span class=special>(</span><span class=identifier>r</span><span class=special>); |
| </span><span class=special>} |
| </span><span class=special>};</span></code></pre> |
| <table width="90%" border="0" align="center"> |
| <tr> |
| <td class="table_title" colspan="12"> Recognition and matching </td> |
| </tr> |
| <tr> |
| <tr> |
| <td class="table_cells"><b>result</b></td> |
| <td class="table_cells">A metafunction that returns a match type given an |
| attribute type (see In-depth: The Parser)</td> |
| </tr> |
| <td class="table_cells"><b>no_match</b></td> |
| <td class="table_cells">Create a failed match</td> |
| </tr> |
| <td class="table_cells"><b>empty_match</b></td> |
| <td class="table_cells">Create an empty match. An empty match is a successful |
| epsilon match (matching length == 0)</td> |
| </tr> |
| <td class="table_cells"><b>create_match</b></td> |
| <td class="table_cells">Create a match given the matching length, an attribute |
| and the iterator pair pointing to the matching portion of the input</td> |
| </tr> |
| <td class="table_cells"><b>group_match</b></td> |
| <td class="table_cells">For non terminals such as rules, this is called after |
| a successful match has been made to allow post processing</td> |
| </tr> |
| <td class="table_cells"><b>concat_match</b></td> |
| <td class="table_cells">Concatenate two match objects</td> |
| </tr> |
| </table> |
| <a name="action_policy"></a> |
| <h2>action_policy</h2> |
| <p> The action policy has only one function for handling semantic actions:</p> |
| <pre> |
| <code><span class=keyword>struct </span><span class=identifier>action_policy |
| </span><span class=special>{ |
| </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ActorT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>AttrT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>IteratorT</span><span class=special>> |
| </span><span class=keyword>void |
| </span><span class=identifier>do_action</span><span class=special>( |
| </span><span class=identifier>ActorT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>actor</span><span class=special>, |
| </span><span class=identifier>AttrT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>val</span><span class=special>, |
| </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>first</span><span class=special>, |
| </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>last</span><span class=special>) </span><span class=keyword>const</span><span class=special>; |
| </span><span class=special>};</span></code></pre> |
| <p> The default action policy forwards to:</p> |
| <pre> |
| <code><span class=identifier>actor</span><span class=special>(</span><span class=identifier>first</span><span class=special>, </span><span class=identifier>last</span><span class=special>);</span></code></pre> |
| <p> If the attribute <tt>val</tt> is of type nil_t. Otherwise:</p> |
| <pre> |
| <code><span class=identifier>actor</span><span class=special>(</span><span class=identifier>val</span><span class=special>);</span></code></pre> |
| <a name="scanner_policies_mixer"></a> |
| <h3>scanner_policies mixer</h3> |
| <p> The class <tt>scanner_policies</tt> combines the three scanner policy classes |
| above into one:</p> |
| <pre> |
| <code><span class=keyword>template </span><span class=special>< |
| </span><span class=keyword>typename </span><span class=identifier>IterationPolicyT </span><span class=special>= </span><span class=identifier>iteration_policy</span><span class=special>, |
| </span><span class=keyword>typename </span><span class=identifier>MatchPolicyT </span><span class=special>= </span><span class=identifier>match_policy</span><span class=special>, |
| </span><span class=keyword>typename </span><span class=identifier>ActionPolicyT </span><span class=special>= </span><span class=identifier>action_policy</span><span class=special>> |
| </span><span class=keyword>struct </span><span class=identifier>scanner_policies</span><span class=special>; |
| </span></code></pre> |
| <p> This <i>mixer</i> class inherits from all the three policies. This scanner_policies |
| class is then used to parameterize the scanner:</p> |
| <pre> |
| <code><span class=keyword>template </span><span class=special>< |
| </span><span class=keyword>typename </span><span class=identifier>IteratorT </span><span class=special>= </span><span class=keyword>char </span><span class=keyword>const</span><span class=special>*, |
| </span><span class=keyword>typename </span><span class=identifier>PoliciesT </span><span class=special>= </span><span class=identifier>scanner_policies</span><span class=special><> </span><span class=special>> |
| </span><span class=keyword>class </span><span class=identifier>scanner</span><span class=special>; |
| </span></code></pre> |
| <p> The scanner in turn inherits from the PoliciesT.</p> |
| <a name="rebinding_policies"></a> |
| <h3>Rebinding Policies</h3> |
| <p> The scanner can be made to rebind to a different set of policies anytime. |
| It has a member function <tt>change_policies(new_policies)</tt>. Given a new |
| set of policies, this member function creates a new scanner with the new set |
| of policies. The result type of the <i>rebound</i> scanner can be can be obtained |
| by calling the metafunction:</p> |
| <pre> |
| <code><span class=identifier>rebind_scanner_policies</span><span class=special><</span><span class=identifier>ScannerT</span><span class=special>, </span><span class=identifier>PoliciesT</span><span class=special>>::</span><span class=identifier>type</span></code></pre> |
| <a name="rebinding_iterators"></a> |
| <h3>Rebinding Iterators</h3> |
| <p> The scanner can also be made to rebind to a different iterator type anytime. |
| It has a member function <tt>change_iterator(first, last)</tt>. Given a new |
| pair of iterator of type different from the ones held by the scanner, this member |
| function creates a new scanner with the new pair of iterators. The result type |
| of the <i>rebound</i> scanner can be can be obtained by calling the metafunction:</p> |
| <pre> |
| <code><span class=identifier>rebind_scanner_iterator</span><span class=special><</span><span class=identifier>ScannerT</span><span class=special>, </span><span class=identifier>IteratorT</span><span class=special>>::</span><span class=identifier>type</span></code></pre> |
| <table border="0"> |
| <tr> |
| <td width="10"></td> |
| <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td> |
| <td width="30"><a href="indepth_the_parser.html"><img src="theme/l_arr.gif" border="0"></a></td> |
| <td width="30"><a href="indepth_the_parser_context.html"><img src="theme/r_arr.gif" border="0"></a></td> |
| </tr> |
| </table> |
| <br> |
| <hr size="1"> |
| <p class="copyright">Copyright © 1998-2003 Joel de Guzman<br> |
| <br> |
| <font size="2">Use, modification and distribution is subject to the Boost Software |
| License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at |
| http://www.boost.org/LICENSE_1_0.txt)</font></p> |
| <p class="copyright"> </p> |
| </body> |
| </html> |
