boost_1_45_0/libs/tokenizer/introduc.htm - nest-learning-thermostat/5.0/boost - Git at Google

 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

 <html>
 <head>
   <meta http-equiv="Content-Language" content="en-us">
   <meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
   <meta name="GENERATOR" content="Microsoft FrontPage 6.0">
   <meta name="ProgId" content="FrontPage.Editor.Document">

   <title>Introduction</title>
 </head>

 <body bgcolor="#FFFFFF">
   <p><img src="../../boost.png" alt="C++ Boost" width="277" height=
   "86"><br></p>

   <h1 align="center">Introduction</h1>

   <p align="left">The boost Tokenizer package provides a flexible and easy to
   use way to break of a string or other character sequence into a series of
   tokens. Below is a simple example that will break up a phrase into
   words.</p>

   <div align="left">
     <pre>
 // simple_example_1.cpp
 #include&lt;iostream&gt;
 #include&lt;boost/tokenizer.hpp&gt;
 #include&lt;string&gt;

 int main(){
    using namespace std;
    using namespace boost;
    string s = "This is,  a test";
    tokenizer&lt;&gt; tok(s);
    for(tokenizer&lt;&gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
        cout &lt;&lt; *beg &lt;&lt; "\n";
    }
 }
 </pre>
   </div>

   <p align="left">You can choose how the string gets broken up. You do this
   by specifying the TokenizerFunction. If you do not specify anything, the
   default TokenizerFunction is char_delimiters_separator&lt;char&gt; which
   defaults to breaking up a string based on space and punctuation. Here is an
   example of using another TokenizerFunction called escaped_list_separator.
   This TokenizerFunction parses a superset of comma separated value (csv)
   lines. The format looks like this</p>

   <p align="left">Field 1,"putting quotes around fields, allows commas",Field
   3</p>

   <p align="left">Below is an example that will break the previous line into
   its 3 fields</p>

   <div align="left">
     <pre>
 // simple_example_2.cpp
 #include&lt;iostream&gt;
 #include&lt;boost/tokenizer.hpp&gt;
 #include&lt;string&gt;

 int main(){
    using namespace std;
    using namespace boost;
    string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3";
    tokenizer&lt;escaped_list_separator&lt;char&gt; &gt; tok(s);
    for(tokenizer&lt;escaped_list_separator&lt;char&gt; &gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
        cout &lt;&lt; *beg &lt;&lt; "\n";
    }
 }
 </pre>
   </div>

   <p align="left">Finally, for some TokenizerFunctions you have to pass in
   something into the constructor in order to do anything interesting. An
   example is offset_separator. This class breaks a string into tokens based
   on offsets for example</p>

   <p align="left">12252001 when parsed using offsets of 2,2,4 becomes 12 25
   2001. Below is an example to parse this.</p>

   <div align="left">
     <pre>
 // simple_example_3.cpp
 #include&lt;iostream&gt;
 #include&lt;boost/tokenizer.hpp&gt;
 #include&lt;string&gt;

 int main(){
    using namespace std;
    using namespace boost;
    string s = "12252001";
    int offsets[] = {2,2,4};
    offset_separator f(offsets, offsets+3);
    tokenizer&lt;offset_separator&gt; tok(s,f);
    for(tokenizer&lt;offset_separator&gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
        cout &lt;&lt; *beg &lt;&lt; "\n";
    }
 }
 </pre>
   </div>

   <p align="left">&nbsp;</p>
   <hr>

   <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src=
   "../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional"
   height="31" width="88"></a></p>

   <p>Revised
   <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>

   <p><i>Copyright &copy; 2001 John R. Bandela</i></p>

   <p><i>Distributed under the Boost Software License, Version 1.0. (See
   accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or
   copy at <a href=
   "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p>
 </body>
 </html>
	<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

	<html>
	<head>
	<meta http-equiv="Content-Language" content="en-us">
	<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
	<meta name="GENERATOR" content="Microsoft FrontPage 6.0">
	<meta name="ProgId" content="FrontPage.Editor.Document">

	<title>Introduction</title>
	</head>

	<body bgcolor="#FFFFFF">
	<p><img src="../../boost.png" alt="C++ Boost" width="277" height=
	"86"><br></p>

	<h1 align="center">Introduction</h1>

	<p align="left">The boost Tokenizer package provides a flexible and easy to
	use way to break of a string or other character sequence into a series of
	tokens. Below is a simple example that will break up a phrase into
	words.</p>

	<div align="left">
	<pre>
	// simple_example_1.cpp
	#include<iostream>
	#include<boost/tokenizer.hpp>
	#include<string>

	int main(){
	using namespace std;
	using namespace boost;
	string s = "This is, a test";
	tokenizer<> tok(s);
	for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
	cout << *beg << "\n";
	}
	}
	</pre>
	</div>

	<p align="left">You can choose how the string gets broken up. You do this
	by specifying the TokenizerFunction. If you do not specify anything, the
	default TokenizerFunction is char_delimiters_separator<char> which
	defaults to breaking up a string based on space and punctuation. Here is an
	example of using another TokenizerFunction called escaped_list_separator.
	This TokenizerFunction parses a superset of comma separated value (csv)
	lines. The format looks like this</p>

	<p align="left">Field 1,"putting quotes around fields, allows commas",Field
	3</p>

	<p align="left">Below is an example that will break the previous line into
	its 3 fields</p>

	<div align="left">
	<pre>
	// simple_example_2.cpp
	#include<iostream>
	#include<boost/tokenizer.hpp>
	#include<string>

	int main(){
	using namespace std;
	using namespace boost;
	string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3";
	tokenizer<escaped_list_separator<char> > tok(s);
	for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){
	cout << *beg << "\n";
	}
	}
	</pre>
	</div>

	<p align="left">Finally, for some TokenizerFunctions you have to pass in
	something into the constructor in order to do anything interesting. An
	example is offset_separator. This class breaks a string into tokens based
	on offsets for example</p>

	<p align="left">12252001 when parsed using offsets of 2,2,4 becomes 12 25
	2001. Below is an example to parse this.</p>

	<div align="left">
	<pre>
	// simple_example_3.cpp
	#include<iostream>
	#include<boost/tokenizer.hpp>
	#include<string>

	int main(){
	using namespace std;
	using namespace boost;
	string s = "12252001";
	int offsets[] = {2,2,4};
	offset_separator f(offsets, offsets+3);
	tokenizer<offset_separator> tok(s,f);
	for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){
	cout << *beg << "\n";
	}
	}
	</pre>
	</div>

	<p align="left"> </p>
	<hr>

	<p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src=
	"../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional"
	height="31" width="88"></a></p>

	<p>Revised
	<!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>

	<p><i>Copyright © 2001 John R. Bandela</i></p>

	<p><i>Distributed under the Boost Software License, Version 1.0. (See
	accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or
	copy at <a href=
	"http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p>
	</body>
	</html>