| <html> |
| <head> |
| <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> |
| <title>Expected failures specification</title> |
| <link rel="stylesheet" href="../../../../style/style.css" type="text/css"> |
| <meta name="generator" content="DocBook XSL Stylesheets V1.74.0"> |
| <link rel="home" href="../../../index.html" title="Boost Test Library"> |
| <link rel="up" href="../test-organization.html" title="Test organization or the house that Jack built"> |
| <link rel="prev" href="master-test-suite.html" title="Master Test Suite"> |
| <link rel="next" href="../fixture.html" title="Fixtures or let me repeat myself"> |
| <script language="JavaScript1.2" src="../../../../js/boost-test.js"></script> |
| </head> |
| <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> |
| <table width="100%"><tr> |
| <td width="10%"><a href="../../../index.html"><img alt="Home" width="229" height="61" border="0" src="../../../../../../../libs/test/docbook/img/boost.test.logo.png"></a></td> |
| <td valign="middle" align="left"> > <a href="../../../utf.html">The Unit Test Framework</a> > <a href="../../user-guide.html">User's guide</a><a href="../../testing-tools.html"> |
| > |
| </a><a href="../test-organization.html">Test organization</a><a href="../fixture.html"> |
| > |
| </a><b>Expected failures specification</b> |
| </td> |
| <td><div class="spirit-nav"> |
| <a href="master-test-suite.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a href="../fixture.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a> |
| </div></td> |
| </tr></table> |
| <hr> |
| <div class="section" lang="en"> |
| <div class="titlepage"><div><div><h5 class="title"> |
| <a name="utf.user-guide.test-organization.expected-failures"></a>Expected failures specification</h5></div></div></div> |
| <p class="first-line-indented"> |
| While in a perfect world all test assertions should pass in order for a test module to pass, in some situations |
| it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not |
| implemented yet and one needs to prepare a library for the release or when particular test fails on some |
| platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature. |
| </p> |
| <p class="first-line-indented"> |
| This feature allows specifying an expected number of failed assertions per test unit. The value is specified |
| during test tree construction, and can't be updated during test execution. |
| </p> |
| <p class="first-line-indented"> |
| The feature is not intended to be used to check for expected functionality failures. To check that a particular |
| input is causing an exception to be thrown use <code class="computeroutput"><a class="link" href="../../testing-tools/reference.html" title="The UTF testing tools reference">BOOST_CHECK_THROW</a></code> family of testing |
| tools. |
| </p> |
| <p class="first-line-indented"> |
| The usage of this feature should be limited and employed only after careful consideration. In general you should |
| only use this feature when it is necessary to force a test module to pass without actually fixing the problem. |
| Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only |
| needs be applied temporarily. |
| </p> |
| <p class="first-line-indented"> |
| You also need to remember that the expected failure specification is per test case. This means that any failed |
| assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an |
| unexpected failure to occur to satisfy this quota. |
| </p> |
| <div class="note"><table border="0" summary="Note"> |
| <tr> |
| <td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/html/images/note.png"></td> |
| <th align="left">Note</th> |
| </tr> |
| <tr><td align="left" valign="top"><p> |
| If an assertion at fault is fixed and passed, while an expected failures specification still present, the test |
| case is going to fail, since the number of failures is smaller than expected. |
| </p></td></tr> |
| </table></div> |
| <div class="section" lang="en"> |
| <div class="titlepage"><div><div><h6 class="title"> |
| <a name="utf.user-guide.test-organization.manual-expected-failures"></a>Usage with manually registered test cases</h6></div></div></div> |
| <p class="first-line-indented"> |
| To set the value of expected failures for the manually registered test unit pass it as a second argument for the |
| test_suite::add call during test unit registration. |
| </p> |
| <div class="example"> |
| <a name="utf.user-guide.test-organization.manual-expected-failures.example16"></a><p class="title"><b>Example 21. Expected failures specification for manually registered test case</b></p> |
| <div class="example-contents"> |
| <pre class="programlisting">#include <boost/test/included/unit_test.hpp> |
| using namespace boost::unit_test; |
| |
| //____________________________________________________________________________// |
| |
| void free_test_function() |
| { |
| BOOST_CHECK( 2 == 1 ); |
| } |
| |
| //____________________________________________________________________________// |
| |
| test_suite* |
| init_unit_test_suite( int, char* [] ) { |
| framework::master_test_suite(). |
| add( BOOST_TEST_CASE( &free_test_function ), 2 ); |
| |
| return 0; |
| } |
| |
| //____________________________________________________________________________// |
| </pre> |
| <table class="simplelist" border="0" summary="Simple list"><tr> |
| <td><code class="literal"><a href="../../../../src/examples/example16.cpp" target="_top">Source code</a></code></td> |
| <td> | </td> |
| <td><code class="literal"><a href="#" target="_top" id="id652090" onclick="toggle_element( 'example16-output', 'id652090', 'Show output', 'Hide output' ); return false;">Show output</a></code></td> |
| </tr></table> |
| <pre class="example-output" id="example16-output">> example --log_level=message |
| Running 1 test case... |
| test.cpp(8): error in "free_test_function": check 2 == 1 failed |
| Test case has less failures then expected |
| |
| *** No errors detected |
| </pre> |
| </div> |
| </div> |
| <br class="example-break"> |
| </div> |
| <div class="section" lang="en"> |
| <div class="titlepage"><div><div><h6 class="title"> |
| <a name="utf.user-guide.test-organization.auto-expected-failures"></a>Usage with automatically registered test cases</h6></div></div></div> |
| <p class="first-line-indented"> |
| To set the number of expected failures for the automatically registered test unit use the macro |
| BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES before the test case definition. |
| </p> |
| <pre class="inline-synopsis"> |
| <a name="BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES"></a>BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(<span class="emphasis"><em>test_case_name</em></span>, <span class="emphasis"><em>number_of_expected_failures</em></span>)</pre> |
| <p class="first-line-indented"> |
| You can use this macro both on a file scope and inside a test suite. Moreover you can use it even if name of test |
| units coincide in different test suites. Expected failures specification applies to the test unit belonging to the same |
| test suite where BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES resides. |
| </p> |
| <div class="example"> |
| <a name="utf.user-guide.test-organization.auto-expected-failures.example17"></a><p class="title"><b>Example 22. Expected failures specification for automatically registered test case</b></p> |
| <div class="example-contents"> |
| <pre class="programlisting">#define BOOST_TEST_MODULE example |
| #include <boost/test/included/unit_test.hpp> |
| |
| //____________________________________________________________________________// |
| |
| BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 1 ) |
| |
| BOOST_AUTO_TEST_CASE( my_test1 ) |
| { |
| BOOST_CHECK( 2 == 1 ); |
| } |
| |
| //____________________________________________________________________________// |
| |
| BOOST_AUTO_TEST_SUITE( internal ) |
| |
| BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 2 ) |
| |
| BOOST_AUTO_TEST_CASE( my_test1 ) |
| { |
| BOOST_CHECK_EQUAL( sizeof(int), sizeof(char) ); |
| BOOST_CHECK_EQUAL( sizeof(int*), sizeof(char) ); |
| } |
| |
| //____________________________________________________________________________// |
| |
| BOOST_AUTO_TEST_SUITE_END()</pre> |
| <table class="simplelist" border="0" summary="Simple list"><tr> |
| <td><code class="literal"><a href="../../../../src/examples/example17.cpp" target="_top">Source code</a></code></td> |
| <td> | </td> |
| <td><code class="literal"><a href="#" target="_top" id="id652235" onclick="toggle_element( 'example17-output', 'id652235', 'Show output', 'Hide output' ); return false;">Show output</a></code></td> |
| </tr></table> |
| <pre class="example-output" id="example17-output">> example --report_level=short |
| Running 2 test cases... |
| test.cpp(10): error in "my_test1": check 2 == 1 failed |
| test.cpp(21): error in "my_test1": check sizeof(int) == sizeof(char) failed [4 != 1] |
| test.cpp(22): error in "my_test1": check sizeof(int*) == sizeof(char) failed [4 != 1] |
| |
| Test suite "example" passed with: |
| 3 assertions out of 3 failed |
| 3 failures expected |
| 2 test cases out of 2 passed |
| </pre> |
| </div> |
| </div> |
| <br class="example-break"> |
| </div> |
| </div> |
| <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> |
| <td align="left"></td> |
| <td align="right"><div class="copyright-footer">Copyright © 2001-2007 Gennadiy Rozental</div></td> |
| </tr></table> |
| <hr> |
| <div class="spirit-nav"> |
| <a accesskey="p" href="master-test-suite.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../test-organization.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../fixture.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a> |
| </div> |
| </body> |
| </html> |
