| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" |
| "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> |
| |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Boost.Python Pickle Support</title> |
| </head> |
| |
| <body> |
| <div> |
| <img src="../../../../boost.png" alt="boost.png (6897 bytes)" align= |
| "center" width="277" height="86" /> |
| <hr /> |
| |
| <h1>Boost.Python Pickle Support</h1>Pickle is a Python module for object |
| serialization, also known as persistence, marshalling, or flattening. |
| |
| <p>It is often necessary to save and restore the contents of an object to |
| a file. One approach to this problem is to write a pair of functions that |
| read and write data from a file in a special format. A powerful |
| alternative approach is to use Python's pickle module. Exploiting |
| Python's ability for introspection, the pickle module recursively |
| converts nearly arbitrary Python objects into a stream of bytes that can |
| be written to a file.</p> |
| |
| <p>The Boost Python Library supports the pickle module through the |
| interface as described in detail in the <a href= |
| "http://www.python.org/doc/current/lib/module-pickle.html">Python Library |
| Reference for pickle.</a> This interface involves the special methods |
| <tt>__getinitargs__</tt>, <tt>__getstate__</tt> and <tt>__setstate__</tt> |
| as described in the following. Note that Boost.Python is also fully |
| compatible with Python's cPickle module.</p> |
| <hr /> |
| |
| <h2>The Boost.Python Pickle Interface</h2>At the user level, the |
| Boost.Python pickle interface involves three special methods: |
| |
| <dl> |
| <dt><strong><tt>__getinitargs__</tt></strong></dt> |
| |
| <dd> |
| When an instance of a Boost.Python extension class is pickled, the |
| pickler tests if the instance has a <tt>__getinitargs__</tt> method. |
| This method must return a Python tuple (it is most convenient to use |
| a boost::python::tuple). When the instance is restored by the |
| unpickler, the contents of this tuple are used as the arguments for |
| the class constructor. |
| |
| <p>If <tt>__getinitargs__</tt> is not defined, <tt>pickle.load</tt> |
| will call the constructor (<tt>__init__</tt>) without arguments; |
| i.e., the object must be default-constructible.</p> |
| </dd> |
| |
| <dt><strong><tt>__getstate__</tt></strong></dt> |
| |
| <dd>When an instance of a Boost.Python extension class is pickled, the |
| pickler tests if the instance has a <tt>__getstate__</tt> method. This |
| method should return a Python object representing the state of the |
| instance.</dd> |
| |
| <dt><strong><tt>__setstate__</tt></strong></dt> |
| |
| <dd>When an instance of a Boost.Python extension class is restored by |
| the unpickler (<tt>pickle.load</tt>), it is first constructed using the |
| result of <tt>__getinitargs__</tt> as arguments (see above). |
| Subsequently the unpickler tests if the new instance has a |
| <tt>__setstate__</tt> method. If so, this method is called with the |
| result of <tt>__getstate__</tt> (a Python object) as the argument.</dd> |
| </dl>The three special methods described above may be <tt>.def()</tt>'ed |
| individually by the user. However, Boost.Python provides an easy to use |
| high-level interface via the |
| <strong><tt>boost::python::pickle_suite</tt></strong> class that also |
| enforces consistency: <tt>__getstate__</tt> and <tt>__setstate__</tt> |
| must be defined as pairs. Use of this interface is demonstrated by the |
| following examples. |
| <hr /> |
| |
| <h2>Examples</h2>There are three files in <tt>boost/libs/python/test</tt> |
| that show how to provide pickle support. |
| <hr /> |
| |
| <h3><a href="../../test/pickle1.cpp"><tt>pickle1.cpp</tt></a></h3>The C++ |
| class in this example can be fully restored by passing the appropriate |
| argument to the constructor. Therefore it is sufficient to define the |
| pickle interface method <tt>__getinitargs__</tt>. This is done in the |
| following way: |
| |
| <ul> |
| <li>1. Definition of the C++ pickle function: |
| <pre> |
| struct world_pickle_suite : boost::python::pickle_suite |
| { |
| static |
| boost::python::tuple |
| getinitargs(world const& w) |
| { |
| return boost::python::make_tuple(w.get_country()); |
| } |
| }; |
| </pre> |
| </li> |
| |
| <li>2. Establishing the Python binding: |
| <pre> |
| class_<world>("world", args<const std::string&>()) |
| // ... |
| .def_pickle(world_pickle_suite()) |
| // ... |
| </pre> |
| </li> |
| </ul> |
| <hr /> |
| |
| <h3><a href="../../test/pickle2.cpp"><tt>pickle2.cpp</tt></a></h3>The C++ |
| class in this example contains member data that cannot be restored by any |
| of the constructors. Therefore it is necessary to provide the |
| <tt>__getstate__</tt>/<tt>__setstate__</tt> pair of pickle interface |
| methods: |
| |
| <ul> |
| <li>1. Definition of the C++ pickle functions: |
| <pre> |
| struct world_pickle_suite : boost::python::pickle_suite |
| { |
| static |
| boost::python::tuple |
| getinitargs(const world& w) |
| { |
| // ... |
| } |
| |
| static |
| boost::python::tuple |
| getstate(const world& w) |
| { |
| // ... |
| } |
| |
| static |
| void |
| setstate(world& w, boost::python::tuple state) |
| { |
| // ... |
| } |
| }; |
| </pre> |
| </li> |
| |
| <li>2. Establishing the Python bindings for the entire suite: |
| <pre> |
| class_<world>("world", args<const std::string&>()) |
| // ... |
| .def_pickle(world_pickle_suite()) |
| // ... |
| </pre> |
| </li> |
| </ul> |
| |
| <p>For simplicity, the <tt>__dict__</tt> is not included in the result of |
| <tt>__getstate__</tt>. This is not generally recommended, but a valid |
| approach if it is anticipated that the object's <tt>__dict__</tt> will |
| always be empty. Note that the safety guard described below will catch |
| the cases where this assumption is violated.</p> |
| <hr /> |
| |
| <h3><a href="../../test/pickle3.cpp"><tt>pickle3.cpp</tt></a></h3>This |
| example is similar to <a href= |
| "../../test/pickle2.cpp"><tt>pickle2.cpp</tt></a>. However, the object's |
| <tt>__dict__</tt> is included in the result of <tt>__getstate__</tt>. |
| This requires a little more code but is unavoidable if the object's |
| <tt>__dict__</tt> is not always empty. |
| <hr /> |
| |
| <h2>Pitfall and Safety Guard</h2>The pickle protocol described above has |
| an important pitfall that the end user of a Boost.Python extension module |
| might not be aware of: |
| |
| <p><strong><tt>__getstate__</tt> is defined and the instance's |
| <tt>__dict__</tt> is not empty.</strong></p> |
| |
| <p>The author of a Boost.Python extension class might provide a |
| <tt>__getstate__</tt> method without considering the possibilities |
| that:</p> |
| |
| <ul> |
| <li>his class is used in Python as a base class. Most likely the |
| <tt>__dict__</tt> of instances of the derived class needs to be pickled |
| in order to restore the instances correctly.</li> |
| |
| <li>the user adds items to the instance's <tt>__dict__</tt> directly. |
| Again, the <tt>__dict__</tt> of the instance then needs to be |
| pickled.</li> |
| </ul> |
| |
| <p>To alert the user to this highly unobvious problem, a safety guard is |
| provided. If <tt>__getstate__</tt> is defined and the instance's |
| <tt>__dict__</tt> is not empty, Boost.Python tests if the class has an |
| attribute <tt>__getstate_manages_dict__</tt>. An exception is raised if |
| this attribute is not defined:</p> |
| <pre> |
| RuntimeError: Incomplete pickle support (__getstate_manages_dict__ not set) |
| </pre>To resolve this problem, it should first be established that the <tt> |
| __getstate__</tt> and <tt>__setstate__</tt> methods manage the |
| instances's <tt>__dict__</tt> correctly. Note that this can be done |
| either at the C++ or the Python level. Finally, the safety guard should |
| intentionally be overridden. E.g. in C++ (from <a href= |
| "../../test/pickle3.cpp"><tt>pickle3.cpp</tt></a>): |
| <pre> |
| struct world_pickle_suite : boost::python::pickle_suite |
| { |
| // ... |
| |
| static bool getstate_manages_dict() { return true; } |
| }; |
| </pre>Alternatively in Python: |
| <pre> |
| import your_bpl_module |
| class your_class(your_bpl_module.your_class): |
| __getstate_manages_dict__ = 1 |
| def __getstate__(self): |
| # your code here |
| def __setstate__(self, state): |
| # your code here |
| </pre> |
| <hr /> |
| |
| <h2>Practical Advice</h2> |
| |
| <ul> |
| <li>In Boost.Python extension modules with many extension classes, |
| providing complete pickle support for all classes would be a |
| significant overhead. In general complete pickle support should only be |
| implemented for extension classes that will eventually be pickled.</li> |
| |
| <li>Avoid using <tt>__getstate__</tt> if the instance can also be |
| reconstructed by way of <tt>__getinitargs__</tt>. This automatically |
| avoids the pitfall described above.</li> |
| |
| <li>If <tt>__getstate__</tt> is required, include the instance's |
| <tt>__dict__</tt> in the Python object that is returned.</li> |
| </ul> |
| <hr /> |
| |
| <h2>Light-weight alternative: pickle support implemented in Python</h2> |
| |
| <h3><a href="../../test/pickle4.cpp"><tt>pickle4.cpp</tt></a></h3>The |
| <tt>pickle4.cpp</tt> example demonstrates an alternative technique for |
| implementing pickle support. First we direct Boost.Python via the |
| <tt>class_::enable_pickling()</tt> member function to define only the |
| basic attributes required for pickling: |
| <pre> |
| class_<world>("world", args<const std::string&>()) |
| // ... |
| .enable_pickling() |
| // ... |
| </pre>This enables the standard Python pickle interface as described in the |
| Python documentation. By "injecting" a <tt>__getinitargs__</tt> method into |
| the definition of the wrapped class we make all instances pickleable: |
| <pre> |
| # import the wrapped world class |
| from pickle4_ext import world |
| |
| # definition of __getinitargs__ |
| def world_getinitargs(self): |
| return (self.get_country(),) |
| |
| # now inject __getinitargs__ (Python is a dynamic language!) |
| world.__getinitargs__ = world_getinitargs |
| </pre>See also the <a href= |
| "../tutorial/doc/html/python/techniques.html#python.extending_wrapped_objects_in_python"> |
| tutorial section</a> on injecting additional methods from Python. |
| <hr /> |
| © Copyright Ralf W. Grosse-Kunstleve 2001-2004. Distributed under the |
| Boost Software License, Version 1.0. (See accompanying file |
| LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) |
| |
| <p>Updated: Feb 2004.</p> |
| </div> |
| </body> |
| </html> |