| <html> |
| <head> |
| <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII"> |
| <title>Python Bindings</title> |
| <link rel="stylesheet" href="../../../doc/src/boostbook.css" type="text/css"> |
| <meta name="generator" content="DocBook XSL Stylesheets V1.75.2"> |
| <link rel="home" href="../index.html" title="The Boost C++ Libraries BoostBook Documentation Subset"> |
| <link rel="up" href="../mpi.html" title="Chapter 12. Boost.MPI"> |
| <link rel="prev" href="../boost/mpi/timer.html" title="Class timer"> |
| <link rel="next" href="design.html" title="Design Philosophy"> |
| </head> |
| <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> |
| <table cellpadding="2" width="100%"><tr> |
| <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../boost.png"></td> |
| <td align="center"><a href="../../../index.html">Home</a></td> |
| <td align="center"><a href="../../../libs/libraries.htm">Libraries</a></td> |
| <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> |
| <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> |
| <td align="center"><a href="../../../more/index.htm">More</a></td> |
| </tr></table> |
| <hr> |
| <div class="spirit-nav"> |
| <a accesskey="p" href="../boost/mpi/timer.html"><img src="../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../mpi.html"><img src="../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="design.html"><img src="../../../doc/src/images/next.png" alt="Next"></a> |
| </div> |
| <div class="section"> |
| <div class="titlepage"><div><div><h2 class="title" style="clear: both"> |
| <a name="mpi.python"></a>Python Bindings</h2></div></div></div> |
| <div class="toc"><dl> |
| <dt><span class="section"><a href="python.html#mpi.python_quickstart">Quickstart</a></span></dt> |
| <dt><span class="section"><a href="python.html#mpi.python_user_data">Transmitting User-Defined Data</a></span></dt> |
| <dt><span class="section"><a href="python.html#mpi.python_collectives">Collectives</a></span></dt> |
| <dt><span class="section"><a href="python.html#mpi.python_skeleton_content">Skeleton/Content Mechanism</a></span></dt> |
| <dt><span class="section"><a href="python.html#mpi.python_compatbility">C++/Python MPI Compatibility</a></span></dt> |
| <dt><span class="section"><a href="python.html#mpi.pythonref">Reference</a></span></dt> |
| </dl></div> |
| <p> |
| Boost.MPI provides an alternative MPI interface from the <a href="http://www.python.org" target="_top">Python</a> |
| programming language via the <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code> module. |
| The Boost.MPI Python bindings, built on top of the C++ Boost.MPI using the |
| <a href="http://www.boost.org/libs/python/doc" target="_top">Boost.Python</a> library, |
| provide nearly all of the functionality of Boost.MPI within a dynamic, object-oriented |
| language. |
| </p> |
| <p> |
| The Boost.MPI Python module can be built and installed from the <code class="computeroutput"><span class="identifier">libs</span><span class="special">/</span><span class="identifier">mpi</span><span class="special">/</span><span class="identifier">build</span></code> directory. |
| Just follow the <a class="link" href="getting_started.html#mpi.config" title="Configure and Build">configuration</a> and <a class="link" href="getting_started.html#mpi.installation" title="Installing and Using Boost.MPI">installation</a> |
| instructions for the C++ Boost.MPI. Once you have installed the Python module, |
| be sure that the installation location is in your <code class="computeroutput"><span class="identifier">PYTHONPATH</span></code>. |
| </p> |
| <div class="section"> |
| <div class="titlepage"><div><div><h3 class="title"> |
| <a name="mpi.python_quickstart"></a>Quickstart</h3></div></div></div> |
| <p> |
| Getting started with the Boost.MPI Python module is as easy as importing |
| <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code>. Our first "Hello, World!" |
| program is just two lines long: |
| </p> |
| <pre class="programlisting"><span class="keyword">import</span> <span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span> <span class="keyword">as</span> <span class="identifier">mpi</span> |
| <span class="keyword">print</span> <span class="string">"I am process %d of %d."</span> <span class="special">%</span> <span class="special">(</span><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">rank</span><span class="special">,</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">size</span><span class="special">)</span> |
| </pre> |
| <p> |
| Go ahead and run this program with several processes. Be sure to invoke the |
| <code class="computeroutput"><span class="identifier">python</span></code> interpreter from |
| <code class="computeroutput"><span class="identifier">mpirun</span></code>, e.g., |
| </p> |
| <pre class="programlisting">mpirun -np 5 python hello_world.py |
| </pre> |
| <p> |
| This will return output such as: |
| </p> |
| <pre class="programlisting">I am process 1 of 5. |
| I am process 3 of 5. |
| I am process 2 of 5. |
| I am process 4 of 5. |
| I am process 0 of 5. |
| </pre> |
| <p> |
| Point-to-point operations in Boost.MPI have nearly the same syntax in Python |
| as in C++. We can write a simple two-process Python program that prints "Hello, |
| world!" by transmitting Python strings: |
| </p> |
| <pre class="programlisting"><span class="keyword">import</span> <span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span> <span class="keyword">as</span> <span class="identifier">mpi</span> |
| |
| <span class="keyword">if</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span> <span class="special">==</span> <span class="number">0</span><span class="special">:</span> |
| <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="string">'Hello'</span><span class="special">)</span> |
| <span class="identifier">msg</span> <span class="special">=</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">1</span><span class="special">)</span> |
| <span class="keyword">print</span> <span class="identifier">msg</span><span class="special">,</span><span class="string">'!'</span> |
| <span class="keyword">else</span><span class="special">:</span> |
| <span class="identifier">msg</span> <span class="special">=</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">)</span> |
| <span class="keyword">print</span> <span class="special">(</span><span class="identifier">msg</span> <span class="special">+</span> <span class="string">', '</span><span class="special">),</span> |
| <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">1</span><span class="special">,</span> <span class="string">'world'</span><span class="special">)</span> |
| </pre> |
| <p> |
| There are only a few notable differences between this Python code and the |
| example <a class="link" href="tutorial.html#mpi.point_to_point" title="Point-to-Point communication">in the C++ tutorial</a>. First |
| of all, we don't need to write any initialization code in Python: just loading |
| the <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code> module makes the appropriate <code class="computeroutput"><span class="identifier">MPI_Init</span></code> and <code class="computeroutput"><span class="identifier">MPI_Finalize</span></code> |
| calls. Second, we're passing Python objects from one process to another through |
| MPI. Any Python object that can be pickled can be transmitted; the next section |
| will describe in more detail how the Boost.MPI Python layer transmits objects. |
| Finally, when we receive objects with <code class="computeroutput"><span class="identifier">recv</span></code>, |
| we don't need to specify the type because transmission of Python objects |
| is polymorphic. |
| </p> |
| <p> |
| When experimenting with Boost.MPI in Python, don't forget that help is always |
| available via <code class="computeroutput"><span class="identifier">pydoc</span></code>: just |
| pass the name of the module or module entity on the command line (e.g., |
| <code class="computeroutput"><span class="identifier">pydoc</span> <span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">communicator</span></code>) to receive complete reference |
| documentation. When in doubt, try it! |
| </p> |
| </div> |
| <div class="section"> |
| <div class="titlepage"><div><div><h3 class="title"> |
| <a name="mpi.python_user_data"></a>Transmitting User-Defined Data</h3></div></div></div> |
| <p> |
| Boost.MPI can transmit user-defined data in several different ways. Most |
| importantly, it can transmit arbitrary <a href="http://www.python.org" target="_top">Python</a> |
| objects by pickling them at the sender and unpickling them at the receiver, |
| allowing arbitrarily complex Python data structures to interoperate with |
| MPI. |
| </p> |
| <p> |
| Boost.MPI also supports efficient serialization and transmission of C++ objects |
| (that have been exposed to Python) through its C++ interface. Any C++ type |
| that provides (de-)serialization routines that meet the requirements of the |
| Boost.Serialization library is eligible for this optimization, but the type |
| must be registered in advance. To register a C++ type, invoke the C++ function |
| <code class="computeroutput"><a class="link" href="../boost/mpi/python/register_serialized.html" title="Function template register_serialized">register_serialized</a></code>. |
| If your C++ types come from other Python modules (they probably will!), those |
| modules will need to link against the <code class="computeroutput"><span class="identifier">boost_mpi</span></code> |
| and <code class="computeroutput"><span class="identifier">boost_mpi_python</span></code> libraries |
| as described in the <a class="link" href="getting_started.html#mpi.installation" title="Installing and Using Boost.MPI">installation section</a>. |
| Note that you do <span class="bold"><strong>not</strong></span> need to link against |
| the Boost.MPI Python extension module. |
| </p> |
| <p> |
| Finally, Boost.MPI supports separation of the structure of an object from |
| the data it stores, allowing the two pieces to be transmitted separately. |
| This "skeleton/content" mechanism, described in more detail in |
| a later section, is a communication optimization suitable for problems with |
| fixed data structures whose internal data changes frequently. |
| </p> |
| </div> |
| <div class="section"> |
| <div class="titlepage"><div><div><h3 class="title"> |
| <a name="mpi.python_collectives"></a>Collectives</h3></div></div></div> |
| <p> |
| Boost.MPI supports all of the MPI collectives (<code class="computeroutput"><span class="identifier">scatter</span></code>, |
| <code class="computeroutput"><span class="identifier">reduce</span></code>, <code class="computeroutput"><span class="identifier">scan</span></code>, |
| <code class="computeroutput"><span class="identifier">broadcast</span></code>, etc.) for any |
| type of data that can be transmitted with the point-to-point communication |
| operations. For the MPI collectives that require a user-specified operation |
| (e.g., <code class="computeroutput"><span class="identifier">reduce</span></code> and <code class="computeroutput"><span class="identifier">scan</span></code>), the operation can be an arbitrary |
| Python function. For instance, one could concatenate strings with <code class="computeroutput"><span class="identifier">all_reduce</span></code>: |
| </p> |
| <pre class="programlisting"><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">all_reduce</span><span class="special">(</span><span class="identifier">my_string</span><span class="special">,</span> <span class="keyword">lambda</span> <span class="identifier">x</span><span class="special">,</span><span class="identifier">y</span><span class="special">:</span> <span class="identifier">x</span> <span class="special">+</span> <span class="identifier">y</span><span class="special">)</span> |
| </pre> |
| <p> |
| The following module-level functions implement MPI collectives: all_gather |
| Gather the values from all processes. all_reduce Combine the results from |
| all processes. all_to_all Every process sends data to every other process. |
| broadcast Broadcast data from one process to all other processes. gather |
| Gather the values from all processes to the root. reduce Combine the results |
| from all processes to the root. scan Prefix reduction of the values from |
| all processes. scatter Scatter the values stored at the root to all processes. |
| </p> |
| </div> |
| <div class="section"> |
| <div class="titlepage"><div><div><h3 class="title"> |
| <a name="mpi.python_skeleton_content"></a>Skeleton/Content Mechanism</h3></div></div></div> |
| <p> |
| Boost.MPI provides a skeleton/content mechanism that allows the transfer |
| of large data structures to be split into two separate stages, with the skeleton |
| (or, "shape") of the data structure sent first and the content |
| (or, "data") of the data structure sent later, potentially several |
| times, so long as the structure has not changed since the skeleton was transferred. |
| The skeleton/content mechanism can improve performance when the data structure |
| is large and its shape is fixed, because while the skeleton requires serialization |
| (it has an unknown size), the content transfer is fixed-size and can be done |
| without extra copies. |
| </p> |
| <p> |
| To use the skeleton/content mechanism from Python, you must first register |
| the type of your data structure with the skeleton/content mechanism <span class="bold"><strong>from C++</strong></span>. The registration function is <code class="computeroutput"><a class="link" href="../boost/mpi/python/register_skeleton_and_c_id920182.html" title="Function template register_skeleton_and_content">register_skeleton_and_content</a></code> |
| and resides in the <code class="computeroutput"><a class="link" href="reference.html#header.boost.mpi.python_hpp" title="Header <boost/mpi/python.hpp>"><boost/mpi/python.hpp></a></code> |
| header. |
| </p> |
| <p> |
| Once you have registered your C++ data structures, you can extract the skeleton |
| for an instance of that data structure with <code class="computeroutput"><span class="identifier">skeleton</span><span class="special">()</span></code>. The resulting <code class="computeroutput"><span class="identifier">skeleton_proxy</span></code> |
| can be transmitted via the normal send routine, e.g., |
| </p> |
| <pre class="programlisting"><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">skeleton</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">))</span> |
| </pre> |
| <p> |
| <code class="computeroutput"><span class="identifier">skeleton_proxy</span></code> objects can |
| be received on the other end via <code class="computeroutput"><span class="identifier">recv</span><span class="special">()</span></code>, which stores a newly-created instance |
| of your data structure with the same "shape" as the sender in its |
| <code class="computeroutput"><span class="error">"</span><span class="identifier">object</span></code> |
| attribute: |
| </p> |
| <pre class="programlisting"><span class="identifier">shape</span> <span class="special">=</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">)</span> |
| <span class="identifier">my_data_structure</span> <span class="special">=</span> <span class="identifier">shape</span><span class="special">.</span><span class="identifier">object</span> |
| </pre> |
| <p> |
| Once the skeleton has been transmitted, the content (accessed via <code class="computeroutput"><span class="identifier">get_content</span></code>) can be transmitted in much |
| the same way. Note, however, that the receiver also specifies <code class="computeroutput"><span class="identifier">get_content</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">)</span></code> |
| in its call to receive: |
| </p> |
| <pre class="programlisting"><span class="keyword">if</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">rank</span> <span class="special">==</span> <span class="number">0</span><span class="special">:</span> |
| <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">get_content</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">))</span> |
| <span class="keyword">else</span><span class="special">:</span> |
| <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">get_content</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">))</span> |
| </pre> |
| <p> |
| Of course, this transmission of content can occur repeatedly, if the values |
| in the data structure--but not its shape--changes. |
| </p> |
| <p> |
| The skeleton/content mechanism is a structured way to exploit the interaction |
| between custom-built MPI datatypes and <code class="computeroutput"><span class="identifier">MPI_BOTTOM</span></code>, |
| to eliminate extra buffer copies. |
| </p> |
| </div> |
| <div class="section"> |
| <div class="titlepage"><div><div><h3 class="title"> |
| <a name="mpi.python_compatbility"></a>C++/Python MPI Compatibility</h3></div></div></div> |
| <p> |
| Boost.MPI is a C++ library whose facilities have been exposed to Python via |
| the Boost.Python library. Since the Boost.MPI Python bindings are build directly |
| on top of the C++ library, and nearly every feature of C++ library is available |
| in Python, hybrid C++/Python programs using Boost.MPI can interact, e.g., |
| sending a value from Python but receiving that value in C++ (or vice versa). |
| However, doing so requires some care. Because Python objects are dynamically |
| typed, Boost.MPI transfers type information along with the serialized form |
| of the object, so that the object can be received even when its type is not |
| known. This mechanism differs from its C++ counterpart, where the static |
| types of transmitted values are always known. |
| </p> |
| <p> |
| The only way to communicate between the C++ and Python views on Boost.MPI |
| is to traffic entirely in Python objects. For Python, this is the normal |
| state of affairs, so nothing will change. For C++, this means sending and |
| receiving values of type <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">object</span></code>, |
| from the <a href="http://www.boost.org/libs/python/doc" target="_top">Boost.Python</a> |
| library. For instance, say we want to transmit an integer value from Python: |
| </p> |
| <pre class="programlisting"><span class="identifier">comm</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="number">17</span><span class="special">)</span> |
| </pre> |
| <p> |
| In C++, we would receive that value into a Python object and then <code class="computeroutput"><span class="identifier">extract</span></code> an integer value: |
| </p> |
| <pre class="programlisting"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">object</span> <span class="identifier">value</span><span class="special">;</span> |
| <span class="identifier">comm</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">value</span><span class="special">);</span> |
| <span class="keyword">int</span> <span class="identifier">int_value</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">extract</span><span class="special"><</span><span class="keyword">int</span><span class="special">>(</span><span class="identifier">value</span><span class="special">);</span> |
| </pre> |
| <p> |
| In the future, Boost.MPI will be extended to allow improved interoperability |
| with the C++ Boost.MPI and the C MPI bindings. |
| </p> |
| </div> |
| <div class="section"> |
| <div class="titlepage"><div><div><h3 class="title"> |
| <a name="mpi.pythonref"></a>Reference</h3></div></div></div> |
| <p> |
| The Boost.MPI Python module, <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code>, has |
| its own <a href="../boost.mpi.html" target="_top">reference documentation</a>, which |
| is also available using <code class="computeroutput"><span class="identifier">pydoc</span></code> |
| (from the command line) or <code class="computeroutput"><span class="identifier">help</span><span class="special">(</span><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span><span class="special">)</span></code> |
| (from the Python interpreter). |
| </p> |
| </div> |
| </div> |
| <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> |
| <td align="left"></td> |
| <td align="right"><div class="copyright-footer">Copyright © 2005-2007 Douglas Gregor, |
| Matthias Troyer, Trustees of Indiana University<p> |
| Distributed under the Boost Software License, Version 1.0. (See accompanying |
| file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top"> |
| http://www.boost.org/LICENSE_1_0.txt </a>) |
| </p> |
| </div></td> |
| </tr></table> |
| <hr> |
| <div class="spirit-nav"> |
| <a accesskey="p" href="../boost/mpi/timer.html"><img src="../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../mpi.html"><img src="../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="design.html"><img src="../../../doc/src/images/next.png" alt="Next"></a> |
| </div> |
| </body> |
| </html> |
