Extending Python with C: Cannot find MPI library
I am writing some Python code using the Message Passing Interface
(MPI), an API used in parallel computing. There exist a number of
Python implementations of MPI, but apparently they all rely on the
Numeric Python (numpy) package. I need to run my code on a particular
machine made available by my university, which does not have numpy and
will not be getting it.
So I am trying to write my own mini-wrapper for MPI in C to extend
my Python program.
There exists a special C compiler to use when compiling C programs
that use MPI, called mpicc.
Here is what I have tried:
I have written a minimal C program that uses MPI. It works correctly
when compiled with mpicc.
Then I've written Python boilerplate code and a Python script to
compile my C program into a Python-includable module. This does not
work a priori because the Python compilation script uses gcc rather
than mpicc. So I have taken the exact gcc command that the script uses
and replaced "gcc" with "mpicc".
This produces an *.so file that compiles without errors.
Unfortunately, it fails when I try to import it in Python. It can't
find the MPI library.
My MPI code looks like this (plus some boilerplate code):
/* Return the MPI rank of the current process. */
int rank(){
int argc;
char **argv;
int rank, size;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Finalize();
return rank;
}
/* Main. A 'Hello World' function. */
int hello() {
int rankNumber = rank();
printf ("Hello, World. I am process %d.\n", rankNumber);
return rankNumber;
}
My Python program that includes it looks like this:
import ctest
ctest.hello()
My error message is this:
[EMAIL PROTECTED] Opgave03]$ mpiexec -n 1 python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in
import ctest
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
[EMAIL PROTECTED] Opgave03]$
Can anyone suggest anything? Can I get MPI to work in Python?
Last time I asked a similar question, someone recommended that I
check out Cython instead of C. Do MPI bindings for Cython exist?
Thanks in advance.
- Claus Appel
--
http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python with C: Cannot find MPI library
On Jun 19, 2:10 pm, Jeroen Ruigrok van der Werven <[EMAIL PROTECTED] nomine.org> wrote: > -On [20080619 13:53], Spectrum ([EMAIL PROTECTED]) wrote: > > > ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined > >symbol: ompi_mpi_comm_world > > [EMAIL PROTECTED] Opgave03]$ > > > Can anyone suggest anything? Can I get MPI to work in Python? > > Sounds like a typical case of not specifying any -L and/or -l options. In > this case the ctest.so module has an undefined reference to the symbol > ompi_mpi_comm_world and cannot resolve it. So either it was not linked > properly or the loader cannot find the required library. > > Apply nm -D and ldd to the .so and see what ails. Thanks for the reply. It is very likely that I am not compiling properly. I am a sucky C programmer. The commands you recommended give the following output: [EMAIL PROTECTED] Opgave03]$ nm -D ctest.so U MPI_Comm_rank U MPI_Finalize U MPI_Init w _Jv_RegisterClasses 18cc A __bss_start w __cxa_finalize w __gmon_start__ U __printf_chk 18cc A _edata 18d4 A _end 06c4 T _fini 0414 T _init 0600 T hello 0650 T main U ompi_mpi_comm_world 05b0 T rank [EMAIL PROTECTED] Opgave03]$ [EMAIL PROTECTED] Opgave03]$ [EMAIL PROTECTED] Opgave03]$ ldd ctest.so linux-gate.so.1 => (0x0011) libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x00113000) libpthread.so.0 => /lib/libpthread.so.0 (0x0025a000) libc.so.6 => /lib/libc.so.6 (0x00342000) libdl.so.2 => /lib/libdl.so.2 (0x00273000) libutil.so.1 => /lib/libutil.so.1 (0x00278000) libm.so.6 => /lib/libm.so.6 (0x0027c000) /lib/ld-linux.so.2 (0x00325000) [EMAIL PROTECTED] Opgave03]$ The same commands, when applied to the mpicc-compiled, correctly- functioning C program (in this case just named 'a.out') give this: [EMAIL PROTECTED] Opgave03]$ ./a.out Hello, World. I am process 0. [EMAIL PROTECTED] Opgave03]$ nm -D a.out U MPI_Comm_rank U MPI_Finalize U MPI_Init 08048818 R _IO_stdin_used w _Jv_RegisterClasses 080499fc A __bss_start 080499f8 D __data_start w __gmon_start__ 08048750 T __libc_csu_fini 08048760 T __libc_csu_init U __libc_start_main 080499fc A _edata 08049b10 A _end 080487f8 T _fini 08048814 R _fp_hw 08048560 T _init 080485f0 T _start 080499f8 W data_start 080486f9 T hello 0804871f T main 08049a00 B ompi_mpi_comm_world U printf 080486c4 T rank [EMAIL PROTECTED] Opgave03]$ ldd a.out linux-gate.so.1 => (0x0011) libmpi.so.0 => /usr/lib/openmpi/1.2.4-gcc/libmpi.so.0 (0x0042f000) libopen-rte.so.0 => /usr/lib/openmpi/1.2.4-gcc/libopen- rte.so.0 (0x003d4000) libopen-pal.so.0 => /usr/lib/openmpi/1.2.4-gcc/libopen- pal.so.0 (0x00344000) libdl.so.2 => /lib/libdl.so.2 (0x00df) libnsl.so.1 => /lib/libnsl.so.1 (0x003b9000) libutil.so.1 => /lib/libutil.so.1 (0x004d8000) libm.so.6 => /lib/libm.so.6 (0x0025c000) libpthread.so.0 => /lib/libpthread.so.0 (0x00287000) libc.so.6 => /lib/libc.so.6 (0x004dc000) /lib/ld-linux.so.2 (0x00325000) [EMAIL PROTECTED] Opgave03]$ I don't know how to interpret these, so can I get some more advice? Thanks in advance. - Claus Appel -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python with C: Cannot find MPI library
On Jun 19, 4:39 pm, Jeroen Ruigrok van der Werven <[EMAIL PROTECTED] nomine.org> wrote: > -On [20080619 16:21], Spectrum ([EMAIL PROTECTED]) wrote: > > > libmpi.so.0 => /usr/lib/openmpi/1.2.4-gcc/libmpi.so.0 > >(0x0042f000) > > libopen-rte.so.0 => /usr/lib/openmpi/1.2.4-gcc/libopen- > >rte.so.0 (0x003d4000) > > libopen-pal.so.0 => /usr/lib/openmpi/1.2.4-gcc/libopen- > >pal.so.0 (0x00344000) > > These libraries are what your binaries (.so and such) needs to link against > as well. I am not sure if you have to add a flag (-mpi) to your compiler or > if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte > -lopen-pal to the incantations. Thanks for the reply. I did some more experimentation. The compiling commands I use are: mpicc -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,- D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer- size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables - D_GNU_SOURCE -fPIC -fPIC -I/usr/include/python2.5 -c ctest.c -o ctest.o gcc -pthread -shared build/temp.linux-i686-2.5/ctest.o -L/usr/lib - lpython2.5 -o ctest.so It turns out that changing the second "gcc" to "mpicc" causes the module to no longer be includable. Python will give this error message: [EMAIL PROTECTED] Opgave03]$ python ctest.py Traceback (most recent call last): File "ctest.py", line 1, in import ctest ImportError: dynamic module does not define init function (initctest) [EMAIL PROTECTED] Opgave03]$ The same happens if I try to add "-L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte -lopen-pal" as options to the second compilation command, as you recommended. Looks like MPI doesn't like Python... -- http://mail.python.org/mailman/listinfo/python-list
