On 11/05/21 at 14:48 +0300, Vassilis Virvilis wrote:
> I believe the problem is that mpirun is built with the internal pmix
> library when there is external available.
>
> bill@odin:~/src/openmpi-4.1.0$ dpkg -l '*pmix*' | grep ^ii
> ii libpmix-dev:amd64 4.0.0-4 amd64 Development files for the
> PMI Exascale library
> ii libpmix2:amd64 4.0.0-4 amd64 Process Management
> Interface (Exascale) library
>
> mpirun is not linked to the external libpmix2
>
> bill@odin:~/src/openmpi-4.1.0$ ldd /usr/bin/mpirun.openmpi
> linux-vdso.so.1 (0x00007fffa1153000)
> libopen-rte.so.40 => /usr/lib/x86_64-linux-gnu/libopen-rte.so.40
> (0x00007f3cdf657000)
> libopen-pal.so.40 => /usr/lib/x86_64-linux-gnu/libopen-pal.so.40
> (0x00007f3cdf5a3000)
> libevent_core-2.1.so.7 =>
> /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f3cdf569000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3cdf3a4000)
> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3cdf387000)
> libhwloc.so.15 => /usr/lib/x86_64-linux-gnu/libhwloc.so.15
> (0x00007f3cdf32e000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f3cdf30a000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3cdf304000)
> libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1
> (0x00007f3cdf2ff000)
> libevent_pthreads-2.1.so.7 =>
> /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f3cdf2fa000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f3cdf72c000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3cdf1b6000)
> libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1
> (0x00007f3cdf18e000)
That's because it is loaded dynamically.
mca_pmix_ext3x.so is linked to libpmix.so.2:
# ldd /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pmix_ext3x.so
linux-vdso.so.1 (0x00007ffeba72a000)
libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40
(0x00007fae76f77000)
libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007fae76e2c000)
libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
(0x00007fae76df2000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fae76dd0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fae76c0b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fae76c05000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fae76bfe000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15
(0x00007fae76ba5000)
libevent_pthreads-2.1.so.7 =>
/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fae76ba0000)
/lib64/ld-linux-x86-64.so.2 (0x00007fae77057000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fae76a5c000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fae76a34000)
> There is this https://github.com/open-mpi/ompi/issues/8335 but it looks
> applied in Debian.
>
> I am trying to find the config.log to understand why it prefers the
> internal version of the pmix library but so far I can't tell.
>
> Can you give me a hint where the config.log lives after a successful build?
It's not available. I rebuilt it locally, and got:
configure:13919: checking if user requested internal PMIx
support(/usr/lib/x86_64-linux-gnu/pmix2)
configure:13932: result: no
configure:13985: checking for pmix.h in /usr/lib/x86_64-linux-gnu/pmix2
configure:13993: result: not found
configure:13995: checking for pmix.h in /usr/lib/x86_64-linux-gnu/pmix2/include
configure:13999: result: found
configure:14048: checking libpmix.* in /usr/lib/x86_64-linux-gnu/pmix2/lib64
configure:14056: result: not found
configure:14058: checking libpmix.* in /usr/lib/x86_64-linux-gnu/pmix2/lib
configure:14062: result: found
configure:14081: checking PMIx version
configure:14091: result: version file found
configure:14099: checking version 4x
configure:14117: gcc -E -I/usr/lib/x86_64-linux-gnu/pmix2/include -Wdate-time
-D_FORTIFY_SOURCE=2 conftest.c
configure:14117: $? = 0
configure:14118: result: found
configure:14305: checking PMIx version to be used
configure:14308: result: external(4x)
(which looks OK)
Lucas
