On 10/11/2012 2:40 PM, Marien Zwart wrote:
I'm going to do something potentially rude and comment on this without
having read the entire thread.
On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <g...@malth.us> wrote:
Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!
Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.
However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.
The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).
Thanks a ton for this info, that explains a lot. Actually, got this
notion testing outside of Gentoo, building from a vanilla python 3.3.0
tarball on a Fedora box. I wanted to see what happens in a completely
"Gentoo-free" environment so as to get some kind of baseline
understanding of how upstream is propagating stuff through the Makefiles
to setup.py.
Probably, I screwed up ./configure (I tried to approximate Gentoo's
arguments), and requested a non-shlib build. Guess it would have been
clearer if I'd mentioned that in my post -- sorry if I worried you :)
I do, of course, intend to try it on my main Gentoo ~amd64 -- if I ever
see it fail to do as you describe, I'll let you know, Marien.
My i686 prefix is up to date and I can confirm that it is linking
against libpython there.
But... wow. In my almost-vanilla i686 cross-prefix (hosted by my
only-moderately-rice multilib ~amd64 workstation), in
=dev-lang/python-2.7.3-r2, the modules are linked like so:
i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-fno-strict-aliasing \
-O2 -march=i686 -pipe -m32 \
-fwrapv -DNDEBUG \
-I. -IInclude -I./Include \
-I${EPREFIX}/usr/include \
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/mathmodule.o
\
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/_math.o
\
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-lm -lpython2.7 \
-o build/lib.linux-i686-2.7/math.so
Which is just shameful. In this case, I'm pretty sure all those
-L{${EPREFIX},}{/usr,}/$(libdir) clauses are not coming from me:
[PREFIX] $ emerge --info --verbose | grep /lib
sys-devel/libtool: 2.4.2::gentoo_prefix
COLLISION_IGNORE="/lib/modules/* *.py[co]"
PKG_CONFIG_PATH="${EPREFIX}/usr/lib/pkgconfig:$( :;
)${EPREFIX}/usr/share/pkgconfig"
PORTAGE_BIN_PATH="${EPREFIX}/usr/lib/portage/bin"
PORTAGE_PYM_PATH="${EPREFIX}/usr/lib/portage/pym"
UNINSTALL_IGNORE="/lib/modules/*"
(snippets edited for readability obv).
Regarding the gcc-as-linker invocation above:
First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).
Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).
Thirdly, prefix x86-linux also clearly suffers from the original problem
I was seeing in cygwin, and which inspired this thread (putting -L.
after $(libdir) paths and therefore linking against the wrong
libpython.so). I never suspected this applied to every prefix python
ebuild, but that's what the above seems to suggest... not cool.
Plus, it repeats itself like a stuttering Makefool -- but that's pretty
standard and only ever seems to bother me :)
Perhaps, this is a consequence of mine being a cross-prefix, somehow.
That's the only thing significantly off-the-beaten-path thing about it.
Even if so, I'd like amd64->i686 cross-prefixes to work and don't see
why they shouldn't.
The only good news I can report is that changing
append-ldflags "-L."
to
export LDFLAGS="-L. ${LDFLAGS}"
does seem to fix the third issue, above.
Looks like, at least for prefix, it's going to take more than the quick
configure patch I was hoping for, if we're to make things fully right.
We'll see soon enough about ~amd64 linux, and I'm spinning up a
freebsd90 vm to kick around as well.
-gmt
