building c extensions with setuptools that are not tied to python installed on build machine

[Matthew Taylor]

Hello pythonistas,

This is my first message to this mailing list, so if my question is
off-topic, please direct me to the correct mailing list.

I manage the NuPIC [1] open source machine intelligence project, which
is a python project with C extensions. We have a build on Travis-CI
[2] that creates binary wheels on release tags (in git). I'm having a
problem with OS X binaries.

The Travis-CI build machine is OS X 10.9 and uses the default Python
2.7.6 that comes with OS X. When binaries are created on this machine,
they seem to be linked somehow to the default Python, because when a
user running a homebrew-installed python (or Anaconda) attempts to
install the binary, there's an awful problem [3], even when the Python
versions are exactly the same (2.7.6). It seems that some connection
is made at compile time when C extensions are built that link the
binary file to the system Python on the OS X machine building the
binary. This means any users working with the homebrew Python or
Anaconda will get nasty thread errors when the program attempts to run
against two Pythons.

Fatal Python error: PyThreadState_Get: no current thread

Does this make sense to anyone? I'm still a little new to Python in
general (especially binary packaging), and it seems like this would be
a common problem for any projects with C extensions that need broad
binary distribution. Does anyone have any suggestions? Please take a
look at our setup.py file [4] and tell me if we're doing something
egregiously wrong.

[1] https://github.com/numenta/nupic
[2] https://travis-ci.org/numenta/nupic
[3] https://github.com/numenta/nupic/issues/1813
[4] https://github.com/numenta/nupic/blob/master/setup.py

Thanks thanks in advance,
-
Matt Taylor
OS Community Flag-Bearer
Numenta
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: building c extensions with setuptools that are not tied to python installed on build machine

[Matthew Taylor]

Ned, thank you for your insight on this problem. I will take your
advice and do some more digging. You've been very helpful.

Regards,

-
Matt Taylor
OS Community Flag-Bearer
Numenta


On Wed, Feb 11, 2015 at 4:23 PM, Ned Deily  wrote:
> In article
> ,
>  Matthew Taylor  wrote:
>> Does this make sense to anyone? I'm still a little new to Python in
>> general (especially binary packaging), and it seems like this would be
>> a common problem for any projects with C extensions that need broad
>> binary distribution. Does anyone have any suggestions? Please take a
>> look at our setup.py file [4] and tell me if we're doing something
>> egregiously wrong.
>
> Just taking a quick look at your setup.py there shows a quite
> complicated build, including SWIG.  One suggestion: keep in mind that
> it's normal on OS X for the absolute path of shared libraries and
> frameworks to be embedded in the linked binaries, including the C (or
> C++) extension module bundles (.so files) built for Python packages.  If
> any of the .so files or any other binary artifacts (executables, shared
> libraries) created by your package are linked to the Python
> interpreter's shared library, that may be why you are getting a mixture
> of Python instances.  One way to check for this is to use:
>
> otool -L *.so *.dylib
>
> on all of the directories containing linked binaries in your project and
> check for paths like:
> /System/Library/Frameworks/Python.framework
>
> That would be a link to the Apple-supplied system Python.
>
> A link to /Library/Frameworks/Python.framework or some other path would
> be to a third-party Python like from python.org or Homebrew.
>
> Due to differences in how the various Pythons are built and depending on
> what the C or C++ code is doing, it may not be possible to have one
> binary wheel that works with different Python instances of the same
> version.  For many simple projects, it does work.
>
> You *could* also ask on the PythonMac SIG list.
>
> https://mail.python.org/mailman/listinfo/pythonmac-sig
>
> --
>  Ned Deily,
>  n...@acm.org
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list