[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
New submission from Hongxu Jia:
To build python for embedded Linux systems,
(http://www.yoctoproject.org/docs/2.3.1/yocto-project-qs/yocto-project-qs.html)
The 32-bit python's multiprocessing.BoundedSemaphore could not work.
1. Prerequisite
- Build 32bit python on 64bit host, add '-m32' to gcc
- Cross compling on linux platform, not have `-pthread'
In configure.ac, ac_cv_pthread=no while cross compiling
...
2001 AC_MSG_CHECKING(whether $CC accepts -pthread)
2002 AC_CACHE_VAL(ac_cv_pthread,
2003 [ac_save_cc="$CC"
2004 CC="$CC -pthread"
2005 AC_RUN_IFELSE([AC_LANG_SOURCE([[
[snip]
2018 ]])],[ac_cv_pthread=yes],[ac_cv_pthread=no],[ac_cv_pthread=no])
...
— Macro: AC_RUN_IFELSE (input, [action-if-true], [action-if-false],
[action-if-cross-compiling])
2. Reproduce with simplifying the prerequisite
$ git clone https://github.com/python/cpython.git; cd ./cpython
$ ac_cv_pthread=no CC="gcc -m32" ./configure
$ make -j 10
$ ./python -c "import multiprocessing; pool_sema =
multiprocessing.BoundedSemaphore(value=1); pool_sema.acquire();
pool_sema.release()"
Traceback (most recent call last):
File "", line 1, in
ValueError: semaphore or lock released too many times
And the issue also caused invoking put of 'multiprocessing.Queue' hung.
$ ./python -c "import multiprocessing; queue_instance =
multiprocessing.Queue(); queue_instance.put(('install'))"
--
components: Library (Lib)
messages: 300052
nosy: hongxu
priority: normal
severity: normal
status: open
title: multiprocessing.BoundedSemaphore of 32-bit python could not work while
cross compiling on linux platform
type: crash
versions: Python 2.7, Python 3.6, Python 3.7
___
Python tracker
<http://bugs.python.org/issue31171>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Hongxu Jia added the comment: 3. Analysis 1) The multiprocessing invokes named semaphore in C library (sem_open/sem_post/sem_getvalue/sem_unlink in ./Modules/_multiprocessing/semaphore.c) 2) The glibc defines two different sem_getvalue since the following commit. https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/sem_getvalue.c;h=1432cc795ece84d5bf31c7e5cafe01cc1a09d98d;hb=042e1521c794a945edc43b5bfa7e69ad70420524 The `__new_sem_getvalue' is the sem_getvalue@@GLIBC_2.1 which work with named semaphore `sem_open' The `__old_sem_getvalue' is the old version sem_getvalue@GLIBC_2.0 which work with unamed semaphore `sem_init' In 32-bit C library, it provides both of them. $ nm -g /lib/i386-linux-gnu/libpthread-2.23.so df30 T sem_getvalue@GLIBC_2.0 df10 T sem_getvalue@@GLIBC_2.1 3) In multiprocessing, it invokes sem_open, so sem_getvalue@@GLIBC_2.1 is supposed. If `-pthread' or `-lpthread' not passed to gcc, the compiled _multiprocessing dynamic library could not explicitly linked to sem_getvalue@@GLIBC_2.1 $ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so U sem_getvalue U sem_open If `-pthread' or `-lpthread' passed to gcc: $ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so U sem_getvalue@@GLIBC_2.1 U sem_open@@GLIBC_2.1.1 4) On 32-bit OS, the multiprocessing was incorrectly linked to sem_getvalue@GLIBC_2.0 which caused the issue. -- ___ Python tracker <http://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Hongxu Jia added the comment:
4. Solution
For cross compiling, there is no `-pthread', so we should explicitly add
`-lpthread' to build multiprocessing.
Peterson tried to do it in the following commit:
...
commit e711cafab13efc9c1fe6c5cd75826401445eb585
Author: Benjamin Peterson
Date: Wed Jun 11 16:44:04 2008 +
Merged revisions 64104,64117 via svnmerge from
svn+ssh://python...@svn.python.org/python/trunk
...
git show e711cafab13efc9c1fe6c5cd75826401445eb585 -- setup.py
--- a/setup.py
+++ b/setup.py
@@ -1110,6 +1110,56 @@ class PyBuildExt(build_ext):
# _fileio -- supposedly cross platform
exts.append(Extension('_fileio', ['_fileio.c']))
+# Richard Oudkerk's multiprocessing module
+if platform == 'win32': # Windows
+macros = dict()
+libraries = ['ws2_32']
+
+elif platform == 'darwin': # Mac OSX
+macros = dict(
+HAVE_SEM_OPEN=1,
+HAVE_SEM_TIMEDWAIT=0,
+HAVE_FD_TRANSFER=1,
+HAVE_BROKEN_SEM_GETVALUE=1
+)
+libraries = []
+
+elif platform == 'cygwin': # Cygwin
+macros = dict(
+HAVE_SEM_OPEN=1,
+HAVE_SEM_TIMEDWAIT=1,
+HAVE_FD_TRANSFER=0,
+HAVE_BROKEN_SEM_UNLINK=1
+)
+libraries = []
+else: # Linux and other unices
+macros = dict(
+HAVE_SEM_OPEN=1,
+HAVE_SEM_TIMEDWAIT=1,
+HAVE_FD_TRANSFER=1
+)
+libraries = ['rt']
+
+if platform == 'win32':
+multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+ '_multiprocessing/semaphore.c',
+ '_multiprocessing/pipe_connection.c',
+ '_multiprocessing/socket_connection.c',
+ '_multiprocessing/win32_functions.c'
+ ]
+
+else:
+multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+ '_multiprocessing/socket_connection.c'
+ ]
+
+if macros.get('HAVE_SEM_OPEN', False):
+multiprocessing_srcs.append('_multiprocessing/semaphore.c')
+
+exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+ define_macros=list(macros.items()),
+ include_dirs=["Modules/_multiprocessing"]))
+# End multiprocessing
It defined variable `libraries' and assigned it according to
host_platform, but forgot to pass it to Extension.
So we should correct it, and add `-lpthread' for linux.
--
___
Python tracker
<http://bugs.python.org/issue31171>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Changes by Hongxu Jia : -- pull_requests: +3087 ___ Python tracker <http://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: +13264 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: -3087 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- keywords: +patch pull_requests: +9808 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: +9809 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: +9810 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: -9809 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: -9808 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31171] multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Change by Hongxu Jia : -- pull_requests: +9812 ___ Python tracker <https://bugs.python.org/issue31171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
