[issue38884] __import__ is not thread-safe on Python 3
Ryan Petrello added the comment: I believe I'm also encountering some version of this bug while importing code from the kombu library within a multi-threaded context. For what it's worth, I'm able to reproduce the reported deadlock (https://bugs.python.org/file48737/issue38884.zip) using python3.8 on RHEL8 and CentOS 8 builds. More details about what I'm encountering here: https://github.com/ansible/awx/issues/5617 https://github.com/ansible/awx/issues/5617#issuecomment-591618205 My intended workaround for now is to just not use a thread (we'll see if it helps): https://github.com/ansible/awx/pull/6093 -- nosy: +ryan.petrello ___ Python tracker <https://bugs.python.org/issue38884> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37429] Python hangs on fork when a logger is in use in a background thread
New submission from Ryan Petrello : It's possible to cause Python child processes to encountered a deadlock and hang on fork by: 1) Spawning one or more threads in a parent process 2) Have one of those threads emit log lines 3) Have the main thread use os.fork() 4) In the child, log immediately The attached Python script can be used to reproduce this issue in Python 2.7.15 and Python 3.6.5 -- components: Library (Lib) files: hang.py messages: 346732 nosy: ryan.petrello priority: normal severity: normal status: open title: Python hangs on fork when a logger is in use in a background thread versions: Python 2.7, Python 3.6 Added file: https://bugs.python.org/file48441/hang.py ___ Python tracker <https://bugs.python.org/issue37429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37429] Python hangs on fork when a logger is in use in a background thread
Ryan Petrello added the comment: This issue seems _very_ similar to https://bugs.python.org/issue6721, but I wasn't able to encounter the exact stack I've been seeing on my end. Specifically, when I run the reproducer script in this issue (hang.py) for a few minutes, hung child processes start to build up. -- ___ Python tracker <https://bugs.python.org/issue37429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37429] Python hangs on fork when a logger is in use in a background thread
Ryan Petrello added the comment:
Here's the stack of one the hung child processes under py2.7:
(gdb) py-bt
#4 Waiting for a lock (e.g. GIL)
#5 Waiting for a lock (e.g. GIL)
#7 Frame 0x7f346a925430, for file /usr/lib64/python2.7/Queue.py, line 118, in
put (self=, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False,
_Condition__lock=, acquire=,
_Condition__waiters=[], release=) at remote 0x7f3467df4150>, mutex=, not_full=<_Condition(_Verbose__verbose=False,
_Condition__lock=, acquire=,
_Condition__waiters=[], release=) at remote 0x7f3467df4110>,
not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=, acquire=, _shutdown=False, _max_workers=2,
_thread_name_prefix='Thread---Type to continue, or q to
quit---
PoolExecutor-0', _threads=set([<...>, <...>]), _work_queue=<...>) at remote
0x7f3467df0f90>, fn=,
args=(, process=1155,
processName='Process-74', args=(), module='hang', filename='hang.py',
levelno=20, exc_text=None, pathname='/tmp/hang.py', lineno=25, msg='exiting',
exc_info=None, funcName='exit', relativeCreated=,
levelname='INFO', msecs=) at remote
0x7f346a972490>,), kwargs={}, f=, _shutdown=False, _max_workers=2,
_thread_name_prefix='ThreadPoolExecutor-0', _threads=set([<...>, <...>]),
_work_queue=<...>) at remote 0x7f3467df0f90>) at remote 0x7f3467df0f50>,
record=, process=1155,
processName='Process-74', args=(), module='hang', filename='hang.py',
levelno=20, exc_text=None, pathname='/tmp/hang.py', lineno=25, msg='exiting',
exc_info=None, funcName='exit', relativeCreated= to continue,
or q to quit---
loat at remote 0x1a7d568>, levelname='INFO', msecs=)
at remote 0x7f346a972490>)
self.executor.submit(self.send, record)
#19 Frame 0x7f346a925240, for file /usr/lib64/python2.7/logging/__init__.py,
line 1318, in callHandlers (self=, _shutdown=False, _max_workers=2,
_thread_name_prefix='ThreadPoolExecutor-0', _threads=set([<...>, <...>]),
_work_queue=<...>) at remote 0x7f3467df0f90>) at remote 0x7f3467df0f50>],
level=10, disabled=0, propagate=1, filters=[]) at remote 0x7f3467de1a90>,
record=, process=1155,
processName='Process-74', args=(), module='hang', filename='hang.py',
levelno=20, exc_text=None, pathname='/tmp/hang.py', lineno=25, msg='exiting',
exc_info=None, funcName='exit', relativeCreated=,
levelname='INFO', msecs=) at remote 0x7f346a972490>,
c=<...>, found=1, hdlr=<...>)
hdlr.handle(record)
#22 Frame 0x7f3467df13e0, for file /usr/lib64/python2.7/logging/__init__.py,
line 1278, in handle (self=, _shut---Type to continue, or q to
quit---
down=False, _max_workers=2, _thread_name_prefix='ThreadPoolExecutor-0',
_threads=set([<...>, <...>]), _work_queue=<...>) at remote 0x7f3467df0f90>) at
remote 0x7f3467df0f50>], level=10, disabled=0, propagate=1, filters=[]) at
remote 0x7f3467de1a90>, record=,
process=1155, processName='Process-74', args=(), module='hang',
filename='hang.py', levelno=20, exc_text=None, pathname='/tmp/hang.py',
lineno=25, msg='exiting', exc_info=None, funcName='exit',
relativeCreated=, levelname='INFO', msecs=) at remote 0x7f346a972490>)
self.callHandlers(record)
#25 Frame 0x1ae60e0, for file /usr/lib64/python2.7/logging/__init__.py, line
1268, in _log (self=, _shutdown=False, _max_workers=2,
_thread_name_prefix='ThreadPoolExecutor-0', _threads=set([<...>, <...>]),
_work_queue=<...>) at remote 0x7f3467df0f90>) at remote 0x7f3467df0f50>],
level=10, disabled=0, propagate=1, filters=[]) at remote 0x7f3467de1a90>,
level=20, msg='exiting', args=(), exc_info=None, extra=None, fn='/tmp/hang.py',
lno=25, func='exit', record=, process=1155,
processName='Process-74', args=(...), ---Type to continue, or q
to quit---
Here's the similar stack under py3.6:
(gdb) bt
#0 0x7f208166cadb in futex_abstimed_wait (cancel=true, private=,
abstime=0x0, expected=0, futex=0x88bbf0)
at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:43
#1 do_futex_wait (sem=sem@entry=0x88bbf0, abstime=0x0)
at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:223
#2 0x7f208166cb6f in __new_sem_wait_slow (sem=0x88bbf0, abstime=0x0)
at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:292
#3 0x7f208166cc0b in __new_sem_wait (sem=)
at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:28
#4 0x7f2081a05a80 in PyThread_acquire_lock_timed (lock=0x88bbf0,
microseconds=microseconds@entry=-1, intr_flag=intr_flag@entry=0)
at /usr/src/debug/Python-3.6.8/Python/thread
[issue37429] Python hangs on fork when a logger is in use in a background thread
Ryan Petrello added the comment: Also, for what it's worth, I'm *not* able to reproduce this hang in Python 3.7.0. -- ___ Python tracker <https://bugs.python.org/issue37429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37429] Python hangs on fork when a logger is in use in a background thread
Ryan Petrello added the comment: Actually, I think I *am* seeing this on 3.7: (gdb) bt #0 0x7f130d530adb in futex_abstimed_wait (cancel=true, private=, abstime=0x0, expected=0, futex=0x142c6d0) at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:43 #1 do_futex_wait (sem=sem@entry=0x142c6d0, abstime=0x0) at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:223 #2 0x7f130d530b6f in __new_sem_wait_slow (sem=0x142c6d0, abstime=0x0) at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:292 #3 0x7f130d530c0b in __new_sem_wait (sem=) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:28 #4 0x00535baa in PyThread_acquire_lock_timed (lock=0x142c6d0, microseconds=microseconds@entry=-1, intr_flag=intr_flag@entry=0) at Python/thread_pthread.h:340 #5 0x00535c24 in PyThread_acquire_lock (lock=, waitflag=waitflag@entry=1) at Python/thread_pthread.h:576 #6 0x0058949a in _enter_buffered_busy (self=0x7f130ca0fbf8) at ./Modules/_io/bufferedio.c:282 #7 buffered_flush (self=0x7f130ca0fbf8, args=) at ./Modules/_io/bufferedio.c:835 #8 0x0043aae8 in _PyMethodDef_RawFastCallDict (kwargs=, nargs=0, args=, self=0x7f130ca0fbf8, method=0x8eeda0 ) at Objects/call.c:482 #9 _PyCFunction_FastCallDict (func=0x7f130aeb3438, args=, nargs=0, kwargs=) at Objects/call.c:582 -- versions: +Python 3.7 ___ Python tracker <https://bugs.python.org/issue37429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24303] OSError 17 due to _multiprocessing/semaphore.c assuming a one-to-one Pid -> process mapping.
Ryan Petrello added the comment: Any chance this patch was every applied to Python3? It looks to me like 3.6 has the old code: https://github.com/python/cpython/blob/3.6/Modules/_multiprocessing/semaphore.c#L448 -- nosy: +ryan.petrello ___ Python tracker <https://bugs.python.org/issue24303> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Changes by Ryan Petrello : Added file: http://bugs.python.org/file43087/signature-from-callable-skip-bound-arg.patch ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Changes by Ryan Petrello : Added file: http://bugs.python.org/file43089/signature-from-callable-skip-bound-arg.patch ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Changes by Ryan Petrello : Added file: http://bugs.python.org/file43090/signature-from-callable-skip-bound-arg.patch ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Ryan Petrello added the comment: Yury/Nick, Any word on this? I know it's minor, but I'd love to see this make it into Python3.6. -- ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Ryan Petrello added the comment: Nick, My use case is an issue of backwards compatibility and multiple Python version support for a library that makes prolific use of the legacy argspec (args, varargs, varkw, defaults) namedtuple, *including* the bound self argument behavior. argspec and signature are quite different, and supporting a Py26-Py36 codebase that handles both simultaneously seemed like quite a burden, so I opted to write a compatibility shim that returned an Argspec-like object for all versions of Python; this seemed the simplest approach to bridge the gap in a codebase that supports Python 2.6 through 3.6, and it's the approach that I've seen other major libraries (like Django) take: https://github.com/pecan/pecan/pull/61 I'm using a similar approach to Python 3.5's getfullargspec() implementation: https://github.com/python/cpython/blob/3.5/Lib/inspect.py#L1069 ...but I don't have a long-term public API for doing it (so I have to rely on the private inspect._signature_from_callable call, which seems icky). -- ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Ryan Petrello added the comment: Nick, My main reasoning for not using it is that it's marked as deprecated in the docstring, and I want to avoid relying on it if disappears in the future :) Warnings or not, the shim that I wrote doesn't use any deprecated code, so that's why I took that approach. -- ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27172] Add skip_bound_arg argument to inspect.Signature.from_callable()
Ryan Petrello added the comment: Nick, That seems reasonable to me :) I've updated my library to just use inspect.getfullargspec for Py3. Thanks for taking the time to walk through this with me! -- ___ Python tracker <http://bugs.python.org/issue27172> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28466] SIGALRM fails to interrupt time.sleep() call on Python 3.5
Changes by Ryan Petrello : -- title: SIGALRM fails to interrupt time.sleep() call on Python 3.6 -> SIGALRM fails to interrupt time.sleep() call on Python 3.5 ___ Python tracker <http://bugs.python.org/issue28466> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28466] SIGALRM fails to interrupt time.sleep() call on Python 3.6
New submission from Ryan Petrello:
I may have found a bug in SIGALRM handling in Python3.5. I've not been able to
reproduce the same issue in Python2.7 or 3.4. Here's a simple example that
illustrates the issue (which I'm able to reproduce on OS X 10.11.3 El Capitan
and Ubuntu 14.04):
$ python2 --version; python3.4 --version; python3.5 --version
Python 2.7.11
Python 3.4.4
Python 3.5.1
$ cat alarm.py
import signal, time
def handler(signum, frame):
print('Signal handler called with signal %s' % signum)
# Set the signal handler and a 1-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(1)
# We should not actually sleep for 10 seconds
time.sleep(10)
signal.alarm(0)
$ time python2 alarm.py
Signal handler called with signal 14
python2 alarm.py 0.04s user 0.02s system 5% cpu 1.075 total
$ time python3.4 alarm.py
Signal handler called with signal 14
python3.4 alarm.py 0.07s user 0.01s system 7% cpu 1.092 total
$ time python3.5 alarm.py
Signal handler called with signal 14
python3.5 alarm.py 0.09s user 0.02s system 1% cpu 10.115 total
Note that when run under python3.5, the program does not exit until 10 seconds
have passed.
--
components: Library (Lib)
messages: 278835
nosy: ryan.petrello
priority: normal
severity: normal
status: open
title: SIGALRM fails to interrupt time.sleep() call on Python 3.6
versions: Python 3.5
___
Python tracker
<http://bugs.python.org/issue28466>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
