[issue37961] Tracemalloc traces do not include original stack trace length
New submission from Julien Danjou : When using the tracemalloc module, the maximum number of frames that are captured is specified at startup via a value to the `start` method. However, if the number of frames is truncated, there's no way to know the original length of the stack traces. -- components: Interpreter Core messages: 350616 nosy: jd priority: normal severity: normal status: open title: Tracemalloc traces do not include original stack trace length versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue37961> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37961] Tracemalloc traces do not include original stack trace length
Change by Julien Danjou : -- keywords: +patch pull_requests: +15222 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15545 ___ Python tracker <https://bugs.python.org/issue37961> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37961] Tracemalloc traces do not include original stack trace length
Julien Danjou added the comment: That's a good question. Considering that Py_DEFAULT_RECURSION_LIMIT is 1000, we could probably settle on 2 bytes by using an uint16_t which ought to be enough unless people regularly trace Python stack traces bigger that are bigger than 2^16. -- ___ Python tracker <https://bugs.python.org/issue37961> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37961] Tracemalloc traces do not include original stack trace length
Julien Danjou added the comment: > In fact, I didn't check: maybe traceback_t size is not change by your PR, > because of memory aligment and padding. It is changed. The current size is should be determined by sizeof(Py_uhash_t + int + int) which is 4 + 4 + 4, so 12 aligned/padded. Using a short will add 2 bytes and potentially some space for 2 more bytes in the future. :) -- ___ Python tracker <https://bugs.python.org/issue37961> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1021318] PyThreadState_Next not thread safe
Julien Danjou added the comment: This is actually true and it's quite easy to produce a bug that triggers a segmentation fault when iterating over a large list of threads create dynamically. That makes PyThreadState_Next barely usable for any program that has a few threads. The head mutex needs to be locked to be able to iterate safely over the list of thread. -- nosy: +jd title: PyThreadState_Next not thread safe? -> PyThreadState_Next not thread safe versions: +Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue1021318> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1021318] PyThreadState_Next not thread safe
Julien Danjou added the comment: It'd be great if you could reopen it. I'm interested in fixing it properly. It's impossible to "solve" in Python 2 since the head mutex is not accessible (it's static in pystate.c) but this is doable with Python 3 (yay). We'd simply need to provide a new API to lock/unlock the interpreter's mutex. How hard to you need the script to reproduce? I can try to spend some time to extract some of my code to have one if that's really needed (I'm using Cython to generate some of the C code). -- ___ Python tracker <https://bugs.python.org/issue1021318> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1021318] PyThreadState_Next not thread safe
Julien Danjou added the comment: Yes, that's the one 👍 -- ___ Python tracker <https://bugs.python.org/issue1021318> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1021318] PyThreadState_Next not thread safe
Julien Danjou added the comment: Victor, do you have an idea of how you would like this mutex to be exposed? I see it only recently changed in Python 3.7 to be part of a structure, though this structure is not public AFAIU. Is adding 2 new lock/unlock C functions good enough? -- ___ Python tracker <https://bugs.python.org/issue1021318> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39164] PyErr_GetExcInfo does not allow to retrieve for an arbitrary thread
Julien Danjou added the comment: Relevant python-dev thread: https://mail.python.org/archives/list/python-...@python.org/thread/QVMFP76R35SXUIM2WPPVPV5XCVMKPDEB/#TLP6MWNML4PRKFFGXHCKNEUMN6UIQ4MT -- ___ Python tracker <https://bugs.python.org/issue39164> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39164] PyErr_GetExcInfo does not allow to retrieve for an arbitrary thread
New submission from Julien Danjou : PyErr_GetExcInfo does not allow to retrieve exception information for an arbitrary thread. As it calls `_PyThreadState_GET` itself, it's impossible to get exception information for a different thread. -- components: C API messages: 359029 nosy: jd priority: normal severity: normal status: open title: PyErr_GetExcInfo does not allow to retrieve for an arbitrary thread type: enhancement versions: Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue39164> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39164] PyErr_GetExcInfo does not allow to retrieve for an arbitrary thread
Change by Julien Danjou : -- keywords: +patch pull_requests: +17188 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17752 ___ Python tracker <https://bugs.python.org/issue39164> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39599] ABI breakage between 3.7.4 and 3.7.5
New submission from Julien Danjou : As I've reported originally on the python-dev list, there seems to be an ABI breakage between 3.7.4 and 3.7.5. https://mail.python.org/archives/list/python-...@python.org/message/J2FGZPS5PS7473TONJTPAVSNXRGV3TFL/ The culprit commit is https://github.com/python/cpython/commit/8766cb74e186d3820db0a855ccd780d6d84461f7 This happens on a custom C module (built via Cython) when using including with -DPy_BUILD_CORE. I'm not sure it'd happen otherwise. I've tried to provide a minimal use case, but since it seems to be a memory overflow, the backtrace does not make any sense and it's hard to reproduce without the orignal code. -- components: C API messages: 361689 nosy: jd priority: normal severity: normal status: open title: ABI breakage between 3.7.4 and 3.7.5 versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue39599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39599] ABI breakage between Python 3.7.4 and 3.7.5: change in PyGC_Head structure
Julien Danjou added the comment: > If you want to ship wheel package, another compromise would be to require > Python 3.7.5 or newer, and check early the Python version. Yes, this is what I do now to avoid getting a segmentation fault. I'm fine with closing this as wontfix or reject, your call. As you said, if it's not part of the stable ABI it makes sense. The sad part if that my code does not access PyGC_Head at all. It's just a side effect when loading the code that makes it crash. :( -- ___ Python tracker <https://bugs.python.org/issue39599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39599] ABI breakage between Python 3.7.4 and 3.7.5: change in PyGC_Head structure
Julien Danjou added the comment: The problem is that AFAIK the wheel system does not allow you to specify and provide a wheel for each minor Python version. For references, the code that segfaults is: https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/profile/collector/stack.pyx -- ___ Python tracker <https://bugs.python.org/issue39599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39599] ABI breakage between Python 3.7.4 and 3.7.5: change in PyGC_Head structure
Julien Danjou added the comment: Right, though if you read the code I linked, the PyGC_Head size is never used. AFAICS the Cython code does not access it either (at least not directly). The code segfaults at import time. I'm not even sure it's possible and the problem isn't in something in CPython itself. -- ___ Python tracker <https://bugs.python.org/issue39599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40545] Expose _PyErr_GetTopmostException
New submission from Julien Danjou : The function _PyErr_GetTopmostException() is not exported and is useful to get access to exceptions info from a PyThreadState. -- components: C API messages: 368327 nosy: jd priority: normal severity: normal status: open title: Expose _PyErr_GetTopmostException type: enhancement versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40545> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40545] Expose _PyErr_GetTopmostException
Change by Julien Danjou : -- keywords: +patch pull_requests: +19292 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19977 ___ Python tracker <https://bugs.python.org/issue40545> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40545] Expose _PyErr_GetTopmostException
Julien Danjou added the comment: I would definitely love to have a public API for this. Would `PyErr_GetTopmostException` be a good candidate for this? (I can open a different bug report if needed) -- ___ Python tracker <https://bugs.python.org/issue40545> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40545] Expose _PyErr_GetTopmostException
Julien Danjou added the comment: PyErr_GetExcInfo() does not allow to specify a PyThreadState as argument. -- ___ Python tracker <https://bugs.python.org/issue40545> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40545] Expose _PyErr_GetTopmostException
Julien Danjou added the comment: I confirm what Victor wrote. :) -- ___ Python tracker <https://bugs.python.org/issue40545> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42647] Unable to use concurrent.futures in atexit hook
New submission from Julien Danjou :
Python 3.9 introduced a regression with concurrent.futures. The following
program works fine on Python < 3.8 but raises an error on 3.9:
```
import atexit
import concurrent.futures
def spawn():
with concurrent.futures.ThreadPoolExecutor() as t:
pass
atexit.register(spawn)
```
```
$ python3.9 rep.py
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 790, in exec_module
File "", line 228, in _call_with_frames_removed
File
"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py",
line 37, in
threading._register_atexit(_python_exit)
File
"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py",
line 1370, in _register_atexit
raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown
```
--
messages: 383058
nosy: jd
priority: normal
severity: normal
status: open
title: Unable to use concurrent.futures in atexit hook
type: crash
versions: Python 3.9
___
Python tracker
<https://bugs.python.org/issue42647>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42647] Unable to use concurrent.futures in atexit hook
Julien Danjou added the comment:
> Is there a real-world situation where it's specifically necessary or even
> beneficial to utilize ThreadPoolExecutor at this point after thread
> finalization rather than earlier in the program? Not that it doesn't exist,
> but to me it intuitively seems very odd to utilize an executor within an
> atexit hook, which are intended to just be resource finalization/cleanup
> functions called at interpreter shutdown. Assuming there is a genuine use
> case I'm not seeing, it may be worth weighing against the decision to convert
> the executors to not use daemon threads, as I presently don't think there's a
> way to (safely) allow that behavior without reverting back to using daemon
> threads.
To put that in perspective, here is the original issue that trigged this bug
for me:
Traceback (most recent call last):
File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush
exp.export(events, start, self._last_export)
File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export
self._upload(client, self.endpoint_path, body, headers)
File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload
self._retry_upload(self._upload_once, client, path, body, headers)
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py",
line 423, in __call__
do = self.iter(retry_state=retry_state)
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py",
line 360, in iter
return fut.result()
File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py",
line 433, in result
return self.__get_result()
File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py",
line 389, in __get_result
raise self._exception
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py",
line 426, in __call__
result = fn(*args, **kwargs)
File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in
_upload_once
client.request("POST", path, body=body, headers=headers)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255,
in request
self._send_request(method, url, body, headers, encode_chunked)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301,
in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250,
in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010,
in _send_output
self.send(msg)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in
send
self.connect()
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in
connect
self.sock = self._create_connection(
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py",
line 88, in create_connection
addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM))
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py",
line 247, in getaddrinfo
addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto,
flags)
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py",
line 841, in _get_resolver
self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py",
line 39, in __init__
self.pool = hub.threadpool
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py",
line 865, in _get_threadpool
self._threadpool = self.threadpool_class(self.threadpool_size, hub=self)
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py",
line 860, in threadpool_class
return GEVENT_CONFIG.threadpool
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
line 50, in getter
return self.settings[setting_name].get()
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
line 146, in get
self.value = self.validate(self._default())
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
line 248, in validate
return self._import_one_of([self.shortname_map.get(x, x) for x in value])
File
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages
[issue41435] Allow to retrieve ongoing exception handled by every threads
New submission from Julien Danjou : In order to do statistical profiling on raised exception, having the ability to poll all the running threads for their currently handled exception would be fantastic. There is an exposed function named `sys._current_frames()` that allows to list the current frame handled by CPython. Having an equivalent for `sys._current_exceptions()` that would return the content of `sys.exc_info()` for each running thread would solve the issue. -- components: Interpreter Core messages: 374575 nosy: jd priority: normal severity: normal status: open title: Allow to retrieve ongoing exception handled by every threads type: enhancement versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue41435> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41435] Allow to retrieve ongoing exception handled by every threads
Change by Julien Danjou : -- keywords: +patch pull_requests: +20832 stage: -> patch review pull_request: https://github.com/python/cpython/pull/21689 ___ Python tracker <https://bugs.python.org/issue41435> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41435] Allow to retrieve ongoing exception handled by every threads
Julien Danjou added the comment: Adding to the thread object might be a good idea, but it does not work if you ever start threads with the low-level `_thread` API. It's also a different design from the existing `sys._current_frames`. Adding it on top of a frame is interesting though. I think internally it's more linked to the execution stack in a thread than to a frame, though. -- ___ Python tracker <https://bugs.python.org/issue41435> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18191] urllib2/urllib.parse.splitport does not handle IPv6 correctly
New submission from Julien Danjou:
>>> import urllib.parse
>>> urllib.parse.splitport("::1")
(':', '1')
This is obviously invalid.
--
components: Library (Lib)
messages: 190968
nosy: jd
priority: normal
severity: normal
status: open
title: urllib2/urllib.parse.splitport does not handle IPv6 correctly
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
___
Python tracker
<http://bugs.python.org/issue18191>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18191] urllib2/urllib.parse.splitport does not handle IPv6 correctly
Julien Danjou added the comment: Attached is a patch inspired by a function written for OpenStack Oslo library. -- keywords: +patch Added file: http://bugs.python.org/file30558/0001-Make-urllib.parse.splitport-handle-IPv6-correctly.patch ___ Python tracker <http://bugs.python.org/issue18191> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1644818] Allow importing built-in submodules
Julien Danjou added the comment: Is there to chance to see this *bug* fixed someday? -- nosy: +jd ___ Python tracker <http://bugs.python.org/issue1644818> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
