[issue45949] Provide pure-Python implementation of Programs/_freeze_module for cross building
Eric Snow added the comment: You're welcome! Thanks for doing the work! -- ___ Python tracker <https://bugs.python.org/issue45949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28315 pull_request: https://github.com/python/cpython/pull/30092 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset 121f1f893a39d0b58d3d2b5597505c154ecaac2a by Eric Snow in branch 'main': bpo-45953: Statically initialize the small ints. (gh-30092) https://github.com/python/cpython/commit/121f1f893a39d0b58d3d2b5597505c154ecaac2a -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28319 pull_request: https://github.com/python/cpython/pull/30096 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46070] broken subinterpreters
Eric Snow added the comment: (related: bpo-44059) Presumably the problem relates to global state used in different interpreters leading to an inconsistent state in the crashing extension (or its dependencies). @graysky, do you know if this was a problem before Python 3.8? -- components: +Subinterpreters nosy: +eric.snow, petr.viktorin, shihai1991 stage: -> test needed type: -> behavior versions: +Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment:
It sounds like this bug is another case where we have made some objects
per-interpreter but others are still global and this is causing problems.
_PyUnicode_EqualToASCIIId() wouldn't have any problems if interpreters
weren't sharing any objects (or were only sharing immutable "immortal"
objects).
For now can we just move the relevant per-interpreter strings from
PyInterpreterState.unicode_state ("interned" and "ids") up to
_PyRuntimeState. They will then be global, which should restore
the correct behavior.
Personally, I'd rather we not revert the original change. Moving the data
to _PyRuntimeState would save me some effort with related work I'm doing
right now.
Of course, the potential bug would still exist in _PyUnicode_EqualToASCIIId().
Could we add a test as part of this fix to verify the failure case described
here actually works?
--
___
Python tracker
<https://bugs.python.org/issue46006>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment: FWIW, it makes sense to me for the interned strings to be per-interpreter eventually. Otherwise strings interned by an interpreter would persist after that interpreter is finalized, potentially leaking memory until the runtime is finalized. However, if we end up with immortal objects then I think all the strings created through _Py_IDENTIFIER() should be global (_PyRuntimeState). Otherwise they must be per-interpreter (if we have a per-interpreter GIL). -- ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Change by Eric Snow : -- pull_requests: +28350 pull_request: https://github.com/python/cpython/pull/30131 ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment: I've created a PR for moving the interned strings and identifiers to _PyRuntimeState until we are ready to move them back to the interpreter. -- ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment: If that seems okay, I'll work on a backport PR for 3.10. -- ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46109] Separate resources and abc docs from other importlib docs
Eric Snow added the comment: +1 to splitting out the importlib.resources docs I'm mostly +1 for splitting out the others (.abc, .machinery, .util) too, not just .abc. Regardless, a ToC at the top of the main page which identifies the submodules (and provides a brief summary for each) would be extra helpful. I suppose that could be helpful regardless of whether or not we move submodule docs out. -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue46109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46109] Separate resources and abc docs from other importlib docs
Eric Snow added the comment: FWIW, one benefit to having everything in one doc (instead of one doc per submodule) is that it's a little easier to see the bigger picture. This is particularly important for folks interested in customizing the import system (rather than just interacting with it). However, they would probably be better served by an explicit section on how to customize the import system. The language reference provides the info but its focus is to specify rather than to guide. So it may make sense to have a dedicated section to at least cover the supported customization approaches (at a high level). This would definitely be covered by a separate issue but it makes less sense if we don't move the submodules to their own docs. -- ___ Python tracker <https://bugs.python.org/issue46109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46109] Separate resources docs from other importlib docs
Eric Snow added the comment: +1 -- ___ Python tracker <https://bugs.python.org/issue46109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment: are there any objections to my PR? -- ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment: +1 on just reverting in both branches. I can deal with my stuff separately. -- ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters
Eric Snow added the comment: > IMO writing a complete rationale for running multiple interpreters in > parallel which require a whole PEP. FYI, I'm planning on having such a PEP published in the next few days. -- ___ Python tracker <https://bugs.python.org/issue46006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Eric Snow added the comment: In summary, regarding the test_embed failure: > FWIW: from test_embed.test_init_setpythonhome: > > if not config['executable']: > config['use_frozen_modules'] = -1 > > From the buildbot test stdout (but not pythoninfo) I see that > config["executable"] is set to "". ... > == > FAIL: test_init_setpythonhome (test.test_embed.InitConfigTests) > -- > ... > - 'use_frozen_modules': 1, > + 'use_frozen_modules': -1, > ?+ > ... ... > - out-of-tree build with system Python 3.8 => succeeds > - out-of-tree build with system Python 3.9 => succeeds > - in-tree build with system Python 3.9 => fails > - in-tree build with system Python 3.8 => fails Note that in-tree builds are working find on other platforms, AFAICS. -- ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Eric Snow added the comment: Just to be clear, the test_embed failure is on AMD64 FreeBSD Non-Debug 3.x (https://buildbot.python.org/all/#/builders/172). I agree that 99fcf15052 (bpo-45582: Port getpath[p].c to Python (GH-29041)) is likely to be the cause. -- nosy: +steve.dower ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Eric Snow added the comment: A FreeBSD builder where it's working: AMD64 FreeBSD Shared 3.x (koobs-freebsd-564d) OS: FreeBSD 14.0-CURRENT main-n244626-cb7cc72c546: GENERIC-NODEBUG (amd64) System: 2-core, 8Gb, VirtualBox (6.x) Guest Features: - Compiler: FreeBSD clang version 11.0.1 (llvmorg-11.0.1-0-g43ff75f2c3fe) - Clang "is" cc (default system compiler) - OpenSSL 1.1.1i-freebsd 8 Dec 2020 Where it isn't: AMD64 FreeBSD Non-Debug 3.x (koobs-freebsd-9e36) Environment: * OS: FreeBSD 12.3-RELEASE r371126 GENERIC amd64 * System: 2-core, 2Gb, VirtualBox (6.x) Guest Features: * cc = FreeBSD clang version 10.0.1 (g...@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) * DTRACE * ZFS * Posix Semaphores The main difference I see is the FreeBSD version. -- ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Change by Eric Snow : -- pull_requests: +28645 pull_request: https://github.com/python/cpython/pull/30438 ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Eric Snow added the comment: I just put up a PR that should fix the failure. -- ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Eric Snow added the comment: New changeset 68c76d9766cccb5fd992b0ac4b39645d9665dbe2 by Eric Snow in branch 'main': bpo-46263: Do not ever expect "use_frozen_modules" to be -1. (gh-30438) https://github.com/python/cpython/commit/68c76d9766cccb5fd992b0ac4b39645d9665dbe2 -- ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Change by Eric Snow : -- status: open -> pending ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46263] FreeBSD buildbots cannot compile Python
Eric Snow added the comment: test_pyobject_freed_is_freed is still failing on AMD64 FreeBSD Shared 3.x. -- stage: patch review -> status: pending -> open ___ Python tracker <https://bugs.python.org/issue46263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46295] Subinterpreters hang in GIL adquisition if an extension module calls PyGILState_Ensure.
Eric Snow added the comment: Is this a duplicate of bpo-15751? (also see bpo-10915) -- ___ Python tracker <https://bugs.python.org/issue46295> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46328] add sys.exception()
Eric Snow added the comment: So sys.exception() will be equivalent to sys.exc_info()[1] (or rather, sys.exc_info() will be (type(sys.exception()), sys.exception(), sys.exception().__traceback__))? That seems good to me. -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue46328> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46328] add sys.exception()
Eric Snow added the comment: FWIW, here's an alternative we should *not* pursue: return just the exception from sys.exc_info(). For compatibility, we would implement `__iter__` and `__getitem__` on BaseException, to duplicate the behavior of the current tuple. There are a number of good reasons why this is *not* what we should do. -- ___ Python tracker <https://bugs.python.org/issue46328> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: I'll look into this today. To reproduce, I should run the urllib3 test suite? -- assignee: -> eric.snow stage: -> needs patch ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: I must be missing something. Here is what I did: ``` $ cd cpython $ git checkout main $ make -j8 $ ./python -v venv ../venv-urllib3 $ cd .. $ git clone https://github.com/urllib3/urllib3 $ cd urllib3 $ ../venv-urllib3/bin/python3 -m pip install nox $ ../venv-urllib3/bin/nox -Rs test-3.11 nox > Running session test-3.11 nox > Session test-3.11 skipped: Python interpreter 3.11 not found. ``` I am not familiar with nox so I'm not sure how to trouble-shoot this. -- ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: It looks like the urllib3 CI is passing for 3.11.0a3: https://github.com/urllib3/urllib3/runs/4762856431 (Ubuntu 3.11-dev test-3.11). Is there a urllib3 issue number you could point me at? -- ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46343] Add PyErr_GetActiveException and PyErr_SetActiveException
Eric Snow added the comment: Why not `PyAPI_FUNC(PyObject *) PyErr_GetActiveException(void);`? -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue46343> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset cf496d657a1a82eaf9ebfb47d721676fef6effa5 by Eric Snow in branch 'main': bpo-45953: Statically allocate and initialize global bytes objects. (gh-30096) https://github.com/python/cpython/commit/cf496d657a1a82eaf9ebfb47d721676fef6effa5 -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: > pip install -r dev-requirements.txt > pip install ".[socks,secure,brotli]" > pytest I was able to reproduce the crash with these steps. Thanks! -- ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: FYI, I get the crash with just: pytest test/contrib/test_pyopenssl.py::TestHTTPS::test_verify_none_and_bad_fingerprint -- ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: It looks like PyThreadState.async_exc is getting set to 0x01 somewhere. There isn't any code like that in https://github.com/python/cpython/commit/32a67246b0d1e08cd50fc3bfa58052cfeb515b2e. However, the struct layout of PyThreadState did change slightly in that commit. The ABI is generally not stable until the first beta (and sometimes a bit after that). Could it be that the wheel for one or more the dependencies was built against an earlier 3.11 release (with the previous PyThreadState layout)? If I move fields around in PyThreadState just right, I can no longer reproduce the problem. -- ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46320] runtime/interp/thread state refactoring leads to segmentation fault
Eric Snow added the comment: I'm glad we were able to figure it out relatively quickly...and without any big headaches. :) And thanks for testing against the alpha releases!!! You're having a positive impact. -- resolution: -> not a bug ___ Python tracker <https://bugs.python.org/issue46320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46360] Inconsistent import behavior for (unusual) submodules
New submission from Eric Snow : Let's look at a hypothetical module "spam" and its submodule "spam.eggs": ``` # spam.py import sys sys.modules['spam.eggs'] = None ``` Then: >>> import spam.eggs >>> import sys >>> sys.modules['spam.eggs'] is None True >>> spam.eggs Traceback (most recent call last): File "", line 1, in AttributeError: module 'spam' has no attribute 'eggs' The key inconsistent behaviors: * `import spam.eggs` succeeded even though the sys.modules entry is None * `import spam.eggs` succeeded even though "spam" isn't a package (e.g. no `__path__`, `spec.submodule_search_locations`, etc.) * the "eggs" attr wasn't bound on "spam" The relevant code is _find_and_load_unlocked() and _find_and_load() in Lib/importlib/_bootstrap.py. In _find_and_load_unlocked() we first import the parent module. Then we have a special case, where we see if "spam.eggs" was added to sys.modules as a side effect. If it was then we short-circuit the rest of import and return the submodule as-is. This leads to some of the inconsistent behavior described above, since the subsequent code (e.g. checks, binding to the parent) get skipped. In _find_and_load() we have code which raises ModuleNotFoundError if the resulting module is None, which acts as a marker that importing the module is disabled. This check is always skipped when importing the module for the first time, leading to the other inconsistent behavior from above. The is definitely a corner case, but os.path demonstrates it's a real scenario. In fact, os.path is what drew my attention to this code. Is it worth fixing? The change shouldn't be invasive so I'm leaning toward yes. It isn't a high priority though. -- assignee: eric.snow components: Interpreter Core messages: 410433 nosy: barry, brett.cannon, eric.snow, ncoghlan priority: normal severity: normal stage: needs patch status: open title: Inconsistent import behavior for (unusual) submodules type: behavior versions: Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset ed57b36c32e521162dbb97199e64a340d3bff827 by Eric Snow in branch 'main': bpo-45953: Statically allocate the main interpreter (and initial thread state). (gh-29883) https://github.com/python/cpython/commit/ed57b36c32e521162dbb97199e64a340d3bff827 -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)
Eric Snow added the comment: > adding a new "gc" member in the _PyRuntimeState structure also causes the ABI > CI check to fail. What if you move it to the end of the struct? -- ___ Python tracker <https://bugs.python.org/issue46070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)
Eric Snow added the comment: > (*) I made the GC state per-interpreter: commit > 7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019) FYI, this was done by me in an earlier comment which we ended up reverting. Later you basically un.reverted that. > The bug is that a C function object (_sre.compile) is created in an > interpreter, tracked by the GC list of this interpreter, and then it is > destroye and untracked in another interpreter. FWIW, at one point I had a branch that supported sharing read-only Py_Buffer data. When the receiving interpreter was done with it I'd call Py_AddPendingCall() to schedule the cleanup in the "owner" interpreter. However, this only worked because I kept track of the owner. Adding that pointer to every object wouldn't be feasible but I suppose there are other things we could do that wouldn't be super inefficient, like don't worry about it for the main interpreter, use a hash table (Victor's idea), borrow some of the bits of the PyObject head to store a flag or even an index into an array (if there are only a few interpreters), or even make the allocator per-interpreter and then extrapolate the interpreter based on the object's address. Regardless, it is still much simpler to make all objects per-interpreter. -- ___ Python tracker <https://bugs.python.org/issue46070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46360] Inconsistent import behavior for (unusual) submodules
Eric Snow added the comment: > I'm going to assume the "even though sys.modules has `None`" case, > which I think is an oversight and should probably get fixed Yep, I agree. That's the case I was looking at in the first place. I noticed the other two as I was hacking together code to verify the None behavior. :) Bothering to change those would be more trouble than its worth. -- ___ Python tracker <https://bugs.python.org/issue46360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
New submission from Eric Snow : The static initializer for `_PyRuntime` is currently defined in Include/internal/pycore_runtime.h. However, it is only needed by Python/pylifecycle.c (and Python/pystate.c for an optimization) and should only be used there. (Also, the initializer is quite large.) So I'm planning on moving it to it's own internal header file. -- assignee: eric.snow components: Interpreter Core messages: 410529 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Move runtime static init to its own header file. versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
Change by Eric Snow : -- keywords: +patch pull_requests: +28785 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30587 ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
Eric Snow added the comment: New changeset bc02eac9d2cb36faffc5027b7ce09e6dd0922a7f by Eric Snow in branch 'main': bpo-46370: Move the static initializer for _PyRuntime to its own header file. (gh-30587) https://github.com/python/cpython/commit/bc02eac9d2cb36faffc5027b7ce09e6dd0922a7f -- ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
Change by Eric Snow : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28786 pull_request: https://github.com/python/cpython/pull/30588 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28787 pull_request: https://github.com/python/cpython/pull/30589 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28788 pull_request: https://github.com/python/cpython/pull/30590 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset 324908ba936d5d262026deebb81f050803848c41 by Eric Snow in branch 'main': bpo-45953: Statically initialize all the PyThreadState fields we can. (gh-30590) https://github.com/python/cpython/commit/324908ba936d5d262026deebb81f050803848c41 -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset 322f962f3ee31d0dbde99e36379de8488ccc6804 by Eric Snow in branch 'main': bpo-45953: Statically initialize all the non-object PyInterpreterState fields we can. (gh-30589) https://github.com/python/cpython/commit/322f962f3ee31d0dbde99e36379de8488ccc6804 -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)
Eric Snow added the comment: > the deep-frozen objects also reference the small ints directly, as well as > the singleton for b"". > Is this even safe across Py_Finalize()/Py_Initialize()? If not, we'll need to > roll that back as well. The small ints and the empty bytes object each have "immortal" refcounts too (9, just like you did in deepfreeze). So they would cause a similar behavior to what Victor reported. Otherwise I wouldn't expect any problems across Py_Finalize()/Py_Initialize(). -- ___ Python tracker <https://bugs.python.org/issue46449> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
New submission from Eric Snow :
`_Py_Identifier` has been useful but at this point there is a faster and
simpler approach we could take as a replacement: statically initialize the
objects as fields on `_PyRuntimeState` and reference them directly through a
macro.
This would involve the following:
* add a `PyUnicodeObject field (not a pointer) to `_PyRuntimeState` for each
string that currently uses `_Py_IDENTIFIER()`
* initialize each object as part of the static initializer for `_PyRuntimeState`
* make each "immortal" (e.g. start with a really high refcount)
* add a macro to look up a given string
* update each location that currently uses `_Py_IDENTIFIER()` to use the new
macro instead
As part of this, we would also do the following:
* get rid of all C-API functions with `_Py_Identifer` parameters
* get rid of the old runtime state related to identifiers
* get rid of `_Py_Identifier`, `_Py_IDENTIFIER()`, etc.
(Note that there are several hundred uses of `_Py_IDENTIFIER()`, including a
number of duplicates.)
Pros:
* reduces indirection (and extra calls) for C-API using the strings (making the
code easier to understand and speeding it up)
* the objects are referenced from a fixed address in the static data section
(speeding things up and allowing the C compiler to optimize better)
* there is no lazy allocation (or lookup, etc.) so there are fewer possible
failures when the objects get used (thus less error return checking)
* simplifies the runtime state
* saves memory (at little, at least)
* the approach for per-interpreter is simpler (if needed)
* reduces the number of static variables in any given C module
* reduces the number of functions in the ("private") C-API
* "deep frozen" modules can use these strings
* other commonly-used strings could be pre-allocated by adding
`_PyRuntimeState` fields for them
Cons:
* churn
* adding a string to the list requires modifying a separate file from the one
where you actually want to use the string
* strings can get "orphaned" (we could prevent this with a check in `make
check`)
* some PyPI packages may rely on `_Py_IDENTIFIER()` (even though it is
"private" C-API)
* some strings may never get used for any given ./python invocation
Note that with a basic partial implementation (GH-30928) I'm seeing a 1%
improvement in performance (see
https://github.com/faster-cpython/ideas/issues/230).
--
assignee: eric.snow
components: Interpreter Core
messages: 411799
nosy: eric.snow, serhiy.storchaka, vstinner
priority: normal
pull_requests: 29107
severity: normal
stage: needs patch
status: open
title: Replace _Py_IDENTIFIER() with statically initialized objects.
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46541>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: ## Background ## `_Py_Identifier` (and `_Py_IDENTIFIER()`, etc.) was added in 2011 [1][2] for several reasons: * provide a consistent approach for a common optimization: caching C-string based string objects * facilitate freeing those objects during runtime finalization The solution involved using a static variable defined, using `_Py_IDENTIFIER()` near the code that needed the string. The variable (a `_Py_Identifier`) would hold the desired C string (statically initialized) and the corresponding (lazily created) `PyUnicodeObject`. The code where the `_Py_Identifier` was defined would then pass it to specialized versions of various C-API that would normally consume a C string or `PyUnicodeObject`. Then that code would use either the C-string or the object (creating and caching it first if not done already). This approach decentralized the caching but also provided a single tracking mechanism that made it easier to clean up the objects. Over the last decade a number of changes were made, including recent changes to make the identifiers per-interpreter and to use a centralized cache. [1] https://github.com/python/cpython/commit/afe55bba33a20f87a58f940186359237064b428f [2] https://mail.python.org/archives/list/python-...@python.org/message/FRUTTE47JO2XN3LXV2J4VB5A5VILILLA/ -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- keywords: +patch pull_requests: +29120 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30941 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46166] Get "self" args or non-null co_varnames from frame object with C-API
Eric Snow added the comment: In addition to what Mark said, note that co_varnames get's populated lazily by the Python-level getter for code.co_varnames. So could you call the Python function before entering the hot path? Regardless, a dedicated C-API for this like Mark suggested would be the better solution. -- ___ Python tracker <https://bugs.python.org/issue46166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 247480a21cb165efdacc346a2d589dfc27e18283 by Eric Snow in branch 'main': bpo-46541: Generate the global objects initializer. (gh-30941) https://github.com/python/cpython/commit/247480a21cb165efdacc346a2d589dfc27e18283 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Eric Snow added the comment: On Tue, Jan 25, 2022 at 4:14 PM STINNER Victor wrote: > Currently, most CI run "make buildbottest" which uses -r option of > libregrtest: randomize tests order. How hard would it be to first randomize the list and then move the slow tests up to a random position in the first half (for example) of the list? -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: > Any chance we could revert the recent renaming of tstate.exc_state and > tstate.root_cframe Yeah, I'll sort this out. Sorry for that. -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +29221 pull_request: https://github.com/python/cpython/pull/31038 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: If necessary, we can keep _Py_IDENTIFIER() (and the functions). Regardless, we can stop using it internally. -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: FYI, I've posted to python-dev for feedback before proceeding: https://mail.python.org/archives/list/python-...@python.org/thread/DNMZAMB4M6RVR76RDZMUK2WRLI6KAAYS/ -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment: > I'm running some long-running (possibly infinite) tasks in the thread pool, > and I cancel them in an `atexit` callback To be clear, by "cancel" you are not talking about Future.cancel(). Rather, your handler causes all running tasks to finish (by sending a special message on the socket corresponding to each running task). Is that right? Some other things I inferred from your atexit handler: * it does not make sure the task associated with the socket finishes (no way of knowing?) * so if a task hangs while trying to stop then the running thread in the ThreadPoolExecutor would block shutdown forever * similarly, if a task is stuck handling a request then it will never receive the special message on the socket, either blocking the send() in your handler or causing ThreadPoolExecutor shutdown/atexit to wait forever * it vaguely implies a 1-to-1 relationship between sockets and *running* tasks * likewise that pending (queued) tasks do not have an associated socket (until started) * so once your handler finishes, any tasks pending in the ThreadPoolExecutor queue will eventually get started but never get stopped by your handler; thus you're back to the deadlock situation Does all that sound right? Most of that is relevant to some possible solutions I have in mind. -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment: > I'm running some long-running (possibly infinite) tasks in the thread pool, > and I cancel them in an `atexit` callback Alternately, perhaps ThreadPoolExecutor isn't the right fit here, as implied by the route you ended up going. It seems like it's not well-suited for long-running (or infinite) tasks. In that case, perhaps the concurrent.futures docs could be more clear about when ThreadPoolExecutor is not a good fit and what the alternatives are. -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment:
FWIW, here's a brain dump about ThreadPoolExecutor and its atexit handler after
having looked at the code.
First, the relationship between the objects involved:
* work item -> Future
* work item -> task (e.g. function)
* queue -> [work item]
* worker -> executor
* worker -> queue
* worker -> currently running work item
* thread -> worker
* ThreadPoolExecutor -> [thread]
* ThreadPoolExecutor -> queue
* global state -> {thread: queue}
Observations:
* a work item's future and task are not aware of each other, and operations on
either have no effect on the other
Next, let's look at the relevant ways the objects can be used:
* publicly
* ThreadPoolExecutor.submit(task) -> Future
* ThreadPoolExecutor.shutdown()
* Future.result() and Future.exception()
* Future.cancel()
* Future.add_done_callback()
* internally
* queue.pop() -> work item
* .run()
* thread.join()
* Future.set_running_or_notify_cancel()
* Future.set_result() and Future.set_exception()
Observations:
* nothing interacts with a worker directly; it is waited on via its thread and
it receives work (or None) via the queue it was given
* once a worker pops the next work item off the queue, nothing else has access
to that work item (the original ThreadPoolExecutor().submit() caller has the
Future, but that's it)
* there is no cancelation mechanism for tasks
* there is no simple way to interact with the queued tasks
* realistically, there is no way to interact with the currently running task
* there is no reliable way to "kill" a thread
Regarding how tasks run:
* ThreadPoolExecutor.submit(task) -> Future
* ThreadPoolExecutor.submit(task) -> work item (Future, task) -> queue
* ThreadPoolExecutor.submit(task) -> thread (worker)
* thread -> worker -> ( queue -> work item -> task )
Observations::
* the worker loop exits if the next item in the queue is None (and the executor
is shutting down)
Now lets look more closely at the atexit handler.
* as you noted, since 3.9 it is registered with threading._register_atexit()
instead of atexit.register()
* the threading atexit handlers run before the regular atexit handlers
* the ThreadPoolExecutor handler does not actually interact with
ThreadPoolExecutor instances directly
* it only deals with a module-global list of (thread, [work item]) pairs, to
which ThreadPoolExecutor instances add items as they go
The handler does the following:
1. disables ThreadPoolExecutor.submit() (for all instances)
2. (indirectly) tells each worker to stop ASAP
3. lets every pending task run (and lets every running task keep running)
4. waits until all tasks have finished
It does not:
* call any ThreadPoolExecutor.shutdown()
* otherwise deal with the ThreadPoolExecutor instances directly
* call Future.cancel() for any of the tasks
* use any timeout in step 4, so it may block forever
* notify tasks that they should finish
* deal well with any long-running (or infinite) task
ThreadPoolExecutor.shutdown() basically does the same thing. However, it only
does the steps above for its own tasks. It also optionally calls
Future.cancel() for each queued task (right before step 2). However, all that
does is keep any queued-but-not-running tasks from starting. Also, you can
optionally skips step 4.
--
___
Python tracker
<https://bugs.python.org/issue41962>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment: > This means that ThreadPoolExecutor's atexit runs before mine, > and since I never get a chance to cancel my tasks, it deadlocks. (assuming we want to support long-running tasks here) With all the above in mind, there are a few things that may help. The first that comes to mind is to have the atexit handler call ThreadPoolExecutor.shutdown() for each instance. So something like this: def _python_exit(): global _shutdown with _global_shutdown_lock: _shutdown = True for executor in list(_executors): executor.shutdown() That would require a little refactoring to make it work. However, the change is simpler if each executor has its own atexit handler: class ThreadPoolExecutor(_base.Executor): def __init__(self, ...): ... threading._register_atexit(self._atexit()) def _atexit(self): global _shutdown _shutdown = True self.shutdown() The value of either approach is that you can then subclass ThreadPoolExecutor to get what you want: class MyExecutor(ThreadPoolExecutor): def shutdown(self, *args, **kwargs): stop_my_tasks() super().shutdown(*args, **kwwargs) One thing I thought about was supporting a per-task finalizer instead, since that aligns more closely with the way ThreadPoolExecutor works. It would only apply So something like one of the following: * ThreadPoolExecutor(finalize_task=) * ThreadPoolExecutor.submit(finalize=) * ThreadPoolExecutor.register_atexit() * (classmethod) ThreadPoolExecutor.register_atexit() * concurrent.futures.register_atexit() (It would probably make sense to pass the list of currently running tasks to the callable.) -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: (thanks Victor: https://mail.python.org/archives/list/python-...@python.org/message/7RMLIJHUWVBZFV747TFEHOE6LNBVQSMM/) 3rd party use of _Py_IDENTIFIER(): * blender + https://github.com/blender/blender/blob/master/source/blender/python/intern/bpy_traceback.c#L53 - copied from core code - "msg", "filename", "lineno", "offset", "text", "" - uses _PyObject_GetAttrId() * datatable + https://github.com/h2oai/datatable/blob/45a87337bc68576c7fb6900f524925d4fb77d6a6/src/core/python/obj.cc#L76 - in C++ wrapper getting sys.stdout, etc. and writing to sys.stdout - has to hack around C++14 support - has a fallback under limited API - "stdin", "stdout", "stderr", "write" - uses _PySys_GetObjectId(), _PyObject_GetAttrId() * multidict (in aiohttp) + https://github.com/aio-libs/multidict/blob/6dedb623cca8e8fe64f502dfa479826efc321385/multidict/_multilib/defs.h#L8 + https://github.com/aio-libs/multidict/blob/6dedb623cca8e8fe64f502dfa479826efc321385/multidict/_multilib/istr.h#L46 + https://github.com/aio-libs/multidict/blob/6dedb623cca8e8fe64f502dfa479826efc321385/multidict/_multilib/pair_list.h#L114 - calling str.lower() - "lower" - uses _PyObject_CallMethodId() * mypy (exclusively in mypyc, including in generated code!) + https://github.com/python/mypy/blob/3c935bdd1332672f5daeae7f3f9a858a45d4/mypyc/lib-rt/dict_ops.c#L76 + https://github.com/python/mypy/blob/3c935bdd1332672f5daeae7f3f9a858a45d4/mypyc/lib-rt/dict_ops.c#L131 - "setdefault", "update" - uses _PyObject_CallMethodIdObjArgs(), _PyObject_CallMethodIdOneArg() + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/pythonsupport.h#L26 + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/pythonsupport.h#L109 - "__mro_entries__", "__init_subclass__" - uses _PyObject_LookupAttrId(), _PyObject_GetAttrId() + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/misc_ops.c#L27 + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/misc_ops.c#L47 - "send", "throw", "close" - uses _PyObject_CallMethodIdOneArg(), _PyObject_GetAttrId() + https://github.com/python/mypy/blob/8c5c915a89ec0f35b3e07332c7090e62f143043e/mypyc/lib-rt/bytes_ops.c#L104 - "join" - uses _PyObject_CallMethodIdOneArg() + https://github.com/python/mypy/blob/3c935bdd1332672f5daeae7f3f9a858a45d4/mypyc/codegen/emitwrapper.py#L326 + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/misc_ops.c#L694 - uses _PyObject_GetAttrId() * pickle5 + https://github.com/pitrou/pickle5-backport/blob/e6117502435aba2901585cc6c692fb9582545f08/pickle5/_pickle.c#L224 + https://github.com/pitrou/pickle5-backport/blob/e6117502435aba2901585cc6c692fb9582545f08/pickle5/compat.h - "getattr" - uses _PyUnicode_FromId() * pysqlite3 + https://github.com/coleifer/pysqlite3/blob/f302859dc9ddb47a1089324dbca3873740b74af9/src/microprotocols.c#L103 + https://github.com/coleifer/pysqlite3/blob/f302859dc9ddb47a1089324dbca3873740b74af9/src/microprotocols.c#L119 - "__adapt__", "__conform__" - uses _PyObject_CallMethodId() + https://github.com/coleifer/pysqlite3/blob/093b88d1a58b141db8cf971c35ea1f6b674d0d02/src/connection.c#L51 + https://github.com/coleifer/pysqlite3/blob/093b88d1a58b141db8cf971c35ea1f6b674d0d02/src/connection.c#L772 - "finalize", "value", "upper", "cursor" - uses _PyObject_CallMethodId(), _PyObject_CallMethodIdObjArgs() + https://github.com/coleifer/pysqlite3/blob/49ce9c7a89a3c9f47ab8d32b6c4e2f7d629c1688/src/module.c#L195 - "upper" - uses _PyObject_CallMethodId() + https://github.com/coleifer/pysqlite3/blob/91b2664f525b19feedfca3f0913302c6f1e8be8a/src/cursor.c#L103 - "upper" - uses _PyObject_CallMethodId() * typed_ast + a fork of CPython's ast code * zodbpickle + a fork of CPython's pickle All of them should be trivial to drop _Py_IDENTIFIER() without any real performance impact or mess. Also, the following implies that PyPy has some sort of _Py_IDENTIFIER() support: https://github.com/benhoyt/scandir/blob/3396aa4155ffde8600a0e9ca50d5872569169b5d/_scandir.c#L51. -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Change by Eric Snow : -- pull_requests: +29389 pull_request: https://github.com/python/cpython/pull/31219 ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Eric Snow added the comment: New changeset 1e6214dbd6a980b47123229aefd60bb2c9341b53 by Eric Snow in branch 'main': bpo-45952: Get the C analyzer tool working again. (gh-31219) https://github.com/python/cpython/commit/1e6214dbd6a980b47123229aefd60bb2c9341b53 -- ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Change by Eric Snow : -- pull_requests: +29390 pull_request: https://github.com/python/cpython/pull/31220 ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Eric Snow added the comment: New changeset c018d3037b5b62e6d48d5985d1a37b91762fbffb by Eric Snow in branch 'main': bpo-45952: Get the C analyzer tool working again. (gh-31220) https://github.com/python/cpython/commit/c018d3037b5b62e6d48d5985d1a37b91762fbffb -- ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 81c72044a181dbbfbf689d7a977d0d99090f26a8 by Eric Snow in branch 'main': bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928) https://github.com/python/cpython/commit/81c72044a181dbbfbf689d7a977d0d99090f26a8 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Change by Eric Snow : -- pull_requests: +29396 pull_request: https://github.com/python/cpython/pull/31225 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Eric Snow added the comment: New changeset 77bab59c8a1f04922bb975cc4f11e5323d1d379d by Eric Snow in branch 'main': bpo-36876: Update the c-analyzer whitelist. (gh-31225) https://github.com/python/cpython/commit/77bab59c8a1f04922bb975cc4f11e5323d1d379d -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Change by Eric Snow : -- pull_requests: +29409 pull_request: https://github.com/python/cpython/pull/31239 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting and immortal objects
Eric Snow added the comment: @Eddie, what can I do to push this forward? FYI, in addition to the python-dev thread a few weeks back, I've brought the matter up with the steering council. [1] Also, if we can get back to performance-neutral (currently at about 4% slower) then there would be a lot less controversy. Even at 2% it may be enough. [1] https://github.com/python/steering-council/issues/103 -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Eric Snow added the comment: New changeset cb68788dcadf43b47292bab7816a5ed9efa69730 by Eric Snow in branch 'main': bpo-36876: Minor cleanup to c-analyzer "ignored" data.' (gh-31239) https://github.com/python/cpython/commit/cb68788dcadf43b47292bab7816a5ed9efa69730 -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Change by Eric Snow : -- pull_requests: +29430 pull_request: https://github.com/python/cpython/pull/31264 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Eric Snow added the comment: New changeset 80e4f262aa27a39abf3fadc19a6323fea4607a8f by Eric Snow in branch 'main': bpo-36876: Make sure the c-analyzer is checking all the source files.' (gh-31264) https://github.com/python/cpython/commit/80e4f262aa27a39abf3fadc19a6323fea4607a8f -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: On Fri, Feb 11, 2022 at 1:36 AM Christoph Reiter wrote: > Sorry if off topic, but I noticed that CPython doesn't deprecate macros in > code, while with gcc/clang it's possible to show compiler warnings for them > using some pragma magic: > [snip] > Maybe that makes getting rid of them easier in the long run? That's a good question. We do have Py_DEPRECATED() (in Include/pyport.h), which is used for symbols. I'm not sure anyone has given much thought to deprecating macros, but it's probably worth considering. I recommend that you post something about this to python-...@python.org. FWIW, here are other explanations of how to deprecate macros: * https://stackoverflow.com/questions/57478368/what-is-the-best-way-to-mark-macro-as-deprecated/57479189#57479189 * https://stackoverflow.com/questions/2681259/how-to-deprecate-a-c-pre-processor-macro-in-gcc/29297970#29297970 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46748] Python.h includes stdbool.h
Eric Snow added the comment: On Mon, Feb 14, 2022 at 2:28 AM Petr Viktorin wrote: > Eric, is this necessary? Would an old-school `int` do? > Or should we say it's 2022 already and everyone needs to use stdbool.hfore > bools? I started using ``bool`` (stdbool.h) when I saw it specified in PEP 7 (https://www.python.org/dev/peps/pep-0007/#c-dialect). If it is a problem then I think it should be answered relative to PEP 7. I'm not sure what the best route is though. Personally, I'd argue that if it's in C99 then it should probably be standard at this point. :) However, I'm not opposed to going back to plain int. (And I know there are definitely a number of old-school folks who prefer plain int. :) ) -- ___ Python tracker <https://bugs.python.org/issue46748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: With core code sorted out, stdlib and 3rd party extension modules are left to sort out. I see the following possibilities: * leave `_Py_IDENTIFIER()` alone (it is already disallowed in core code) * change `_Py_IDENTIFIER()` to create static string objects (then get rid of global state) * get rid of `_Py_IDENTIFIER()` a. provide a public alternative (needs a PEP) b. first work with 3rd party projects to stop using it -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- Removed message: https://bugs.python.org/msg413241 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: With core code sorted out, stdlib and 3rd party extension modules are left to sort out. I see the following possibilities: 1. leave `_Py_IDENTIFIER()` alone (it is already disallowed in core code) 2. change `_Py_IDENTIFIER()` to create static string objects (then get rid of global state) 3. get rid of `_Py_IDENTIFIER()` a. provide a public alternative (needs a PEP) b. first work with 3rd party projects to stop using it -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29494 pull_request: https://github.com/python/cpython/pull/31344 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
New submission from Eric Snow : Currently it is created dynamically from the tuple freelist. -- assignee: eric.snow components: Interpreter Core messages: 413268 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Statically allocate and initialize the empty tuple. versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
Eric Snow added the comment: Also see bpo-45953. -- ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
Change by Eric Snow : -- keywords: +patch pull_requests: +29495 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31345 ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29496 pull_request: https://github.com/python/cpython/pull/31346 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 12360aa159c42c7798fd14225d271e6fd84db7eb by Eric Snow in branch 'main': bpo-46541: Discover the global strings. (gh-31346) https://github.com/python/cpython/commit/12360aa159c42c7798fd14225d271e6fd84db7eb -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29513 pull_request: https://github.com/python/cpython/pull/31363 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29514 pull_request: https://github.com/python/cpython/pull/31364 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 6c8958948666403f2370ca7b4c0a52b2010ec16d by Eric Snow in branch 'main': bpo-46541: Drop the check for orphaned global strings. (gh-31363) https://github.com/python/cpython/commit/6c8958948666403f2370ca7b4c0a52b2010ec16d -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 4d8a515d193a4c9f3844704f974ddb870d7ee383 by Eric Snow in branch 'main': bpo-46541: Scan Fewer Files in generate_global_objects.py (gh-31364) https://github.com/python/cpython/commit/4d8a515d193a4c9f3844704f974ddb870d7ee383 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46765] Replace Locally Cached Strings with Statically Initialized Objects
New submission from Eric Snow : This removes a number of static variables and is a little more efficient. -- assignee: eric.snow components: Interpreter Core messages: 413313 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Replace Locally Cached Strings with Statically Initialized Objects versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46765> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46765] Replace Locally Cached Strings with Statically Initialized Objects
Change by Eric Snow : -- keywords: +patch pull_requests: +29516 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31366 ___ Python tracker <https://bugs.python.org/issue46765> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46772] Statically Initialize PyArg_Parser in clinic.py
New submission from Eric Snow : The code generated by clinic.py is already partially statically initialized. Currently we init the other fields in Python/getargs.c:parser_init(), which runs the first time we try to use each parser. AFAICS, that remaining init that could be done statically using the data we have available in clinic.py during code generation. My primary interest is in static init of PyArg_Parser.kwtuple, which is a tuple containing only strings. -- assignee: eric.snow components: Interpreter Core messages: 413351 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Statically Initialize PyArg_Parser in clinic.py versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46773] Add a Private API for Looking Up Global Objects
New submission from Eric Snow : We need this to statically initialize PyArg_Parser.kwtuple. (See bpo-46772.) For now this will be a "private" API (leading underscore). Ultimately, we'll want a Public API, so we can eventually stop exposing *any* objects as symbols in the C-API. However, that will need a PEP. -- assignee: eric.snow components: Interpreter Core messages: 413352 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Add a Private API for Looking Up Global Objects versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46773> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46772] Statically Initialize PyArg_Parser in clinic.py
Change by Eric Snow : -- dependencies: +Add a Private API for Looking Up Global Objects, Statically allocate and initialize the empty tuple. ___ Python tracker <https://bugs.python.org/issue46772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: (from https://github.com/python/cpython/pull/31376#issuecomment-1041836106) [corona10] > Should we create the separate bpo issue if module changes are too noisy? I think it's fine to use the one issue. There are only 26 modules with `NEEDS_PY_IDENTIFIER` and none have much risk with the change. So it's unlikely that we'll get any discussion about any specific module. If we do, we can easily create an issue then for the module in question. If one of the modules had many uses of `_Py_IDENTIFIER()` then it might make sense to split it out, but IIRC none of them have very many. -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43974] Define Py_BUILD_CORE_MODULE in extensions instead of setup.py and Modules/Setup
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue43974> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
