[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
On 23Feb2022 2240, h.vetin...@gmx.com wrote: On top of that, I think the formulation you chose does not map correctly to actual compiler capabilities. C99 (minus C11 optionals) only arrived in VS2019, there were still gaps in VS2017. Additionally, Visual Studio version doesn't directly map to the compiler version. For example, VS2022 contains multiple older versions of the compiler. Currently we use the v142 toolset for releases (selected by default in desktop development workloads in VS 2019), and can't migrate to v143 (selected by default in VS 2022) until some compiler bugs are fixed. If we're going to specify anything about MSVC, I'd prefer it was the v142 label rather than the Visual Studio version (though we can include the latter as a convenience for the reader, and hopefully clearly enough that people stop filing bug reports about it...) Cheers, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KJ5XCX7IOV6GR2MWPXV6TNJ3P5G5NLSW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 2)
Responses inline below. -eric On Tue, Feb 22, 2022 at 7:22 PM Inada Naoki wrote: > > For a recent example, see > > https://mail.python.org/archives/list/python-dev@python.org/message/B77BQQFDSTPY4KA4HMHYXJEV3MOU7W3X/. > > It is not proven example, but just a hope at the moment. So option is > fine to prove the idea. > > Although I can not read the code, they said "patching ASLR by patching > `ob_type` fields;". > It will cause CoW for most objects, isn't it? > > So reducing memory write don't directly means reducing CoW. > Unless we can stop writing on a page completely, the page will be copied. Yeah, they would have to address that. > > CPU cache invalidation exists regardless. With the current GIL the > > effect it is reduced significantly. > > It's an interesting point. We can not see the benefit from > pypeformance, because it doesn't use much data and it runs one process > at a time. > So the pyperformance can not make enough stress to the last level > cache which is shared by many cores. > > We need multiprocess performance benchmark apart from pyperformance, > to stress the last level cache from multiple cores. > It helps not only this PEP, but also optimizing containers like dict and set. +1 > Can proposed optimizations to eliminate the penalty guarantee that > every __del__, weakref are not broken, > and no memory leak occurs when the Python interpreter is initialized > and finalized multiple times? > I haven't confirmed it yet. They will not break __del__ or weakrefs. No memory will leak after finalization. If any of that happens then it is a bug. > FWIW, I filed an issue to remove hash cache from bytes objects. > https://github.com/faster-cpython/ideas/issues/290 > > Code objects have many bytes objects, (e.g. co_code, co_linetable, etc...) > Removing it will save some RAM usage and make immortal bytes truly > immutable, safe to be shared between interpreters. +1 Thanks! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QKMPALMWGF5366C6PQRSIIFVNXKF4UAM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 2)
On Wed, Feb 23, 2022 at 9:16 AM Petr Viktorin wrote: >>> But tp_dict is also public C-API. How will that be handled? >>> Perhaps naively, I thought static types' dicts could be treated as >>> (deeply) immutable, and shared? >> >> They are immutable from Python code but not from C (due to tp_dict). >> Basically, we will document that tp_dict should not be used directly >> (in the public API) and refer users to a public getter function. I'll >> note this in the PEP. > > What worries me is that existing users of the API haven't read the new > documentation. What will happen if users do use it? > Or worse, add things to it? We will probably set it to NULL, so the user code would fail or crash. I suppose we could set it to a dummy object that emits helpful errors. However, I don't think that is worth it. We're talking about where users are directly accessing tp_dict of the builtin static types, not their own. That is already something they should definitely not be doing. > (Hm, the current docs are already rather confusing -- 3.2 added a note > that "It is not safe to ... modify tp_dict with the dictionary C-API.", > but above that it says "extra attributes for the type may be added to > this dictionary [in some cases]") Yeah, the docs will have to be clarified. >> Having thought about it some more, I don't think this PEP should be >> strictly bound to per-interpreter GIL. That is certainly my personal >> motivation. However, we have a small set of users that would benefit >> significantly, the change is relatively small and simple, and the risk >> of breaking users is also small. > > Right, with the recent performance improvements it's looking like it > might stand on its own after all. Great! >> Honestly, it might not have needed a PEP in the first place if I >> had been a bit more clear about the idea earlier. > > Maybe it's good to have a PEP to clear that up :) Yeah, the PEP process has been helpful for that. :) -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AKFMFZ45UJXED24YRB4NHQ4HT442XVSP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 2)
On Wed, Feb 23, 2022 at 4:21 PM Antonio Cuni wrote: > When refcheck=True (the default), numpy raises an error if you try to resize > an array inplace whose refcnt > 2 (although I don't understand why > 2 and > not > 1, and the docs aren't very clear about this). > > That said, relying on the exact value of the refcnt is very bad for > alternative implementations and for HPy, and in particular it is impossible > to implement ndarray.resize(refcheck=True) correctly on PyPy. So from this > point of view, a wording which explicitly restricts the "legal" usage of the > refcnt details would be very welcome. Thanks for the feedback and example. It helps. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/D23Z3C7CQIIGALDRSU4RDDM7GVUAASGW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 3)
I've updated PEP 683 for the feedback I've gotten. Thanks again for that! The updated PEP text is included below. The largest changes involve either the focus of the PEP (internal mechanism to mark objects immortal) or the possible ways that things can break on older 32-bit stable ABI extensions. All other changes are smaller. Given the last round of discussion, I'm hoping this will be the last round before we go to the steering council. -eric PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow , Eddie Elizondo Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCXFQ4AMTW6F6OQFINSIFYBRFCR/ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: 15-Feb-2022, 19-Feb-2022, 28-Feb-2022 Resolution: Abstract Currently the CPython runtime maintains a `small amount of mutable state `_ in the allocated memory of each object. Because of this, otherwise immutable objects are actually mutable. This can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python's scalability. This proposal mandates that, internally, CPython will support marking an object as one for which that runtime state will no longer change. Consequently, such an object's refcount will never reach 0, and so the object will never be cleaned up. We call these objects "immortal". (Normally, only a relatively small number of internal objects will ever be immortal.) The fundamental improvement here is that now an object can be truly immutable. Scope - Object immortality is meant to be an internal-only feature. So this proposal does not include any changes to public API or behavior (with one exception). As usual, we may still add some private (yet publicly accessible) API to do things like immortalize an object or tell if one is immortal. Any effort to expose this feature to users would need to be proposed separately. There is one exception to "no change in behavior": refcounting semantics for immortal objects will differ in some cases from user expectations. This exception, and the solution, are discussed below. Most of this PEP focuses on an internal implementation that satisfies the above mandate. However, those implementation details are not meant to be strictly proscriptive. Instead, at the least they are included to help illustrate the technical considerations required by the mandate. The actual implementation may deviate somewhat as long as it satisfies the constraints outlined below. Furthermore, the acceptability of any specific implementation detail described below does not depend on the status of this PEP, unless explicitly specified. For example, the particular details of: * how to mark something as immortal * how to recognize something as immortal * which subset of functionally immortal objects are marked as immortal * which memory-management activities are skipped or modified for immortal objects are not only CPython-specific but are also private implementation details that are expected to change in subsequent versions. Implementation Summary -- Here's a high-level look at the implementation: If an object's refcount matches a very specific value (defined below) then that object is treated as immortal. The CPython C-API and runtime will not modify the refcount (or other runtime state) of an immortal object. Aside from the change to refcounting semantics, there is one other possible negative impact to consider. A naive implementation of the approach described below makes CPython roughly 4% slower. However, the implementation is performance-neutral once known mitigations are applied. Motivation == As noted above, currently all objects are effectively mutable. That includes "immutable" objects like ``str`` instances. This is because every object's refcount is frequently modified as the object is used during execution. This is especially significant for a number of commonly used global (builtin) objects, e.g. ``None``. Such objects are used a lot, both in Python code and internally. That adds up to a consistent high volume of refcount changes. The effective mutability of all Python objects has a concrete impact on parts of the Python community, e.g. projects that aim for scalability like Instragram or the effort to make the GIL per-interpreter. Below we describe several ways in which refcount modification has a real negative effect on such projects. None of that would happen for objects that are truly immutable. Reducing CPU Cache Invalidation --- Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that
