Thanks for the responses. I've replied inline below. -eric
On Mon, Feb 21, 2022 at 9:11 AM Petr Viktorin <encu...@gmail.com> wrote: > > On 19. 02. 22 8:46, Eric Snow wrote: > > Thanks to all those that provided feedback. I've worked to > > substantially update the PEP in response. The text is included below. > > Further feedback is appreciated. > > Thank you! This version is much clearer. I like the PEP more and more! Great! > I've sent a PR with a some typo fixes: > https://github.com/python/peps/pull/2348 Thank you. > > Public Refcount Details > [...] > > As part of this proposal, we must make sure that users can clearly > > understand on which parts of the refcount behavior they can rely and > > which are considered implementation details. Specifically, they should > > use the existing public refcount-related API and the only refcount value > > with any meaning is 0. All other values are considered "not 0". > > Should we care about hacks/optimizations that rely on having the only > reference (or all references), e.g. mutating a tuple if it has refcount > 1? Immortal objects shouldn't break them (the special case simply won't > apply), but this wording would make them illegal. > AFAIK CPython uses this internally, but I don't know how > prevalent/useful it is in third-party code. Good point. As Terry suggested, we could also let 1 have meaning. Regardless, any documented restriction would only apply to users of the public C-API, not to internal code. > > _Py_IMMORTAL_REFCNT > > ------------------- > > > > We will add two internal constants:: > > > > #define _Py_IMMORTAL_BIT (1LL << (8 * sizeof(Py_ssize_t) - 4)) > > #define _Py_IMMORTAL_REFCNT (_Py_IMMORTAL_BIT + (_Py_IMMORTAL_BIT / 2)) > > As a nitpick: could you say this in prose? > > * ``_Py_IMMORTAL_BIT`` has the third top-most bit set. > * ``_Py_IMMORTAL_REFCNT`` has the third and fourth top-most bits set. Sure. > > Immortal Global Objects > > ----------------------- > > > > All objects that we expect to be shared globally (between interpreters) > > will be made immortal. That includes the following: > > > > * singletons (``None``, ``True``, ``False``, ``Ellipsis``, > > ``NotImplemented``) > > * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) > > * all static objects in ``_PyRuntimeState.global_objects`` (e.g. > > identifiers, > > small ints) > > > > All such objects will be immutable. In the case of the static types, > > they will be effectively immutable. ``PyTypeObject`` has some mutable > > start (``tp_dict`` and ``tp_subclasses``), but we can work around this > > by storing that state on ``PyInterpreterState`` instead of on the > > respective static type object. Then the ``__dict__``, etc. getter > > will do a lookup on the current interpreter, if appropriate, instead > > of using ``tp_dict``. > > But tp_dict is also public C-API. How will that be handled? > Perhaps naively, I thought static types' dicts could be treated as > (deeply) immutable, and shared? They are immutable from Python code but not from C (due to tp_dict). Basically, we will document that tp_dict should not be used directly (in the public API) and refer users to a public getter function. I'll note this in the PEP. > Perhaps it would be best to leave it out here and say say "The details > of sharing ``PyTypeObject`` across interpreters are left to another PEP"? > Even so, I'd love to know the plan. What else would you like to know? There isn't much to it. For each of the builtin static types we will keep the relevant mutable state on PyInterpreterState and look it up there in the relevant getters (e.g. __dict__ and __subclasses__). > (And even if these are internals, > changes to them should be mentioned in What's New, for the sake of > people who need to maintain old extensions.) +1 > > Object Cleanup > > -------------- > > > > In order to clean up all immortal objects during runtime finalization, > > we must keep track of them. > > > > For GC objects ("containers") we'll leverage the GC's permanent > > generation by pushing all immortalized containers there. During > > runtime shutdown, the strategy will be to first let the runtime try > > to do its best effort of deallocating these instances normally. Most > > of the module deallocation will now be handled by > > ``pylifecycle.c:finalize_modules()`` which cleans up the remaining > > modules as best as we can. It will change which modules are available > > during __del__ but that's already defined as undefined behavior by the > > docs. Optionally, we could do some topological disorder to guarantee > > that user modules will be deallocated first before the stdlib modules. > > Finally, anything leftover (if any) can be found through the permanent > > generation gc list which we can clear after finalize_modules(). > > > > For non-container objects, the tracking approach will vary on a > > case-by-case basis. In nearly every case, each such object is directly > > accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or > > ``PyInterpreterState`` field. We may need to add a tracking mechanism > > to the runtime state for a small number of objects. > > Out of curiosity: How does this extra work affect in the performance? Is > it part of the 4% slowdown? The slowdown is exclusively due to the change to Py_INCREF() and Py_DECREF(). If there are any objects that must be specially tracked, that will have insignificant performance impact. > And from the other thread: > > On 17. 02. 22 18:23, Eric Snow wrote: > > On Thu, Feb 17, 2022 at 3:42 AM Petr Viktorin <encu...@gmail.com> wrote: > >>>> Weren't you planning a PEP on subinterpreter GIL as well? Do you > want to > >>>> submit them together? > >>> > >>> I'd have to think about that. The other PEP I'm writing for > >>> per-interpreter GIL doesn't require immortal objects. They just > >>> simplify a number of things. That's my motivation for writing this > >>> PEP, in fact. :) > >> > >> Please think about it. > >> If you removed the benefits for per-interpreter GIL, the motivation > >> section would be reduced to is memory savings for fork/CoW. (And lots of > >> performance improvements that are great in theory but sum up to a 4% > loss.) > > > > Sounds good. Would this involve more than a note at the top of the PEP? > > No, a note would work great. If you read the motivation carefully, it's > (IMO) clear that it's rather weak without the other PEP. But that > realization shouldn't come as a surprise to the reader. Having thought about it some more, I don't think this PEP should be strictly bound to per-interpreter GIL. That is certainly my personal motivation. However, we have a small set of users that would benefit significantly, the change is relatively small and simple, and the risk of breaking users is also small. In fact, we regularly have more disruptive changes to internals that do not require a PEP. So it seems like the bar should be pretty low for this one (assuming we get the performance penalty low enough). If it were some massive or broadly impactful (or even clearly public) change then I suppose you could call the motivation weak. However, this isn't that sort of PEP. Honestly, it might not have needed a PEP in the first place if I had been a bit more clear about the idea earlier. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TSHZB7VWSS7XXPCMKXISNXGGCHM7PJOP/ Code of Conduct: http://python.org/psf/codeofconduct/
