[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
> On 7 Jun 2021, at 05:05, Inada Naoki wrote: > > Hi, folks, > > Since Python 3.8, PyArg_Parse*() APIs and Py_BuildValue() APIs emitted > DeprecationWarning when > '#' format is used without PY_SSIZE_T_CLEAN defined. > In Python 3.10, they raise a RuntimeError, not a warning. Extension > modules can not use '#' format with int. > > So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11? > Extension modules can use '#' format with ssize_t, without > PY_SSIZE_T_CLEAN defined. > > Or should we wait one more version? Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot. I guess this depends a little on what promises the stable ABI makes, the functions are still there but behave differently than before. P.S. I’d be in favour of just dropping PY_SSIZE_T_CLEAN completely (that is use Py_ssize_t unconditionally) to simplify the code base, apart from being slightly worried about the impact on the stable ABI. AFAIK The define was meant as a temporary transition mechanism when Py_ssize_t was introduced in the, by now, ancient past. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QZAHDJFZ5AMMPJE363BRK4XE57QPJO4P/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev wrote: > Its a bit late to complain (and I’m not affected by this myself), but those > functions are part of the stable ABI. The change in 3.10 will break any > extensions that use the stable ABI, use these functions and don’t use > PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given > that the stable ABI doesn’t seem to be used a lot. Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API* change. At the ABI level, what changed if that C extensions built (with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an error on Python 3.10 (for a few specific argument formats using "#"). Ah you are right, it's an incompatible ABI change. It might be possible to keep the backward compatibility at the ABI level by adding a 3rd flavor of "parse" functions: * parse with size_t: no change * parse without size_t: stable ABI * parse without size_t which raises an exception on "#" formats: new Python 3.10 functions It's already painful to have 2 flavors of each functions. Adding a 3rd flavor would make the maintenance burden even worse, whereas Inada-san wants to opposite (remove the 2nd flavor to only have one!). A more general question is: do we still want to keep backward compatibility with Python 3.2 released 10 years ago, or is it ok to start with a new stable ABI which drops backward compatibility with Python 3.5 (for example)? It's maybe time to replace "abi3" with "abi4" in Python 3.10? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ENFYNPCPTEZQSBVB5BA5XYNVFUDUA4GF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
> On 9 Jun 2021, at 11:13, Victor Stinner wrote: > > On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev > wrote: >> Its a bit late to complain (and I’m not affected by this myself), but those >> functions are part of the stable ABI. The change in 3.10 will break any >> extensions that use the stable ABI, use these functions and don’t use >> PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given >> that the stable ABI doesn’t seem to be used a lot. > > Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API* > change. At the ABI level, what changed if that C extensions built > (with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an > error on Python 3.10 (for a few specific argument formats using "#"). > Ah you are right, it's an incompatible ABI change. :-( > > > It might be possible to keep the backward compatibility at the ABI > level by adding a 3rd flavor of "parse" functions: > > * parse with size_t: no change > * parse without size_t: stable ABI > * parse without size_t which raises an exception on "#" formats: new > Python 3.10 functions > > It's already painful to have 2 flavors of each functions. Adding a 3rd > flavor would make the maintenance burden even worse, whereas Inada-san > wants to opposite (remove the 2nd flavor to only have one!). I don’t think it is necessary to introduce a 3th variant, for 3.11+ we could do something like this: * [3.11] Add deprecation warnings in the C headers to the few functions with a “PY_SSIZE_T_CLEAN” variant when “PY_SSIZE_T_CLEAN” is not defined * [3.12+] Change the headers to behave as if “PY_SSIZE_T_CLEAN” is defined, and only keep the non-PY_SSIZE_T_CLEAN variants for the stable ABI (which would include dropping non-PY_SSIZE_T_CLEAN variants for private functions). The PY_SSIZE_T_CLEAN variants would keep their "_PysizeT” suffix in the (stable or unstable) ABI. This wouldn’t allow dropping the non-PY_SSIZE_T_CLEAN variants entirely, at least not until we’re fine with breaking the stable ABI. Another disadvantage is that this might require changes in code that doesn’t even use “#” in format strings in 3.11. > > A more general question is: do we still want to keep backward > compatibility with Python 3.2 released 10 years ago, or is it ok to > start with a new stable ABI which drops backward compatibility with > Python 3.5 (for example)? > > It's maybe time to replace "abi3" with "abi4" in Python 3.10? Personally I’d say it is too soon for that, especially when the CPython speedup project (Guido, Mark, et.al.) is just started and HPy is far from finished. Either project might teach us what changes are needed for a long term stable ABI. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MF52ZHEGWEXPZ77FYQ32XLFFPJ66LSPS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
I think stable ABI keeps symbols, signatures, and memory layouts. I don't think stable ABI keeps all behaviors. For example, Py_CompileString() is stable ABI. When we add `async` keyword, Py_CompileString() starts raising an Error for source code using `async` name. Is it ABI change? I don't think so. I want to drop Py_UNICODE support in Python 3.12. It is another incompatible change in PyArg_Parse*() *API*. Users can not use "u" format after it. It is an incompatible *API* change, but not an *ABI* change. I suspect we had made many incompatible *API* changes in stable ABIs already. If I am wrong, can we stop keeping stable ABI at Python 3.12? Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution. Regards, -- Inada Naoki ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T7CPD4LHAVU5TMMCZ7CXNMOUL3D7ZR5O/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
On Wed, 9 Jun 2021 at 11:36, Inada Naoki wrote: > If I am wrong, can we stop keeping stable ABI at Python 3.12? > Python 4.0 won't come in foreseeable future. Stable ABI blocks Python > evolution. Conversely, the stable ABI allows projects to build cross-version binary wheels. Not many projects do that yet, but it's definitely something we'd like to see more of. Needing new binary builds every version blocks users from testing new versions of Python in advance of the release. Also, I often use the stable ABI when embedding, so that I can replace the Python interpreter without needing to recompile my application and redeploy new binaries everywhere (my use case is pretty niche, though, so I wouldn't like to claim I represent a typical user...). But I do agree that we should either start keeping to the commitments that we made around the stability of the stable ABI, or we should abandon it properly and declare it no longer supported. Having something that sort of works except when we accidentally broke it doesn't help anyone. Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HXOMSUACA5GOH2SFU526R54AJSZGRNXW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
On 09. 06. 21 13:09, Paul Moore wrote: On Wed, 9 Jun 2021 at 11:36, Inada Naoki wrote: If I am wrong, can we stop keeping stable ABI at Python 3.12? Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution. Conversely, the stable ABI allows projects to build cross-version binary wheels. Not many projects do that yet, but it's definitely something we'd like to see more of. Needing new binary builds every version blocks users from testing new versions of Python in advance of the release. [...] But I do agree that we should either start keeping to the commitments that we made around the stability of the stable ABI, or we should abandon it properly and declare it no longer supported. Having something that sort of works except when we accidentally broke it doesn't help anyone. I don't think I made actual commitments regarding the API. The docs do say: "we recommend testing an extension with all minor Python versions it supports". Also, when the API breaks, you get a Python exception; if the ABI does, you get segfaults. So breaking the API is much less severe, but still -- please think about the effect on users who just want their compiled extensions to keep working. On 09. 06. 21 13:09, Paul Moore wrote: Also, I often use the stable ABI when embedding, so that I can replace the Python interpreter without needing to recompile my application and redeploy new binaries everywhere (my use case is pretty niche, though, so I wouldn't like to claim I represent a typical user...). I hope this use case becomes non-niche. I would love it if embedders tell people to just use any Python they have lying around, instead of vendoring it (or more realistically, embedding JS or Lua instead). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3VYTTABB2UB6HVQHASYONSYQDBHDL3OU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.
> On 9 Jun 2021, at 12:28, Inada Naoki wrote: > > I think stable ABI keeps symbols, signatures, and memory layouts. > I don't think stable ABI keeps all behaviors. As often “it depends”. Behaviour is IMHO part of the API/ABI contract. That said, that does not necessarily mean that we cannot change behaviour at all, but that the cost to users for such changes should be taken into account. > > For example, Py_CompileString() is stable ABI. > When we add `async` keyword, Py_CompileString() starts raising an > Error for source code using `async` name. > Is it ABI change? I don't think so. I agree. But its not as easy as “it is not an ABI change because it only changes functionality of a function”. The interface contract of Py_CompileString is that it compiles Python code. If the rules for what valid Python code is change (such as the introduction of ‘async’ as a hard keyword) the function can start to reject input that was accepted earlier. That’s IMHO different from a change to Py_CompileString that starts raising an error unconditionally because we no longer want to expose the API. > > I want to drop Py_UNICODE support in Python 3.12. It is another > incompatible change in PyArg_Parse*() *API*. > Users can not use "u" format after it. It is an incompatible *API* > change, but not an *ABI* change. It is an ABI change: an extensions targetting the stable ABI no longer works due to a change in implementation. That doesn’t necessarily mean the change cannot be made, especially when a deprecation warning is emitted before the feature is removed. > > I suspect we had made many incompatible *API* changes in stable ABIs already. > > If I am wrong, can we stop keeping stable ABI at Python 3.12? > Python 4.0 won't come in foreseeable future. Stable ABI blocks Python > evolution. The annoying part of the stable ABI is that it still exposes some implementation details and behaviour that make it harder to write correct code (such as borrowed references, these can be very convenient but are also easy to misuse). That’s one reason why HPy is an interesting project, even when only targeting CPython. And to be clear: I’m not opposed to the change for the “#” format character and the removal of the “u” format you mention earlier. Ronald > > Regards, > -- > Inada Naoki — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AGTHK7XIS6OPS5H6Z2ZA3XWHILJ4R4OZ/ Code of Conduct: http://python.org/psf/codeofconduct/
