[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:

> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum  wrote:
>
> I'm not convinced that advertising an API as being Unstable (in the
> documentation?) is going to solve any kind of problem. People love to
> use private APIs, and they do it simply because it's technically
> possible :-) At the end of the day, we have to help them updating
> their code, otherwise we (at least my Red Hat team) cannot update
> Python.
>
> I designed the internal C API to be more annoying to be used (need to
> define a macro, need more specific #include) in the hope that people
> will think twice before using it :-)
>


The changes you've made have been excellent, and the existing 3 categories
(stable public ABI, stable public API, unstable internal API) cover the
vast majority of cases.

The final case that isn't quite covered yet is to offer a "semi-stable" API
category for use cases that are intrinsically coupled to implementation
details that may change between feature releases, but should remain stable
within a release series.

The concrete motivating example for the new category is the extra APIs you
need in order to provide an alternative eval loop implementation.

The internal API category doesn't properly cover that case, as the APIs
there are free to change even in maintenance releases, and setting
Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
need.

Regular public functions may work in some cases, but aren't necessarily
practical in others (such as exposing the internal frame details for use in
alternative eval loops).

>From an implementation PoV, my own suggestion would be to define a new API
tier with an opt-in macro rather than relying solely on documentation or
naming conventions.

For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
"Include/cpython/semi_stable/" (I don't like "unstable" as potential
terminology here, since the internal API is already unstable - we're
splitting the difference between that and the long term stability of the
full public API)

Cheers,
Nick.



> Victor
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZBNJTAXS6TWQY7QH5H5XZS4CP64ZQAUU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Nick Coghlan]

On Sun, 3 Apr 2022, 12:58 pm Steven D'Aprano,  wrote:

>
> I haven't considered pickling, or deep-copying. I don't think there is
> any way to get access to the underlying dict and modify it, except
> perhaps via ctypes, so I think it is as immutable as it is possible to
> get from Python code. Feel free to take that as a challenge to break it.
>


MappingProxyType does expose the underlying mapping to the other operand
during type coercion for binary operators (and trying to prevent that
causes enough problems that we decided the problem wasn't worth fixing):
https://bugs.python.org/issue43838


For a lot of use cases, MappingProxyType is already a "good enough"
immutable dict, but it does have a definite discoverability problem.

Cheers,
Nick.



>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DN5BIOAZSG7YFVVHVRVF66MJC5YNJON/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Victor Stinner]

PEP 603 adds a frozenmap type to collections, implemented as HAMT.

For a read-only *dict*, I proposed PEP 416 "Add a frozendict builtin
type" in 2012. It was rejected.
https://peps.python.org/pep-0416/

The outcome of this PEP was the addition of the types.MappingProxy
type (previously, it already existed but it was somehow hidden:
type(int.__dict__)).

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IQMBELHLSWSEYQI7BEPTJTYP47BWTBUV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Descriptions in unittest and avoiding confusion

[Jason R. Coombs]

For the edification of all involved, this post summarizes a somewhat surprising 
behavior in unittest around docstrings.

In bpo-46126, I reported an issue where I’d 
observed that CPython developers were avoiding the use of docstrings in 
unittests due to what was perceived as a masking of crucial information.

On further investigation, it turns out that with the “descriptions” setting of 
the TextTestRunner is set to True (the default), unittest will emit the first 
line of a test’s docstring (if present) in addition to the location of the 
test. As a result, for tests that have docstrings, the output varies depending 
on this setting. Verbose output with docstrings and descriptions enabled:

test_entry_points_unique_packages 
(test.test_importlib.test_metadata_api.APITests)
Entry points should only be exposed for the first package ... ERROR


Without docstrings or with descriptions disabled:

test_entry_points_unique_packages 
(test.test_importlib.test_metadata_api.APITests) ... ERROR


The output with the docstrings is more descriptive, providing more context and 
detail about the intention of the failed test. Because of the additional 
detail, however, unittest has elected to use a newline between the test 
location and the description, meaning the test result no longer appears on the 
same line as the test location, and thus if one were to grep for the test name, 
the result would be omitted, and if one were to grep for ERROR, the test name 
would be omitted.

As part of the investigation, I published In Python, use docstrings or 
comments? 
exploring the motivations for and value added by allowing docstrings in test 
functions.

It’s still an open consideration whether 
the unittest UX should format descriptions in the output differently to provide 
more consistent output while still allowing docstrings.

Based on this information, CPython will most likely (pending 
GH-32128) continue to use the 
default behavior (descriptions enabled) but will also allow for docstrings in 
its own tests.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DAJSMSVL4S2EQVJD3HCG3KI6AIIXTURM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Guido van Rossum]

+1 to Nick's analysis and proposal. I had been mulling over my own reply
but this just about covers it.

On Sun, Apr 3, 2022 at 6:29 AM Nick Coghlan  wrote:

> On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:
>
>> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum 
>> wrote:
>>
>> I'm not convinced that advertising an API as being Unstable (in the
>> documentation?) is going to solve any kind of problem. People love to
>> use private APIs, and they do it simply because it's technically
>> possible :-) At the end of the day, we have to help them updating
>> their code, otherwise we (at least my Red Hat team) cannot update
>> Python.
>>
>> I designed the internal C API to be more annoying to be used (need to
>> define a macro, need more specific #include) in the hope that people
>> will think twice before using it :-)
>>
>
>
> The changes you've made have been excellent, and the existing 3 categories
> (stable public ABI, stable public API, unstable internal API) cover the
> vast majority of cases.
>
> The final case that isn't quite covered yet is to offer a "semi-stable"
> API category for use cases that are intrinsically coupled to implementation
> details that may change between feature releases, but should remain stable
> within a release series.
>
> The concrete motivating example for the new category is the extra APIs you
> need in order to provide an alternative eval loop implementation.
>
> The internal API category doesn't properly cover that case, as the APIs
> there are free to change even in maintenance releases, and setting
> Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
> need.
>
> Regular public functions may work in some cases, but aren't necessarily
> practical in others (such as exposing the internal frame details for use in
> alternative eval loops).
>
> From an implementation PoV, my own suggestion would be to define a new API
> tier with an opt-in macro rather than relying solely on documentation or
> naming conventions.
>
> For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
> "Include/cpython/semi_stable/" (I don't like "unstable" as potential
> terminology here, since the internal API is already unstable - we're
> splitting the difference between that and the long term stability of the
> full public API)
>
> Cheers,
> Nick.
>
>
>
>> Victor
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TBWSFAGJNCDEVNK4VSQTEUKYX6FRBUYA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Itamar O]

On Sun, Apr 3, 2022 at 6:32 AM Nick Coghlan  wrote:

> On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:
>
>> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum 
>> wrote:
>>
>> I'm not convinced that advertising an API as being Unstable (in the
>> documentation?) is going to solve any kind of problem. People love to
>> use private APIs, and they do it simply because it's technically
>> possible :-) At the end of the day, we have to help them updating
>> their code, otherwise we (at least my Red Hat team) cannot update
>> Python.
>>
>> I designed the internal C API to be more annoying to be used (need to
>> define a macro, need more specific #include) in the hope that people
>> will think twice before using it :-)
>>
>
>
> The changes you've made have been excellent, and the existing 3 categories
> (stable public ABI, stable public API, unstable internal API) cover the
> vast majority of cases.
>
> The final case that isn't quite covered yet is to offer a "semi-stable"
> API category for use cases that are intrinsically coupled to implementation
> details that may change between feature releases, but should remain stable
> within a release series.
>
> The concrete motivating example for the new category is the extra APIs you
> need in order to provide an alternative eval loop implementation.
>
> The internal API category doesn't properly cover that case, as the APIs
> there are free to change even in maintenance releases, and setting
> Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
> need.
>
> Regular public functions may work in some cases, but aren't necessarily
> practical in others (such as exposing the internal frame details for use in
> alternative eval loops).
>
> From an implementation PoV, my own suggestion would be to define a new API
> tier with an opt-in macro rather than relying solely on documentation or
> naming conventions.
>
> For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
> "Include/cpython/semi_stable/" (I don't like "unstable" as potential
> terminology here, since the internal API is already unstable - we're
> splitting the difference between that and the long term stability of the
> full public API)
>
> Cheers,
> Nick.
>

+1 for an official "semi stable API tier".
It's already the case that essentially anything in Python can change,
it's just a question of how quickly and with how much friction.
Public APIs need to go through a multi-version deprecation cycle
(https://peps.python.org/pep-0387/#making-incompatible-changes).
Private internal APIs can (theoretically) change without notice between
patch releases.
There's a missing tier for APIs that can change without notice between
feature releases,
but are guaranteed(*) to be backwards compatible within a feature release,
and the PEP 523 frame evaluation API is an excellent example for this need
(maybe any newly added API should always go through this stage for a few
releases?).

Even though the docs (
https://docs.python.org/3.10/c-api/intro.html#include-files) explicitly
call out that _Py-prefixed APIs are internal and should not be used by
extensions,
this isn't the case in practice, so introducing the 3-tier concept could be
an opportunity
to clean up this situation a bit.
What exactly should be the naming conventions per tier, and the names of
the tiers,
is bikeshedding that should happen after there's agreement about the tiers
concept :-)

(*) "guaranteed" with exceptions of course (e.g. security or other critical
issue)


>
>
>
>> Victor
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZBNJTAXS6TWQY7QH5H5XZS4CP64ZQAUU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DQ5AUS3UBQGJV6YLQ4CJ5F5M7RWFC7DY/
Code of Conduct: http://python.org/psf/codeofconduct/