[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 22. 03. 22 19:07, Victor Stinner wrote:

Hi,

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):

* https://github.com/python/cpython/pull/32052
* https://github.com/python/cpython/pull/32054

API:

* _PyFrameEvalFunction type
* _PyInterpreterState_GetEvalFrameFunc()
* _PyInterpreterState_SetEvalFrameFunc()
* (undocumented) _PyEval_EvalFrameDefault()

The private API to get/set the eval function *is* documented at:
https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc

I added the Get/Set functions so debuggers don't have to access
directly to the PyInterpreterState structure which has been moved to
the internal C API in Python 3.8.

This API causes me multiple issues:

* It's a private API and I'm trying to remove the private API from the
public C API header files.
* The _PyFrameEvalFunction type is not stable: it got a new "tstate"
parameter in Python 3.9 and the type of the second parameter changed
from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
* These functions use the _PyInterpreterFrame type which is part of
the internal C API.

While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.

I propose to move theses API to the internal header files
(Include/internals/) to clarify that it's not part of the public C API
and that there is no backward compatibility warranty.

The change is being discussed at:
https://bugs.python.org/issue46850

--

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild. I'm not
sure if these ones should be moved to the internal C API. They can be
left unchanged, since they don't use a type only defined by the
internal C API.


PEP 523 clearly meant for these to be used by external tools, but made 
them private. That sounds like a contradiction.


Brett/Dino, what was the intent here?

IMO, if external code should use these, they should lose the leading 
underscore.



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CW3EKBNZCPYPCQMNEXPBV2BKTKI3Y6TX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/22/2022 11:28 PM, Victor Stinner wrote:

On Tue, Mar 22, 2022 at 7:33 PM Steve Dower  wrote:

After a normal deprecation period, yes?


There is no backward compatibility warranty and no deprecation process
for private APIs.


And yet you're asking the question, which means you know these are 
special ;)


The PEP says: "The API is purposefully listed as private to communicate 
the fact that there are no semantic guarantees of the API between Python 
releases."


Absence/presence isn't a semantic guarantee, it's an availability 
guarantee. Code using them should be able to rely on their presence, and 
ideally their prototype (though it seems we've messed that up in the 
past), but shouldn't expect code that worked against 3.8 to also work 
against 3.9 or 3.10.


Perhaps in hindsight, we could have not used the underscore and just 
explicitly described them as being behaviorally unstable between major 
versions. I guess that would have raised exactly the same question though.


The point is, it's a documented API that we've told people they can use. 
We can't simply revoke that without telling people that it's going to 
happen, even if we covered ourselves for there being version changes 
that affect how they need to be used.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF75IDAHS66BKJZDAAQDGOW2IGU2KME5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Enhancing generic type documentation in the standard library

[Luciano Ramalho]

Hello, everyone!

I believe our documentation about types needs another overhaul.

We now have lots of generic types in the standard library, but their
formal type parameters are poorly documented—or not documented at
all—in the standard library documentation.

More importantly: the documentation we have about specific
covariant/contravariant types is now in entries in the `typing` module
that are all deprecated since PEP 585 was implemented in Python 3.9.

Below I present two of many examples where the documentation of a
generic type is not great.

However, if people agree this is a problem, we need to discuss where
and how to put the documentation in a way that is not too disruptive
to users of Python who don't know or don't care about type hints, for
many reasons that we should not judge.

For example, where do we document the fact that `dict` accepts two
invariant formal type parameters, and that `frozenset` accepts one
contravariant type parameter?

What do you think?

Cheers,

Luciano

_
EXAMPLE 1: `Callable` variance is not documented

For example, in the `Callable` type, the `ReturnType` parameter is
covariant, and the `ParameterType` parameters are all contravariant.

But there is no mention of variance in this entry:
https://docs.python.org/3/library/typing.html?highlight=typing#typing.Callable

Also, no mention of the fact that `collections.abc.Callable` is generic here:
https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable

PEP 483—The Theory of Type Hints—is the only official Python doc where
I found the information about the variance of the formal type
parameters of `Callable`. The explanation there is brilliant [0].

[0] https://peps.python.org/pep-0483/#covariance-and-contravariance

Regardless, the intended audience of PEPs is "core developers"—which
is neither a superset nor a subset of "Python devs now using type
hints". We should not rely on PEPs to document features for Python
users in general.

_
EXAMPLE 2: `Generator` variance could be better documented

The entry for `typing.Generator` [1] has this heading:

class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co])

Answer quickly: how many formal type parameters does `Generator`
require? Which are covariant? Which are contravariant?

[1] 
https://docs.python.org/3/library/typing.html?highlight=typing#typing.Generator

Nowhere in that page [1] there's an explanation of the `*_co` and
`*_contra` naming convention, much less their semantics.

Fortunately, the text of the `typing.Generator` entry says: "A
generator can be annotated by the generic type `Generator[YieldType,
SendType, ReturnType]".

Unfortunately, `typing.Generator` is deprecated and will be gone in 5
years or so...

The same issue applies to all the other generic types: builtins
(`dict`, `frozenset`), ABCs, etc.
Their formal type parameters they accept as generics are either
undocumented, or documented in parts of the `typing` module that are
already deprecated.

Thoughts?

--
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UGXWIADYG37N3ML4NBAKYF2C536HR273/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Brett Cannon]

On Wed, Mar 23, 2022 at 2:48 AM Petr Viktorin  wrote:

> On 22. 03. 22 19:07, Victor Stinner wrote:
> > Hi,
> >
> > I proposed two PRs to move the private C API (Include/cpython/) of PEP
> > 523 "Adding a frame evaluation API to CPython" to the internal C API
> > (Include/internals/):
> >
> > * https://github.com/python/cpython/pull/32052
> > * https://github.com/python/cpython/pull/32054
> >
> > API:
> >
> > * _PyFrameEvalFunction type
> > * _PyInterpreterState_GetEvalFrameFunc()
> > * _PyInterpreterState_SetEvalFrameFunc()
> > * (undocumented) _PyEval_EvalFrameDefault()
> >
> > The private API to get/set the eval function *is* documented at:
> >
> https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc
> >
> > I added the Get/Set functions so debuggers don't have to access
> > directly to the PyInterpreterState structure which has been moved to
> > the internal C API in Python 3.8.
> >
> > This API causes me multiple issues:
> >
> > * It's a private API and I'm trying to remove the private API from the
> > public C API header files.
> > * The _PyFrameEvalFunction type is not stable: it got a new "tstate"
> > parameter in Python 3.9 and the type of the second parameter changed
> > from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
> > * These functions use the _PyInterpreterFrame type which is part of
> > the internal C API.
> >
> > While Pyston didn't bring a JIT compiler to Python with PEP 523,
> > debuggers were made faster by using this API. Debuggers like pydevd,
> > debugpy and ptvsd use it.
> >
> > I propose to move theses API to the internal header files
> > (Include/internals/) to clarify that it's not part of the public C API
> > and that there is no backward compatibility warranty.
> >
> > The change is being discussed at:
> > https://bugs.python.org/issue46850
> >
> > --
> >
> > PEP 523 API added more private functions for code objects:
> >
> > * _PyEval_RequestCodeExtraIndex()
> > * _PyCode_GetExtra()
> > * _PyCode_SetExtra()
> >
> > The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> > pydevd debugger. The two others seem to be unused in the wild. I'm not
> > sure if these ones should be moved to the internal C API. They can be
> > left unchanged, since they don't use a type only defined by the
> > internal C API.
>
> PEP 523 clearly meant for these to be used by external tools, but made
> them private. That sounds like a contradiction.
>
> Brett/Dino, what was the intent here?
>

>From the PEP :
"The API is purposefully listed as private to communicate the fact that
there are no semantic guarantees of the API between Python releases."

-Brett


>
> IMO, if external code should use these, they should lose the leading
> underscore.
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OI73PLAILLG75IKKZAADIH4GSOIXFWIQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Enhancing generic type documentation in the standard library

[Brett Cannon]

On Wed, Mar 23, 2022 at 11:03 AM Luciano Ramalho 
wrote:

> Hello, everyone!
>
> I believe our documentation about types needs another overhaul.
>

The SC somewhat agrees! 😉 See
https://mail.python.org/archives/list/typing-...@python.org/thread/TVMQJXOJFOYFPDMQDFG6G4B6J3MLRYKB/
where we have asked for at least the specs to get consolidated into proper
documentation instead of spread out across various PEPs. My personal hope
is this will also lead to better docs overall in a central location instead
of spread out among modules, tools, etc.

-Brett


>
> We now have lots of generic types in the standard library, but their
> formal type parameters are poorly documented—or not documented at
> all—in the standard library documentation.
>
> More importantly: the documentation we have about specific
> covariant/contravariant types is now in entries in the `typing` module
> that are all deprecated since PEP 585 was implemented in Python 3.9.
>
> Below I present two of many examples where the documentation of a
> generic type is not great.
>
> However, if people agree this is a problem, we need to discuss where
> and how to put the documentation in a way that is not too disruptive
> to users of Python who don't know or don't care about type hints, for
> many reasons that we should not judge.
>
> For example, where do we document the fact that `dict` accepts two
> invariant formal type parameters, and that `frozenset` accepts one
> contravariant type parameter?
>
> What do you think?
>
> Cheers,
>
> Luciano
>
> _
> EXAMPLE 1: `Callable` variance is not documented
>
> For example, in the `Callable` type, the `ReturnType` parameter is
> covariant, and the `ParameterType` parameters are all contravariant.
>
> But there is no mention of variance in this entry:
>
> https://docs.python.org/3/library/typing.html?highlight=typing#typing.Callable
>
> Also, no mention of the fact that `collections.abc.Callable` is generic
> here:
>
> https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable
>
> PEP 483—The Theory of Type Hints—is the only official Python doc where
> I found the information about the variance of the formal type
> parameters of `Callable`. The explanation there is brilliant [0].
>
> [0] https://peps.python.org/pep-0483/#covariance-and-contravariance
>
> Regardless, the intended audience of PEPs is "core developers"—which
> is neither a superset nor a subset of "Python devs now using type
> hints". We should not rely on PEPs to document features for Python
> users in general.
>
> _
> EXAMPLE 2: `Generator` variance could be better documented
>
> The entry for `typing.Generator` [1] has this heading:
>
> class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co])
>
> Answer quickly: how many formal type parameters does `Generator`
> require? Which are covariant? Which are contravariant?
>
> [1]
> https://docs.python.org/3/library/typing.html?highlight=typing#typing.Generator
>
> Nowhere in that page [1] there's an explanation of the `*_co` and
> `*_contra` naming convention, much less their semantics.
>
> Fortunately, the text of the `typing.Generator` entry says: "A
> generator can be annotated by the generic type `Generator[YieldType,
> SendType, ReturnType]".
>
> Unfortunately, `typing.Generator` is deprecated and will be gone in 5
> years or so...
>
> The same issue applies to all the other generic types: builtins
> (`dict`, `frozenset`), ABCs, etc.
> Their formal type parameters they accept as generics are either
> undocumented, or documented in parts of the `typing` module that are
> already deprecated.
>
> Thoughts?
>
> --
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> |  Technical Principal at ThoughtWorks
> |  Twitter: @ramalhoorg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UGXWIADYG37N3ML4NBAKYF2C536HR273/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54IOPEZKV5YEGIBKWX5FUJ6RFC5W47YH/
Code of Conduct: http://python.org/psf/codeofconduct/