[Python-Dev] Re: The current state of typing PEPs

[Stephen J. Turnbull]

Executive summary:

The typing-suspicious crowd has a valid complaint about PEPs 563 and
649, but it's not that they weren't warned.

Christopher Barker writes:

 > Annotations can be, and are, used for other things than "typing". I
 > just noticed that PEP 563 apparently deprecated those other uses
 > (well, sort of: "uses for annotations incompatible with the
 > aforementioned PEPs should be considered deprecated")

Not a PEP proponent (or even a typing user), but I thought this had
been made clear long ago.  My understanding is that optional,
incremental type hints are and have always been considered the primary
use case for annotations by the BDFL and AFAICT the SC following the
BDFL.  If compatibility with typing is an issue, then the burden of
implementing that is on the other application.  Typing *might* do
something to help, but it's not obligated to do so.

 > One example is a use case of mine -- I have built a hierarchical
 > object system, built on dataclasses, in which the annotation
 > absolutely has to be an actual type(class) object. PEP 563 will
 > very much break this use case. [...]

 > I'm not entirely sure if PEP 649 would work for my use case --
 > likely it would, at least with modest modifications, rather than a
 > new API.

If PEP 649 works for you, I expect you could work with PEP 563 by
moving Larry's data descriptor concept out of the compiler and into
your library code.

You could also appeal to the "might do something" caveat above.  Yury
suggested providing both the immediately evaluated object (if there
were no forward references, I guess) and the string representation,
which would be backward compatible with your use case.  The idea was
rejected on the basis that it doesn't help with the primary use in
typing, and doesn't remove the need for a __future__ import during the
transition period.  You could ask that that decision be revisited,
since it would ensure that uses of __annotations__ like yours continue
to work.

 > But I suspect I'm not alone in not really noticing that statement,

I'm not surprised people haven't noticed that statement.  I am
surprised that a lot of folks haven't noticed that this comes up
occasionally and the answer every time is "we're not going to go out
of our way to break other use cases, but typing *is* the primary use
case and will take precedence if the question comes up".

I will grant that restricting the type of compiled annotations from
typing.Any to "string serializing an object's AST" could reasonably be
said to be "going out of your way to break other use cases".  You have
a valid beef here, although it's not obvious to me how it should be
resolved.

So I'm not saying that you don't have an interest in the future
semantics of this dunder.  I am +1 on Python providing relief for your
use case.

 > But the fact is that I, among others, have been a bit uncomfortable
 > about the focus on typing in Python for years.

You mean y'all are uncomfortable with the popularity of typing.  You
wouldn't care if it wasn't used outside of hugely complex proprietary
codebases you'll never see.

Thing is, reasoning about programs is What We Do as programmers, and
efforts to make it easier to do that for complex programs, and to
provide software to help with that task, are here to stay and they
will be used in mixed company.  Even in Python.  (And that's why, as
someone who doesn't use typing, I support typing.  Anything that helps
*other people* to write better code, I can get behind.  My code, I'm
going to write as incompetently as ever.[1] ;-)

 > But when issues are raised, we have been repeatedly told that
 > typing is, and always will remain, optional.

This has always been in the context of *your* source code.  For
example, nobody ever claimed you could keep it out of anybody else's
source code (although that's clearly what a lot of those "raising
issues" want!)  Still, features like stub files were provided so
typists could work and play nicely with non-typists if they wanted to
(and many typists want that, themselves).

But dunders are the property of the language (or sometimes the
implementation), and they always have been.  If you use them in a way
that's not documented to work, you're at risk even if it happens to
work now.  PEP 3107 didn't document that anything would work. ;-)


Footnotes: 
[1]  Barry, do not tell Abhilash I wrote that.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2O3JHKJ2CM7OFNGT664WXLN2DKU3YGIS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Steven D'Aprano]

On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote:

> One example is a use case of mine -- I have built a hierarchical object
> system, built on dataclasses, in which the annotation absolutely has to be
> an actual type(class) object. PEP 563 will very much break this use case.

I don't think that's an insurmountable problem. I think that all you 
need is a small class decorator to evaluate the stringified annotations 
back to the real things. The PEP tells us the right way to evaluate 
annotations, so all(?) you need is a decorator to do that to each method 
in your class, and Bob's your uncle.

Maybe PEP 563 could include a decorator in the typing module to 
destringify all the annotations in a class or function?

As far as I can see from a brief scan of the PEP, and based on knowing 
next to nothing about your use-case, the only hypothetical problem might 
be this line in the PEP:

"Consequently, using local state in annotations is no longer possible in 
general."

but it's not entirely clear to me what Łukasz means by that, or whether 
it will affect your use-case.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3RYHEEARXUCK3S6WIZALAU75PEPG7GR2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Rob Cliffe via Python-Dev]

On 25/11/2021 15:15, Stephen J. Turnbull wrote:

Executive summary:

The typing-suspicious crowd has a valid complaint about PEPs 563 and
649, but it's not that they weren't warned.


“As you will no doubt be aware, the plans for development of the 
outlying regions of the Galaxy require the building of a hyperspatial 
express route through your star system, and regrettably your planet is 
one of those scheduled for demolition. The process will take slightly 
less than two of your Earth minutes. Thank you.”


Well we were warned, so everything's fine.
Rob Cliffe___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GKYEYE4XV2L276FMBWYFSIAZHZT3QXSX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Christopher Barker]

First:

At this point, I am not advocating anything in particular. I am just asking
that the SC makes a clear statement about the intention at this point.

>From Barry's email:
"""
This is also a call to you, the folks who both care deeply about typing in
Python, and have a solid understanding of the subject matter (or
willingness to learn) to help us!  There are use cases we don’t know about,
and unclear requirements from both the static and dynamic typing users.
"""
I don't care deeply about typing, and I'm not a static or dynamic typing
user. So this implies that a decision has been made that annotations are
for typing, and typing only.

(not that you're not allowed to use for anything else, of course you are,
but that other uses won;t be taken into account when designing the new
interface)

But I have never seen that clearly stated anywhere. The closest is from PEP
563, where it says:

"""
With this in mind, uses for annotations incompatible with the
aforementioned PEPs should be considered deprecated.
"""

Which pretty much makes the point, but it's a bit subtle -- what does
"incompatible' mean?

On Thu, Nov 25, 2021 at 7:15 AM Stephen J. Turnbull <
stephenjturnb...@gmail.com> wrote:

> The typing-suspicious crowd has a valid complaint about PEPs 563 and
> 649, but it's not that they weren't warned.
>

Well, "typing-suspicious" is part of it, but the real problem is folks that
might be using annotations for non-typing purposes.

As for warning: yes and no -- what's new here is that as far as I know,
this is the first time since the py3 transition that a __future__ import
would be made standard behavior. And you'd have to be paying a lot of
attention to know that was going to happen. And I suspect that only folks
involved in the development of typing tools were paying that much attention.

In the case of the py3 transition -- it was very well known and documented
that lots of things would be changing, and we had a very long transition
period.

But the challenge with this is that there is no way to raise a depreciation
warning for not using a __future__ import. So while we all should have been
testing our code with:

from __future__ import annotations

For the last few versions, I doubt that very many people not developing
typing tools were doing that.

  > Annotations can be, and are, used for other things than "typing". I
>  > just noticed that PEP 563 apparently deprecated those other uses
>  > (well, sort of: "uses for annotations incompatible with the
>  > aforementioned PEPs should be considered deprecated")
>
> Not a PEP proponent (or even a typing user), but I thought this had
> been made clear long ago.  My understanding is that optional,
> incremental type hints are and have always been considered the primary
> use case for annotations by the BDFL and AFAICT the SC following the
> BDFL.


well no -- not "always" -- look at PEP 3107, when annotations were first
introduced:

"By itself, Python does not attach any particular meaning or significance
to annotations."

and it goes on to give non-typing examples.

Since then, the shift from "nothing more than a way of associating
arbitrary Python expressions with various parts of a function at
compile-time" to a system for defining types has been gradual, with a
number of PEPs, and, as far as i can tell, PEP 563 is the  first time that
a change was proposed that would affect other uses for annotations.

Anyway, that is water under the bridge. But the reason I bring it up is
because of all the many thousands of Python users, I pay probably more
attention to changes than most -- I have followed Python-ideas and
Python-dev for years. So I'm guessing that there are other folks out there
that will be surprised if and when the behavior of annotations changes.


> If compatibility with typing is an issue, then the burden of
> implementing that is on the other application.


That does indeed seem to be the current intent. And I'm happy to deal with
that -- I just want it to be clear that I (and others) have to wait and see
how it shakes out, and then figure out what to do, or if I should
contribute to the conversation now. If non-typing use cases of annotations
are still considered important, then we should encourage folks like me to
be part of the conversation.


>   > I'm not entirely sure if PEP 649 would work for my use case --
>  > likely it would, at least with modest modifications, rather than a
>  > new API.
>
> If PEP 649 works for you, I expect you could work with PEP 563 by
> moving Larry's data descriptor concept out of the compiler and into
> your library code.
>

That would indeed require maybe not a change to the API, but certainly some
complex code. But yeah, probably doable one way or another.


>   > But I suspect I'm not alone in not really noticing that statement,
>
>   I am
> surprised that a lot of folks haven't noticed that this comes up
> occasionally and the answer every time is "we're not going to go out
> of our way to break other 

[Python-Dev] Re: The current state of typing PEPs

[Greg Ewing]

On 26/11/21 4:15 am, Stephen J. Turnbull wrote:

My understanding is that optional,
incremental type hints are and have always been considered the primary
use case for annotations by the BDFL


I'm not sure that's true. The way I remember it, back when
annotations were first introduced, the BDFL didn't say that,
or at least didn't say it very clearly. It sounded more like
type hints were just one of many possible uses, and he
encouraged people to experiment. There were even discussions
about coming up with a convention to manage conflicting uses
of annotations in the same code. That wouldn't have happened
if typing were considered the only supported use.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R5TWU6HTOKLCTY2QZDGEAR74TGHZHQQ4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Oscar Benjamin]

On Thu, 25 Nov 2021 at 15:16, Stephen J. Turnbull
 wrote:
>
> Executive summary:
>
> The typing-suspicious crowd has a valid complaint about PEPs 563 and
> 649, but it's not that they weren't warned.
>
> Christopher Barker writes:
>
>  > Annotations can be, and are, used for other things than "typing". I
>  > just noticed that PEP 563 apparently deprecated those other uses
>  > (well, sort of: "uses for annotations incompatible with the
>  > aforementioned PEPs should be considered deprecated")
>
> Not a PEP proponent (or even a typing user), but I thought this had
> been made clear long ago.  My understanding is that optional,
> incremental type hints are and have always been considered the primary
> use case for annotations by the BDFL and AFAICT the SC following the
> BDFL.  If compatibility with typing is an issue, then the burden of
> implementing that is on the other application.  Typing *might* do
> something to help, but it's not obligated to do so.

This was not my understanding of annotations when they were introduced e.g.:
https://www.python.org/dev/peps/pep-3107/#use-cases

As I remember it, a decision about the purpose of annotations was
*explicitly* not made when they were introduced. It was clear that
typing was a major potential use and then at some point (around about
the introduction of the typing module) there seemed to be a shift in
people's understanding of what annotations were for. Eventually that
reached the point that people who were particularly interested in
typing had no memory of the fact that the purpose of annotations had
not really been specified as being about typing in the first place.

It looks to me like Chris has identified in PEP 563 what is
potentially the earliest reference (in an accepted PEP) to the idea
that non-typing uses of annotations are to be discouraged.

--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BERGEZTXG4XSEIJQHQWC3IZLXYEZPZ6C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Rob Cliffe via Python-Dev]

My 2¢ from a position of *extreme ignorance*, not to mention zero 
interest in annotations (all welcome to shoot me down in flames, or 
treat me with contemptuous silence.  But *occasionall*y the "idiot's" 
point of view is worth considering, so here goes).


ISTM that typing/annotations have been allowed to drift without a clear 
end in view, rather like a ship that is steered in one direction by one 
person, then in another by someone else, with nobody knowing what the 
destination port is.  As witness the conflicting views in this thread.  
At the risk of being rude, the phrase "headless chickens" comes to mind.
ISTM that it is time to take a step back and decide on a definite 
policy.  Perhaps a definitive pronouncement from Guido.  Or the SC. Or a 
discussion from all parties that reaches an acceptable conclusion.  Then 
stick to it.  Even if it causes some backward incompatibility - ISTM 
that things have changed so rapidly that this really would be an 
acceptable evil if it results in clarity for the future.  Meanwhile, do 
not change anything, do not approve any PEPs, until this conclusion is 
reached.
I apologise if I am being presumptuous in commenting on a subject I know 
almost nothing about.


/I'm just a soul whose intentions are good,//
//Oh Lord, please don't let me be misunderstood.//
/
Best wishes
Rob Cliffe


On 25/11/2021 23:51, Oscar Benjamin wrote:

On Thu, 25 Nov 2021 at 15:16, Stephen J. Turnbull
  wrote:

Executive summary:

The typing-suspicious crowd has a valid complaint about PEPs 563 and
649, but it's not that they weren't warned.

Christopher Barker writes:

  > Annotations can be, and are, used for other things than "typing". I
  > just noticed that PEP 563 apparently deprecated those other uses
  > (well, sort of: "uses for annotations incompatible with the
  > aforementioned PEPs should be considered deprecated")

Not a PEP proponent (or even a typing user), but I thought this had
been made clear long ago.  My understanding is that optional,
incremental type hints are and have always been considered the primary
use case for annotations by the BDFL and AFAICT the SC following the
BDFL.  If compatibility with typing is an issue, then the burden of
implementing that is on the other application.  Typing *might* do
something to help, but it's not obligated to do so.

This was not my understanding of annotations when they were introduced e.g.:
https://www.python.org/dev/peps/pep-3107/#use-cases

As I remember it, a decision about the purpose of annotations was
*explicitly* not made when they were introduced. It was clear that
typing was a major potential use and then at some point (around about
the introduction of the typing module) there seemed to be a shift in
people's understanding of what annotations were for. Eventually that
reached the point that people who were particularly interested in
typing had no memory of the fact that the purpose of annotations had
not really been specified as being about typing in the first place.

It looks to me like Chris has identified in PEP 563 what is
potentially the earliest reference (in an accepted PEP) to the idea
that non-typing uses of annotations are to be discouraged.

--
Oscar
___
Python-Dev mailing list --python-dev@python.org
To unsubscribe send an email topython-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-dev@python.org/message/BERGEZTXG4XSEIJQHQWC3IZLXYEZPZ6C/
Code of Conduct:http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7NRLCIENXLWBPM3EA67ZIN3X3K25DMEY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Guido van Rossum]

On Thu, Nov 25, 2021 at 3:49 PM Greg Ewing 
wrote:

> On 26/11/21 4:15 am, Stephen J. Turnbull wrote:
> > My understanding is that optional,
> > incremental type hints are and have always been considered the primary
> > use case for annotations by the BDFL
>
> I'm not sure that's true. The way I remember it, back when
> annotations were first introduced, the BDFL didn't say that,
> or at least didn't say it very clearly. It sounded more like
> type hints were just one of many possible uses, and he
> encouraged people to experiment. There were even discussions
> about coming up with a convention to manage conflicting uses
> of annotations in the same code. That wouldn't have happened
> if typing were considered the only supported use.
>

My memory is also hazy, but I'm quite sure that *in my mind* annotations
were intended as a compromise between conflicting proposals for *typing*.
We didn't have agreement on the syntax or semantics, but we did know we
wanted to do something with types eventually. Some folks wanted to enforce
types at runtime. Others wanted to use them to generate faster code. Yet
others wanted types to be checked by the compiler. The term "gradual
typing" wasn't invented (or hadn't reached our community) yet, and offline
static type checking wasn't something we had thought of either (I think).
But it was clear that typing would have to be optional.

Fortunately the current situation is that there are few people who want to
use annotations for non-typing, but there are still conflicting use cases,
in particular offline static type checking on the one hand (which started
with mypy and PEP 484) and some runtime use of types on the other hand (
pydantic  being the main example
IIUC).

Tools like pydantic seem to have adopted the *notation* of PEP 484 fully,
but their *interpretation* is different. This was explicitly provided for
by PEP 484 (the design there carefully ensures that annotations are fully
introspectable) but it wasn't anticipated to be a big use case, hence the
mistaken belief at the time PEP 563 was accepted that stringifying
annotations would be sufficient for the introspection use case.

So I'm in agreement with Stephen, but since he left out any mention of
offline static checking, his observation isn't helping much.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VCER56HAJRGST6SNV7V44B5C6E46FJVS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Steven D'Aprano]

On Fri, Nov 26, 2021 at 12:15:10AM +0900, Stephen J. Turnbull wrote:

> I will grant that restricting the type of compiled annotations from
> typing.Any to "string serializing an object's AST" could reasonably be
> said to be "going out of your way to break other use cases".  You have
> a valid beef here, although it's not obvious to me how it should be
> resolved.

I don't think that's what PEP 563 says. Annotations are not *restricted* 
to only be strings, it is merely that when the function or class object 
is built, the annotations are left as strings.

So we can post-process annotations and destringify them:

>>> from __future__ import annotations
>>> def func(arg:float) -> int:
... pass
... 
>>> func.__annotations__
{'arg': 'float', 'return': 'int'}
>>> func.__annotations__['return'] = int
>>> func.__annotations__['arg'] = float
>>> func.__annotations__
{'arg': , 'return': }


There may be scoping issues to be sorted out, but I don't think they are 
insurmountable.


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/24PKPJTI5SIES32C2KZVF6R6D3RLF6BI/
Code of Conduct: http://python.org/psf/codeofconduct/