Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On Thu, Nov 2, 2017 at 3:45 PM, Steven D'Aprano wrote: > On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote: > > > This PEP proposes changing function annotations and variable annotations > > so that they are no longer evaluated at function definition time. > > Instead, they are preserved in ``__annotations__`` in string form. > > This means that now *all* annotations, not just forward references, are > no longer validated at runtime and will allow arbitrary typos and > errors: > > def spam(n:itn): # now valid > ... > > Up to now, it has been only forward references that were vulnerable to > that sort of thing. Of course running a type checker should pick those > errors up, but the evaluation of annotations ensures that they are > actually valid (not necessarily correct, but at least a valid name), > even if you happen to not be running a type checker. That's useful. > > Are we happy to live with that change? > Within functions misspellings won't be caught until you invoke a function: def spam(s): return itn(s) # no error unless spam() is called We've lived with this for a long time and generally people seem to be happy with it. The vast majority of code in non-trivial programs (where type annotations are useful) tends to be within functions, so this will only slightly increase the number of things that won't be caught without running tests (or running the type checker). As type checking has become the main use case for annotations, using annotations without a type checker is fast becoming a marginal use case. Type checkers can easily and reliably validate that names in annotations aren't misspelled. > * forward references: when a type hint contains names that have not been > > defined yet, that definition needs to be expressed as a string > > literal; > > After all the discussion, I still don't see why this is an issue. > Strings makes perfectly fine forward references. What is the problem > that needs solving? Is this about people not wanting to type the leading > and trailing ' around forward references? > Let's make a thought experiment. What if every forward reference would require special quoting? Would Python programmers be happy with this? Say, let's use ! as a suffix to mark a forward reference. They make perfectly fine forward references. They are visually pretty unobtrusive (I'm not suggesting $ or other ugly perlisms): def main(): args = parse_args!() # A forward reference do_stuff!(args) # Explicit is better than implicit def parse_args(): ... def do_stuff(args): ... Of course, I'm not seriously proposing this, but this highlights the fact that in normal code forward references "just work" (at least usually), and if we'd require a special quoting mechanism to use them anywhere, Python would look uglier and more inconsistent. Nobody would be happy with this change, even though you'd only have to type a single ! character extra -- that's not a lot work, right? I think that the analogy is reasonable. In type checked code annotations are one of most widely used language features -- it's quite possible to have annotations for almost every function in a code base. This is not a marginal feature, and people expect commonly used features to feel polished and usable, not inconsistent and hacky. It's quite possible that the first type annotated experiment a user writes requires the use of forward references, and this gives a pretty bad first impression -- not unlike how the ! forward reference would make the first impression of using non-type-checked Python pretty awkward. Here are more arguments why literal escapes are a usability problem: 1) It's hard to predict when string quotes are needed. Real-world large code bases tend to have a lot of import cycles, and string literal escapes are often needed within import cycles. However, they aren't always needed. To annotate code correctly, you frequently need to understand how the file you are editing is related to other modules in terms of import cycle structure. In large code bases this can be very difficult to keep in your head, so basically adding forward references becomes a matter of tweak-until-it-works. So either each time you write an annotation, you can look at how imports are structured -- to see whether a particular type needs to be quoted -- or you can guess and hope for the best. This is a terrible user experience and increases cognitive load significantly. Our goal should not be to just have something that technically 'works', as this is a very low standard. I want Python to be easy to use, intuitive and elegant. I don't expect that anybody who has annotated large code bases could consider string literal forward references to be any of those. 2) It's one of the top complaints from users. Even a non-user with a basic understanding of mypy told me what amounts to "Python doesn't have real static typing; forward references make it obvious that types are just an afterthought". 3) It's not
[Python-Dev] PEP 589 discussion (TypedDict) happening at typing-sig@
Hi everyone, I submitted PEP 589 (TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys) for discussion to typing-sig [1]. Here's an excerpt from the abstract of the PEP: PEP 484 defines the type Dict[K, V] for uniform dictionaries, where each value has the same type, and arbitrary key values are supported. It doesn't properly support the common pattern where the type of a dictionary value depends on the string value of the key. This PEP proposes a type constructor typing.TypedDict to support the use case where a dictionary object has a specific set of string keys, each with a value of a specific type. Jukka Lehtosalo [1] https://mail.python.org/mailman3/lists/typing-sig.python.org/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: In support of PEP 649
On Fri, Apr 16, 2021 at 8:27 AM Inada Naoki wrote: > After PEP 563, only `'List[int]'` is practical so we can stop > supporting `List["int"]` and others at some point. > There's a lot of code written before PEP 563 was available (and code that needs to support earlier Python versions) that uses string literal quoting in various places. If Python stops supporting List["X"], it will be a significant backward compatibility break. Some of these string literal escapes are made unnecessary by PEP 563, but there are still cases where string literals are needed that neither PEP 563 nor PEP 649 address. Examples include type aliases, type variable definitions and base classes, but there are others [1]. These aren't evaluated in a type annotation context, so manual string literal escaping would still be needed. > So playing with runtime type will become easier in the future. > > Am I wrong? > In some cases things will become easier, but other common use cases still seem to be unresolved. I haven't seen any proposal that can completely replace all uses of string literal escapes. I guess one option would be to only allow string literal escaping to be used outside type annotations in the future. I think that this could be feasible with a suitable deprecation period. Also, the use of "if TYPE_CHECKING" breaks the runtime use of annotations. PEP 563 and PEP 649 reduce the need for string literal escaping in this use case, but it mostly helps static type checkers. This is pretty common in codebases that use static type checking. My main issue with PEP 649 is that it only addresses a subset of remaining issues, i.e. it doesn't go far enough. It's also not clear to me if PEP 649 (or PEP 563) can be extended to cover the remaining runtime issues, or if we'd need a *third* approach to solve them. Jukka [1] https://www.python.org/dev/peps/pep-0563/#forward-references ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZFMXY7PLLT5OGU4HRRXSQ5GUCLCKJ7W7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 563 in light of PEP 649
On Fri, Apr 16, 2021 at 5:28 PM Łukasz Langa wrote: > [snip] I say "compromise" because as Inada Naoki measured, there's still a > non-zero performance cost of PEP 649 versus PEP 563: > > - code size: +63% > - memory: +62% > - import time: +60% > > > Will this hurt some current users of typing? Yes, I can name you multiple > past employers of mine where this will be the case. Is it worth it for > Pydantic? I tend to think that yes, it is, since it is a significant > community, and the operations on type annotations it performs are in the > sensible set for which `typing.get_type_hints()` was proposed. > Just to give some more context: in my experience, both import time and memory use tend to be real issues in large Python codebases (code size less so), and I think that the relative efficiency of PEP 563 is an important feature. If PEP 649 can't be made more efficient, this could be a major regression for some users. Python server applications need to run multiple processes because of the GIL, and since code objects generally aren't shared between processes (GC and reference counting makes it tricky, I understand), code size increases tend to be amplified on large servers. Even having a lot of RAM doesn't necessarily help, since a lot of RAM typically implies many CPU cores, and thus many processes are needed as well. I can see how both PEP 563 and PEP 649 bring significant benefits, but typically for different user populations. I wonder if there's a way of combining the benefits of both approaches. I don't like the idea of having toggles for different performance tradeoffs indefinitely, but I can see how this might be a necessary compromise if we don't want to make things worse for any user groups. Jukka ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PBJ6MBQIE3DVQUUAO764PIQ3TWGLBS3X/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations
On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger wrote: > Larry Hastings wrote: > > On 8/11/21 12:02 AM, Thomas Grainger wrote: > > > I think as long as there's a test case for something like > > > @dataclass > > > class Node: > > > global_node: ClassVar[Node | None] > > > left: InitVar[Node | None] > > > right: InitVar[None | None] > > > > > > the bug https://bugs.python.org/issue33453 and the current > implementation > https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6... > shows this is a tricky problem > > > The most straightforward workaround for this is to skip the decorator > > syntax. With PEP 649 active, this code should work: > > class Node: > > global_node: ClassVar[Node | None] > > left: InitVar[Node | None] > > right: InitVar[None | None] > > Node = dataclass(Node) > > //arry/ > > the decorator version simply has to work > I also think that it would be unfortunate if the decorator version wouldn't work. This is a pretty basic use case. If we go through a lot of trouble to redesign how type annotations behave, it would be great if we could address as many common pain points as possible. Otherwise I see a risk that we'll have another (third!) redesign in Python 3.12 or 3.13. Jukka ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6U27SVAIHIRTFFJVKJWCNIXEJM6YFO6H/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations
On Wed, Aug 11, 2021 at 2:56 PM Thomas Grainger wrote: > Would: > ``` > @dataclass > class Node: > global_node: __class__ | None > ``` > > "Just work" with co_annotations? > This feels too specialized to me. It would be great to also handle forward references to other classes and cyclic references, which are also somewhat common: @dataclass class NodeA: component: NodeB | None @dataclass class NodeB: component: NodeA | None Another, slightly more complex example would be cyclic references within two modules in an import cycle. For example, NodeA and NodeB could be defined in different modules. The common thing is that the dependency cycle can only be fully resolved after we have created both type objects. The import cycle case is probably less common but I've seen it in real-world code. Jukka ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XFS6VSXGV6QMEY5AB7VO64OZMHAYSDSZ/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] PEP 484 update proposal: annotating decorated declarations
Even if Callable types will soon support keyword arguments, the syntax for Callables will look quite different from function definitions and this inconsistency may hurt readability, at least for more complex signatures. We could work around this by using the def syntax for the declared type of a decorator. For example: @declared_type def session(url: str) -> ContextManager[DatabaseSession]: ... # Explicit '...' @contextmanager def session(url: str) -> Iterator[DatabaseSession]: s = DatabaseSession(url) ... This would be quite similar to how overloads work, so there is a precedent for something like this. We could require or recommend that the declared type comes immediately before the implementation so that the entire definition of a single function would not be too spread out. This won't help if the decorated type is not a callable. We could support this use case by using the normal variable annotation syntax: thing: Decorated # Declared type of 'thing' @decorator def thing() -> int: ... Jukka On Wed, May 10, 2017 at 12:27 AM, Naomi Seyfer wrote: > Stay tuned for the pep that allows callable to take keyword args. > > On May 9, 2017, at 3:59 PM, Brett Cannon wrote: > > The idea seems reasonable to me when viewing type hints as a form of > documentation as it helps remind people how they are expected to call the > final function. > > One worry I do have, though, is Callable doesn't support keyword-only > parameters, so declared_type won't work in all cases without Callable > gaining such support (for those that don't know, Callable didn't start with > that support as Callable has been meant for callback scenarios up to this > point). > > On Tue, 9 May 2017 at 10:21 Guido van Rossum wrote: > >> There's a PR to the peps proposal here: >> https://github.com/python/peps/pull/242 >> >> The full text of the current proposal is below. The motivation for this >> is that for complex decorators, even if the type checker can figure out >> what's going on (by taking the signature of the decorator into account), >> it's sometimes helpful to the human reader of the code to be reminded of >> the type after applying the decorators (or a stack thereof). Much >> discussion can be found in the PR. Note that we ended up having `Callable` >> in the type because there's no rule that says a decorator returns a >> function type (e.g. `property` doesn't). >> >> This is a small thing but I'd like to run it by a larger audience than >> the core mypy devs who have commented so far. There was a brief discussion >> on python-ideas (my original >> <https://mail.python.org/pipermail/python-ideas/2017-April/045548.html>, >> favorable reply >> <https://mail.python.org/pipermail/python-ideas/2017-May/045550.html> by >> Nick, my response >> <https://mail.python.org/pipermail/python-ideas/2017-May/045557.html>). >> >> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan >> Levkivskyi and Jukka Lehtosalo. >> >> If there's no further debate here I'll merge it into the PEP and an >> implementation will hopefully appear in the next version of the typing >> module (also hopefully to be included in CPython 3.6.2 and 3.5.4). >> >> Here's the proposed text (wordsmithing suggestions in the PR please): >> >> +Decorators >> +-- >> + >> +Decorators can modify the types of the functions or classes they >> +decorate. Use the ``decorated_type`` decorator to declare the type of >> +the resulting item after all other decorators have been applied:: >> + >> + from typing import ContextManager, Iterator, decorated_type >> + from contextlib import contextmanager >> + >> + class DatabaseSession: ... >> + >> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]]) >> + @contextmanager >> + def session(url: str) -> Iterator[DatabaseSession]: >> + s = DatabaseSession(url) >> + try: >> + yield s >> + finally: >> + s.close() >> + >> +The argument of ``decorated_type`` is a type annotation on the name >> +being declared (``session``, in the example above). If you have >> +multiple decorators, ``decorated_type`` must be topmost. The >> +``decorated_type`` decorator is invalid on a function declaration that >> +is also decorated with ``overload``, but you can annotate the >> +implementation of the overload series with ``decorated_type``. >> + >> >> -- >> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>) >> ___
