[Python-Dev] Re: In support of PEP 649

[Samuel Colvin]

There are a number of issues around recursive types, in general PEP 563
doesn't make a big difference either way in this case.

I think you mean something like

from pydantic import BaseModel
from typing import Optional

class Foo(BaseModel):
x: int
foo: Optional['Foo']

Foo.update_forward_refs()
print(Foo(x=1, foo={'x': 1}))

The only difference with "from __future__ import annotations" is you can
unquote Foo, e.g. Optional[Foo]. In all cases you still need
"Foo.update_forward_refs()" since Foo is not defined while creating Foo.

You could also use ForwardRef: "FooType =
ForwardRef('Foo')...Optional[FooType]", but that's slightly more verbose
and doesn't add anything.

(*Sidenote:* as I'm writing this, I realise you could know what Foo is
while creating Foo (same name -> reference to itself) and perhaps
automatically call update_forward_refs() at the end of
ModelMetaclass.__new__(), but you still need the
public update_forward_refs() for more complex cases; not least bodging
around PEP 563 scopes, see here

.)

Samuel
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XSDPDMFRCW4726NN5ZN2UVXHWVZCNH7X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Samuel Colvin]

>
> As far as I know, both Pydantic and marshmallow start using annotation
> for runtime type after PEP 563 is accepted. Am I right?


Not quite, pydantic was released in June 2017 (see HN post:
https://news.ycombinator.com/item?id=14477222) and always used annotations,
PEP 563 was created in September 2017, a few months later.

I can't speak about Marshmallow, back then I don't think it used
annotations at all, but maybe it does now.

When PEP 563 is accepted, there are some tools using runtime type in
> annotation, but they are not adopted widely yet.


Yes, that's completely fair.

Samuel
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FVFL7FO4RPGSJPP5QJ2U4KBU6IWCT2YX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [Release manager team communication] 3.10 feature freeze is 2 weeks away

[Pablo Galindo Salgado]

Hi everyone,

This is a friendly reminder from the release manager team that Python 3.10
feature freeze is two weeks away (Monday, 3 May 2021).

Please, be aware of the following:

* No new features or improvements should be merged after feature freeze.
>From the devguide (https://devguide.python.org/devcycle/#beta) :
> After a first beta release is published, no new features are accepted.
Only bug fixes can now be committed. This is when core developers should
> concentrate on the task of fixing regressions and other new issues filed
by users who have downloaded the alpha and beta release

* If your change involves some platform-specific behaviour or has a
non-trivial amount of C code, make sure you run the buildbots
in your Pull Request by using the "test-with-buildbots" label (
https://discuss.python.org/t/now-you-can-test-a-pr-with-the-buildbots-before-merging/2966
).
Alternatively you could check the buildbots post-merge in the buildbot
server:
https://buildbot.python.org/all/#/builders?tags=%2B3.x&tags=%2Bstable
This is very important because if problems are detected at the time of the
release, the release management team may have to revert
the changes and therefore those will not be included in Python 3.10.

* The master branch will be renamed to main (check our previous
communication about how this will be done and how you should
proceed:
https://mail.python.org/archives/list/python-dev@python.org/thread/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
)

If you have any questions or concerns, please contact me as soon as
possible.

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SIJQE3BZ6ICCGNJWFR4YR65BQBJJZZAZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [Release manager team communication] 3.10 feature freeze is 2 weeks away

[Pablo Galindo Salgado]

Hi everyone,

This is a friendly reminder from the release manager team that Python 3.10
feature freeze is two weeks away (Monday, 3 May 2021).

Please, be aware of the following:

* No new features or improvements should be merged after feature freeze.
>From the devguide (https://devguide.python.org/devcycle/#beta) :
> After a first beta release is published, no new features are accepted.
Only bug fixes can now be committed. This is when core developers should
> concentrate on the task of fixing regressions and other new issues filed
by users who have downloaded the alpha and beta release

* If your change involves some platform-specific behaviour or has a
non-trivial amount of C code,
*make sure you run the buildbotsin your Pull Request* by using the
"test-with-buildbots" label (
https://discuss.python.org/t/now-you-can-test-a-pr-with-the-buildbots-before-merging/2966
).
Alternatively you could check the buildbots post-merge in the buildbot
server:
https://buildbot.python.org/all/#/builders?tags=%2B3.x&tags=%2Bstable
This is *very important* because if problems are detected at the time of
the release,
*the release management team may have to revertthe changes* and therefore
those will not be included in Python 3.10.

* The master branch will be renamed to main (check our previous
communication about how this will be done and how you should
proceed:
https://mail.python.org/archives/list/python-dev@python.org/thread/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
)

If you have any questions or concerns, please contact me as soon as
possible.

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L4FJDHA7JGPX64HX6L7IDAJ5R4F6KR6G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Larry Hastings]

I noticed something this morning: there's another way in which Inada 
Naoki's benchmark here is--possibly?--unrealistic.


As mentioned, his benchmark generates a thousand functions, each of 
which takes exactly three parameters, and each of those parameters 
randomly chooses one of three annotations.  In current trunk (not in my 
branch, I'm behind), there's an optimization for stringized annotations 
that compiles the annotations into a tuple, and then when you pull out 
__annotations__ on the object at runtime it converts it into a dict on 
demand.


This means that even though there are a thousand functions, they only 
ever generate one of nine possible tuples for these annotation tuples.  
And here's the thing: our lovely marshal module is smart enough to 
notice that these tuples /are/ duplicates, and it'll throw away the 
duplicates and replace them with references to the original.


Something analogous /could/ happen in the PEP 649 branch but currently 
doesn't.  When running Inada Noki's benchmark, there are a total of nine 
possible annotations code objects.  Except, each function generated by 
the benchmark has a unique name, and I incorporate that name into the 
name given to the code object (f"{function_name}.__co_annotations__"). 
Since each function name is different, each code object name is 
different, so each code object /hash/ is different, and since they 
aren't /exact/ duplicates they are never consolidated.


Inada Naoki has suggested changing this, so that all the annotations 
code objects have the same name ("__co_annotations__").  If we made that 
change, I'm pretty sure the code size delta in this synthetic benchmark 
would drop.  I haven't done it because the current name of the code 
object might be helpful in debugging, and I'm not convinced this would 
have an effect in real-world code.


But... would it?  Someone, and again I think it's Inada Naoki, suggests 
that in real-world applications, there are often many, many functions in 
a single module that have identical signatures.  The annotation-tuples 
optimization naturally takes advantage of that.  PEP 649 doesn't.  
Should it? Would this really be beneficial to real-world code bases?


Cheers,


//arry/


On 4/16/21 12:26 PM, Larry Hastings wrote:



Please don't confuse Inada Naoki's benchmark results with the effect 
PEP 649 would have on a real-world codebase.  His artifical benchmark 
constructs a thousand empty functions that take three parameters with 
randomly-chosen annotations--the results provides some insights but 
are not directly applicable to reality.


PEP 649's effects on code size / memory / import time are contingent 
on the number of annotations and the number of objects annotated, not 
the overall code size of the module.  Expressing it that way, and 
suggesting that Python users would see the same results with 
real-world code, is highly misleading.


I too would be interested to know the effects PEP 649 had on a 
real-world codebase currently using PEP 563, but AFAIK nobody has 
reported such results.



//arry/

On 4/16/21 11:05 AM, Jukka Lehtosalo wrote:
On Fri, Apr 16, 2021 at 5:28 PM Łukasz Langa > wrote:


[snip] I say "compromise" because as Inada Naoki measured,
there's still a non-zero performance cost of PEP 649 versus PEP 563:

- code size: +63%
- memory: +62%
- import time: +60%


Will this hurt some current users of typing? Yes, I can name you
multiple past employers of mine where this will be the case. Is
it worth it for Pydantic? I tend to think that yes, it is, since
it is a significant community, and the operations on type
annotations it performs are in the sensible set for which
`typing.get_type_hints()` was proposed.


Just to give some more context: in my experience, both import time 
and memory use tend to be real issues in large Python codebases (code 
size less so), and I think that the relative efficiency of PEP 563 is 
an important feature. If PEP 649 can't be made more efficient, this 
could be a major regression for some users. Python server 
applications need to run multiple processes because of the GIL, and 
since code objects generally aren't shared between processes (GC and 
reference counting makes it tricky, I understand), code size 
increases tend to be amplified on large servers. Even having a lot of 
RAM doesn't necessarily help, since a lot of RAM typically implies 
many CPU cores, and thus many processes are needed as well.


I can see how both PEP 563 and PEP 649 bring significant benefits, 
but typically for different user populations. I wonder if there's a 
way of combining the benefits of both approaches. I don't like the 
idea of having toggles for different performance tradeoffs 
indefinitely, but I can see how this might be a necessary compromise 
if we don't want to make things worse for any user groups.


Jukka

___
Python-Dev mailing list --p

[Python-Dev] Re: PEP 563 in light of PEP 649

[Larry Hastings]

Oops: where I said nine, I should have said, twenty-seven.  3-cubed.  
Should have had my coffee /before/ posting.  Carry on!



//arry/

On 4/19/21 10:51 AM, Larry Hastings wrote:



I noticed something this morning: there's another way in which Inada 
Naoki's benchmark here is--possibly?--unrealistic.


As mentioned, his benchmark generates a thousand functions, each of 
which takes exactly three parameters, and each of those parameters 
randomly chooses one of three annotations.  In current trunk (not in 
my branch, I'm behind), there's an optimization for stringized 
annotations that compiles the annotations into a tuple, and then when 
you pull out __annotations__ on the object at runtime it converts it 
into a dict on demand.


This means that even though there are a thousand functions, they only 
ever generate one of nine possible tuples for these annotation 
tuples.  And here's the thing: our lovely marshal module is smart 
enough to notice that these tuples /are/ duplicates, and it'll throw 
away the duplicates and replace them with references to the original.


Something analogous /could/ happen in the PEP 649 branch but currently 
doesn't.  When running Inada Noki's benchmark, there are a total of 
nine possible annotations code objects.  Except, each function 
generated by the benchmark has a unique name, and I incorporate that 
name into the name given to the code object 
(f"{function_name}.__co_annotations__"). Since each function name is 
different, each code object name is different, so each code object 
/hash/ is different, and since they aren't /exact/ duplicates they are 
never consolidated.


Inada Naoki has suggested changing this, so that all the annotations 
code objects have the same name ("__co_annotations__").  If we made 
that change, I'm pretty sure the code size delta in this synthetic 
benchmark would drop.  I haven't done it because the current name of 
the code object might be helpful in debugging, and I'm not convinced 
this would have an effect in real-world code.


But... would it?  Someone, and again I think it's Inada Naoki, 
suggests that in real-world applications, there are often many, many 
functions in a single module that have identical signatures.  The 
annotation-tuples optimization naturally takes advantage of that.  PEP 
649 doesn't.  Should it?  Would this really be beneficial to 
real-world code bases?


Cheers,


//arry/


On 4/16/21 12:26 PM, Larry Hastings wrote:



Please don't confuse Inada Naoki's benchmark results with the effect 
PEP 649 would have on a real-world codebase.  His artifical benchmark 
constructs a thousand empty functions that take three parameters with 
randomly-chosen annotations--the results provides some insights but 
are not directly applicable to reality.


PEP 649's effects on code size / memory / import time are contingent 
on the number of annotations and the number of objects annotated, not 
the overall code size of the module.  Expressing it that way, and 
suggesting that Python users would see the same results with 
real-world code, is highly misleading.


I too would be interested to know the effects PEP 649 had on a 
real-world codebase currently using PEP 563, but AFAIK nobody has 
reported such results.



//arry/

On 4/16/21 11:05 AM, Jukka Lehtosalo wrote:
On Fri, Apr 16, 2021 at 5:28 PM Łukasz Langa > wrote:


[snip] I say "compromise" because as Inada Naoki measured,
there's still a non-zero performance cost of PEP 649 versus PEP 563:

- code size: +63%
- memory: +62%
- import time: +60%


Will this hurt some current users of typing? Yes, I can name you
multiple past employers of mine where this will be the case. Is
it worth it for Pydantic? I tend to think that yes, it is, since
it is a significant community, and the operations on type
annotations it performs are in the sensible set for which
`typing.get_type_hints()` was proposed.


Just to give some more context: in my experience, both import time 
and memory use tend to be real issues in large Python codebases 
(code size less so), and I think that the relative efficiency of PEP 
563 is an important feature. If PEP 649 can't be made more 
efficient, this could be a major regression for some users. Python 
server applications need to run multiple processes because of the 
GIL, and since code objects generally aren't shared between 
processes (GC and reference counting makes it tricky, I understand), 
code size increases tend to be amplified on large servers. Even 
having a lot of RAM doesn't necessarily help, since a lot of RAM 
typically implies many CPU cores, and thus many processes are needed 
as well.


I can see how both PEP 563 and PEP 649 bring significant benefits, 
but typically for different user populations. I wonder if there's a 
way of combining the benefits of both approaches. I don't like the 
idea of having toggles for different performance tradeoffs 
indefinitely, but I can s

[Python-Dev] Should Python typing leverage PEP 263 as a pre-processor step?

[Luciano Ramalho]

How about leveraging the `# coding=` hook that exists
since 2001 to enable the alternative syntax some are advocating for
type hints?

PEP 263—Defining Python Source Code Encodings
https://www.python.org/dev/peps/pep-0263/

I've seen experiments in the wild using that to support syntax
extensions to Python. Just for fun, years ago I wrote a POC for
Sucuri—a Python dialect with the keywords in Portuguese. The
pre-processor simply replaced the Portuguese keywords with the English
ones.

Cheers,

Luciano


-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XRLAOOIJZRPDT2AW6LW4UVBEJD5NKLV5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should Python typing leverage PEP 263 as a pre-processor step?

[Guido van Rossum]

We had a similar thing at Dropbox, where `# coding: pyxl` would enable a
preprocessor that allowed HTML embedded in the Python code. It translated
this to function calls and string literals.

There were however several drawbacks:

- Installing the codec is a bit tricky, and if you don't have it the code
can't run
- Other tooling won't know about the new syntax (someone went through
heroic efforts to write a PyCharm extension for pyxl but it got stale very
quickly)
- The codec slows down the import process
- Keeping line numbers accurate in the codec is painful
- Occasionally, when debugging mysteries, you end up needing to see the
source code after preprocessing, which requires yet another custom tool

In the end the project was withdrawn and we switched to more mainstream
templating solutions.

I suspect that many of the issues with pyxl would also plague using this
approach as a way to customize typing syntax, and I don't think it would be
an improvement over the status quo.




On Mon, Apr 19, 2021 at 11:26 AM Luciano Ramalho 
wrote:

> How about leveraging the `# coding=` hook that exists
> since 2001 to enable the alternative syntax some are advocating for
> type hints?
>
> PEP 263—Defining Python Source Code Encodings
> https://www.python.org/dev/peps/pep-0263/
>
> I've seen experiments in the wild using that to support syntax
> extensions to Python. Just for fun, years ago I wrote a POC for
> Sucuri—a Python dialect with the keywords in Portuguese. The
> pre-processor simply replaced the Portuguese keywords with the English
> ones.
>
> Cheers,
>
> Luciano
>
>
> --
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> |  Technical Principal at ThoughtWorks
> |  Twitter: @ramalhoorg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XRLAOOIJZRPDT2AW6LW4UVBEJD5NKLV5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VKAQSVBTJZMTZCUYV6L2QZEPE24QH6DQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Ethan Furman]

On 4/19/21 10:51 AM, Larry Hastings wrote:

Something analogous /could/ happen in the PEP 649 branch but currently doesn't.  When running Inada Noki's benchmark, 
there are a total of nine possible annotations code objects.  Except, each function generated by the benchmark has a 
unique name, and I incorporate that name into the name given to the code object (f"{function_name}.__co_annotations__"). 
Since each function name is different, each code object name is different, so each code object /hash/ is different, and 
since they aren't /exact/ duplicates they are never consolidated.


I hate anonymous functions, so the name is very important to me.  The primary code base I work on does have hundreds of 
methods with the same signature -- unfortunately, many of the also have the same name (four levels of super() calls is 
not unusual, and all to the same read/write/create parent methods from read/write/create child methods).  In such a case 
would the name make a meaningful difference?


Or maybe the name can be store when running in debug mode, and not stored with 
-O ?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XWZUVJSZMUB5Q4FVMKJWP2XK45LRTG7J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Antoine Pitrou]

On Mon, 19 Apr 2021 13:37:56 -0700
Ethan Furman  wrote:
> On 4/19/21 10:51 AM, Larry Hastings wrote:
> 
> > Something analogous /could/ happen in the PEP 649 branch but currently 
> > doesn't.  When running Inada Noki's benchmark, 
> > there are a total of nine possible annotations code objects.  Except, each 
> > function generated by the benchmark has a 
> > unique name, and I incorporate that name into the name given to the code 
> > object (f"{function_name}.__co_annotations__"). 
> > Since each function name is different, each code object name is different, 
> > so each code object /hash/ is different, and 
> > since they aren't /exact/ duplicates they are never consolidated.  
> 
> I hate anonymous functions, so the name is very important to me.

You are unlikely to notice the name of the code object underlying
__co_annotations__, aren't you?

> Or maybe the name can be store when running in debug mode, and not stored 
> with -O ?

Almost nobody uses -O.  Optimizations that are enabled only in -O are
useless.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BVPWTWVKFEL4XNSWSELWY5DW62DNWY64/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Larry Hastings]

On 4/19/21 1:37 PM, Ethan Furman wrote:

On 4/19/21 10:51 AM, Larry Hastings wrote:

Something analogous /could/ happen in the PEP 649 branch but 
currently doesn't.  When running Inada Noki's benchmark, there are a 
total of nine possible annotations code objects.  Except, each 
function generated by the benchmark has a unique name, and I 
incorporate that name into the name given to the code object 
(f"{function_name}.__co_annotations__"). Since each function name is 
different, each code object name is different, so each code object 
/hash/ is different, and since they aren't /exact/ duplicates they 
are never consolidated.


I hate anonymous functions, so the name is very important to me. The 
primary code base I work on does have hundreds of methods with the 
same signature -- unfortunately, many of the also have the same name 
(four levels of super() calls is not unusual, and all to the same 
read/write/create parent methods from read/write/create child 
methods).  In such a case would the name make a meaningful difference?


Or maybe the name can be store when running in debug mode, and not 
stored with -O ?



I think it needs to have /a/ name.  But if it made a difference, perhaps 
it could use f"{function_name}.__co_annotations__" normally, and simply 
"__co_annotations__" with -O.


Note also that this is the name of the annotations code object, although 
I think the annotations function object reuses the name too.  Anyway, 
under normal circumstances, the Python programmer would have no reason 
to interact directly with the annotations code/function object, so it's 
not likely it will affect them one way or another.  The only time they 
would see it would be, say, if the calculation of an annotation threw an 
exception, in which case it seems like seeing 
f"{function_name}.__co_annotations__" in the traceback might be a 
helpful clue in diagnosing the problem.



I'd want to see some real numbers before considering changes here.  If 
it has a measurable and beneficial effect on real-world code, okay! 
let's change it!  But my suspicion is that it doesn't really matter.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZAPCP4MFDOF34E3G2TWAVY7JUQRHDOOB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Rob Cliffe via Python-Dev]

On 19/04/2021 22:01, Antoine Pitrou wrote:

Almost nobody uses -O. Optimizations that are enabled only in -O are
useless.
Data point: I use -O.¹  Not frequently, not usually, but I have a few 
large² programs that I add features to from time to time and will 
doubtless continue to do so.  I need their __debug__-enabled diagnostics 
and asserts during development and testing.  When I run them "for real", 
I use -O.
Of course, since I never use annotations, type hints or whatever, this 
may not be very relevant to this thread.

Best wishes
Rob Cliffe

¹ Which makes me "almost nobody" - well, I knew that already. 😁
² Your idea of "large" may not be the same as mine - I'm talking a few 
thousand lines.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y7QURU4NES4WCBSHOGXZV4B4ZHD2UC27/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]

[Nathaniel Smith]

On Mon, Apr 5, 2021 at 9:48 AM Irit Katriel  wrote:
> On Mon, Apr 5, 2021 at 11:01 AM Nathaniel Smith  wrote:
>> - I'm uncomfortable with how in some contexts we treat EG's as placeholders 
>> for the contained exceptions, and other places we treat them like a single 
>> first-class exceptions. (Witness all the feedback about "why not just catch 
>> the ExceptionGroup and handle it by hand?", and imagine writing the docs 
>> explaining all the situations where that is or isn't a good idea and the 
>> pitfalls involved...) If we could somehow pick one and stick to it then I 
>> think that would make it easier for users to grasp. (My gut feeling is that 
>> making them pure containers is better, which to me is why it makes sense for 
>> them to be @final and why I keep hoping we can figure out some better way 
>> for plain 'except' and EGs to interact.)
>
>
> I'm not sure what you mean by "a placeholder". An exception group is an 
> exception, and except* is a new syntax that helps you manipulate exception 
> groups.  I don't think the confusion you mention was due to the fact that 
> except can catch EGs, but rather due to the fact that the simplest examples 
> don't show you why except is not good enough, and why we need except*.
>
> Suppose we did as you suggest and made exception group a container which is 
> not exception. Now suppose that an exception is raised in an except* block. 
> What is the context of this exception? Do we change the requirement that an 
> exception's context is always an exception?
>
> How do you raise an exception group if it's not an Exception? Do we go back 
> to allowing random objects being raised?

Ah, right. I'm talking about the conceptual/semantic level, not the
implementation. For the implementation, backcompat certainly means we
need some single object that can represent the whole group of
exceptions, so that it can be returned by sys.exc_info(), C code can
access it through the tstate, etc. But the more important question is
how we explain this thing to users, and make it easy for them to do
the right thing. Is the EG the exception they're trying to catch or
otherwise work with? or is the EG just a detail that most users should
ignore because they're focused on the leaf exceptions inside?

For the concurrency use case (asyncio/trio/etc.), it's all about the
leaf exceptions. ExceptionGroup([SomeError()]) and SomeError() both
mean exactly the same thing, and treating them differently is
basically always a bug. The ExceptionGroup object doesn't tell you
anything about what went wrong, like normal exceptions do. It just
tells you that there was some frame in between where the exception was
raised and where it was caught where some other exception *could* have
happened, but didn't. You *can* give 'except ExceptionGroup' a
meaning, but that meaning doesn't make sense -- it's like saying "I
want to catch any exceptions that was raised by code that *could* have
raised a different exception instead" or "I want to catch all
exceptions whose traceback contains an entry that's a certain subclass
of TracebackType". Similarly, it doesn't make sense to attach error
strings to an EG, or define custom subclasses, etc.. Ideally, plain
'except' would (somehow) handle 'EG([SomeError()])' and 'SomeError()'
in exactly the same way; and if it doesn't, then using plain 'except'
in concurrent code is going to usually be a bug.

For other use cases, it does make sense to think of EG as a regular
exception. Like, if Hypothesis wants to report that it ran some tests
and there were failures, then modelling that as a single
HypothesisError seems like a nice API. You never need 'except*'
semantics to catch part of a HypothesisError. 'except HypothesisError'
is totally sensible. The only real value you get from the PEP is that
if you *don't* catch the HypothesisError, then the default traceback
machinery can now automatically include info about the individual test
failures.

If we were starting from scratch, I don't think this would be a big
conflict. I think the PEP would focus 100% on the concurrency case.
That's the one that's really difficult to solve without language
support. And, if you solve that, then there's a very natural way to
handle the Hypothesis case too: do 'raise HypothesisError from
EG(individual failures)'. That makes it explicit that HypothesisError
is a single exception that should be handled as a unit, while still
letting you introspect the individual exceptions and including them in
the default traceback output. (In fact Trio uses this trick right now,
in it's TCP connection code [1]. We report a single
OSError("connection failed"), but then include all the details about
each individual failure in its __cause__.)

The actual problem is that we're not starting from scratch; we're
trying to retrofit the concurrency-style semantics onto a system
that's currently completely focused around the Hypothesis-style
semantics. So the PEP ends up with a kind of mish-mash of the two
appro

[Python-Dev] Re: PEP 563 in light of PEP 649

[Jelle Zijlstra]

El lun, 19 abr 2021 a las 14:17, Larry Hastings ()
escribió:

> On 4/19/21 1:37 PM, Ethan Furman wrote:
>
> On 4/19/21 10:51 AM, Larry Hastings wrote:
>
> Something analogous /could/ happen in the PEP 649 branch but currently
> doesn't.  When running Inada Noki's benchmark, there are a total of nine
> possible annotations code objects.  Except, each function generated by the
> benchmark has a unique name, and I incorporate that name into the name
> given to the code object (f"{function_name}.__co_annotations__"). Since
> each function name is different, each code object name is different, so
> each code object /hash/ is different, and since they aren't /exact/
> duplicates they are never consolidated.
>
>
> I hate anonymous functions, so the name is very important to me.  The
> primary code base I work on does have hundreds of methods with the same
> signature -- unfortunately, many of the also have the same name (four
> levels of super() calls is not unusual, and all to the same
> read/write/create parent methods from read/write/create child methods).  In
> such a case would the name make a meaningful difference?
>
> Or maybe the name can be store when running in debug mode, and not stored
> with -O ?
>
>
> I think it needs to have *a* name.  But if it made a difference, perhaps
> it could use f"{function_name}.__co_annotations__" normally, and simply
> "__co_annotations__" with -O.
>
> Note also that this is the name of the annotations code object, although I
> think the annotations function object reuses the name too.  Anyway, under
> normal circumstances, the Python programmer would have no reason to
> interact directly with the annotations code/function object, so it's not
> likely it will affect them one way or another.  The only time they would
> see it would be, say, if the calculation of an annotation threw an
> exception, in which case it seems like seeing
> f"{function_name}.__co_annotations__" in the traceback might be a helpful
> clue in diagnosing the problem.
>
If the line numbers in the traceback are right, I'm not sure the function
name would make much of a difference.

>
> I'd want to see some real numbers before considering changes here.  If it
> has a measurable and beneficial effect on real-world code, okay! let's
> change it!  But my suspicion is that it doesn't really matter.
>
>
> Cheers,
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZAPCP4MFDOF34E3G2TWAVY7JUQRHDOOB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V3N5ILAED6CWOEXL6L7XHRTUIS3AKN4M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should Python typing leverage PEP 263 as a pre-processor step?

[Glenn Linderman]

On 4/19/2021 12:44 PM, Guido van Rossum wrote:
We had a similar thing at Dropbox, where `# coding: pyxl` would enable 
a preprocessor that allowed HTML embedded in the Python code. It 
translated this to function calls and string literals.


There were however several drawbacks:

- Installing the codec is a bit tricky, and if you don't have it the 
code can't run
- Other tooling won't know about the new syntax (someone went through 
heroic efforts to write a PyCharm extension for pyxl but it got stale 
very quickly)

- The codec slows down the import process
- Keeping line numbers accurate in the codec is painful
- Occasionally, when debugging mysteries, you end up needing to see 
the source code after preprocessing, which requires yet another custom 
tool


In the end the project was withdrawn and we switched to more 
mainstream templating solutions.


I suspect that many of the issues with pyxl would also plague using 
this approach as a way to customize typing syntax, and I don't think 
it would be an improvement over the status quo.



How does this "similar thing" compare to the recently announced imphook 
module?



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QYSPZ2NNJQNEKXD2F72AFQLCX7XGO4LD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should Python typing leverage PEP 263 as a pre-processor step?

[Guido van Rossum]

On Mon, Apr 19, 2021 at 8:16 PM Glenn Linderman 
wrote:

> On 4/19/2021 12:44 PM, Guido van Rossum wrote:
> > We had a similar thing at Dropbox, where `# coding: pyxl` would enable
> > a preprocessor that allowed HTML embedded in the Python code. It
> > translated this to function calls and string literals.
> >
> > There were however several drawbacks:
> >
> > - Installing the codec is a bit tricky, and if you don't have it the
> > code can't run
> > - Other tooling won't know about the new syntax (someone went through
> > heroic efforts to write a PyCharm extension for pyxl but it got stale
> > very quickly)
> > - The codec slows down the import process
> > - Keeping line numbers accurate in the codec is painful
> > - Occasionally, when debugging mysteries, you end up needing to see
> > the source code after preprocessing, which requires yet another custom
> > tool
> >
> > In the end the project was withdrawn and we switched to more
> > mainstream templating solutions.
> >
> > I suspect that many of the issues with pyxl would also plague using
> > this approach as a way to customize typing syntax, and I don't think
> > it would be an improvement over the status quo.
>
>
> How does this "similar thing" compare to the recently announced imphook
> module?
>

For one, pyxl is a better name. :-)

Seriously, as long as the purpose is to allow using a different grammar
(for part of the file) that is then transpiled to proper Python, all but
the first issue apply exactly the same.

Getting the hook to run (my first bullet) actually sounds *more*
complicated, because you have to include code in your `__main__` module
that installs the hook(s).

But I have zero experience with imphook, I just skimmed its PyPI home page,
which has an example that calls
```
imphook.add_import_hook(hook, (".conf",))
```

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EJOEHORIVSGNKJON26WPGZF7AAVOUZGD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally

[Larry Hastings]

As long as I'm gravedigging old conversations...!  Remember this one, 
also from January of this year?  Here's a link to the thread in the 
c.l.p-d Mailman archive.  The first message in the thread is a good 
overview of the problem:


   
https://mail.python.org/archives/list/python-dev@python.org/thread/AWKVI3NRCHKPIDPCJYGVLW4HBYTEOQYL/


Here's kind of where we left it:

On 1/12/21 7:48 PM, Guido van Rossum wrote:
On Tue, Jan 12, 2021 at 6:35 PM Larry Hastings > wrote:


On 1/12/21 5:28 PM, Brett Cannon wrote:

The other thing to keep in mind is we are talking about every
module, class, and function getting 64 bytes ... which I bet
isn't that much.


Actually it's only every module and class.  Functions don't have
this problem because they've always stored __annotations__
internally--meaning, peeking in their __dict__ doesn't work, and
they don't support inheritance anyway.  So the number is even
smaller than that.

If we can just make __annotations__ default to an empty dict on
classes and modules, and not worry about the memory consumption,
that goes a long way to cleaning up the semantics.


I would like that very much. And the exception for functions is 
especially helpful.



First of all, I've proposed a function that should also help a lot:

   https://bugs.python.org/issue43817

The function will be called inspect.get_annotations(o).  It's like 
typing.get_type_hints(o) except less opinionated.  This function would 
become the best practice for everybody who wants annotations**, like so:


   import inspect
   if hasattr(inspect, "get_annotations"):
    how_i_get_annotations = inspect.get_annotations
   else:
    # do whatever it was I did in Python 3.9 and before...


** Everybody who specifically wants /type hints/ should instead call 
typing.get_type_hints(), and good news!, /that/ function has existed for 
several versions now.  So they probably already /do/ call it.



I'd still like to add a default empty __annotations__ dict to all 
classes and modules for Python 3.10, for everybody who doesn't switch to 
using this as-yet-unwritten inspect.get_annotations() function.  The 
other changes I propose in that thread (e.g. deleting __annotations__ 
always throws TypeError) would be nice, but honestly they aren't high 
priority.  They can wait until after Python 3.10.  Just these these two 
things (inspect.get_annotations() and always populating __annotations__ 
for classes and modules) would go a long way to cleaning up how people 
examine annotations.


Long-term, hopefully we can fold the desirable behaviors of 
inspect.get_annotations() into the language itself, at which point we 
could probably deprecate the function.  That wouldn't be until a long 
time from now of course.



Does this need a lot of discussion, or can I just go ahead with the bpo 
and PR and such?  I mean, I'd JFDI, as Barry always encourages, but 
given how much debate we've had over annotations in the last two weeks, 
I figured I should first bring it up here.



Happy two-weeks'-notice,


//arry/

p.s. I completely forgot about this until just now--sorry.  At least I 
remembered before Python 3.10b1!


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J4LZEIZTYZQWGIM5VZGNMQPWB5ZWVEXP/
Code of Conduct: http://python.org/psf/codeofconduct/