[Python-Dev] Re: In support of PEP 649

[Inada Naoki]

On Fri, Apr 16, 2021 at 6:03 AM Bluenix  wrote:
>
> Please accept PEP 649!
>
> Python's type hinting has become so much more useful than originally thought, 
> and without this change much of that will be hindered. For example (you 
> already know about Pydantic and FastAPI) 
> [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows 
> you to use typehinting to specify how arguments should be converted. Take the 
> following code:
>
> ```py
> import discord
> from discord.ext import commands
>
> bot = commands.Bot(command_prefix='>')
>
> @bot.command()
> # discord.py reads the typehints and converts the arguments accordingly
> async def reply(ctx, member: discord.Member, *, text: str):  # ctx is always 
> passed
> await ctx.send(f'{member.mention}! {text}')
>
> bot.run('token')
> ```
>
> I must say, this is extremely ergonomic design! Don't break it :)

Are you sure about PEP 563 break it and discord.py can not fix it?

As far as my understanding, PEP 563 don't hurt this use case so much:

* This use case evaluates runtime type information only once. So
eval() overhead is not a problem.
* Currently, annotation is very very complex and varies. For example,
List[int] will be `List[int]`, `List['int']`, `'List[int]'`,
`'List["int"]'`, `List[ForwardRef('int')]` etc...
  After PEP 563, only `'List[int]'` is practical so we can stop
supporting `List["int"]` and others at some point.
  So playing with runtime type will become easier in the future.

Am I wrong?

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NSZGWCABWFYWZZTNCE5VE5ZVC3OUJCNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Antoine Pitrou]

On Thu, 15 Apr 2021 16:12:57 -0700
Brett Cannon  wrote:
> 
> So I don't think version comparison occurs often enough to be in the
> stdlib, and the fact that an external project exists which isn't interested
> in being put in the stdlib suggests to me it isn't worth doing.
> 
> But that's a separate topic to discuss at the language summit. :)

Ah, I should submit a question then.

> > That specific way is a Python standard (PEP 440). Having the
> > functionality in the stdlib would encourage people to use it. Not
> > having it in the stdlib encourages people to use adhoc version parsing,
> > or worse, naive string comparison.
> 
> I don't know if I draw the same line since the packaging community is
> relying on 'packaging' without issue for this exact thing.

Right, but the functionality is useful generally, not only for the
packaging generally.

> But this thread is about standardizing on __version__ which then led to
> wanting to bring in some Version object into the stdlib to make that more
> useful. So to me they are tied together, else a separate thread should
> probably be started if a new module *just *for version parsing support is
> desired.

You're right, this thread was hijacked somewhat for the Version object
discussion.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U2KO5AZRIHNXWKRQ5YEY3CGW4FZ6SAEU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Antoine Pitrou]

On Fri, 16 Apr 2021 10:35:19 +0900
Inada Naoki  wrote:
> 
> And personally, I love static typing but I don't use type hint for
> performance/memory usage reason.
> I spend much effort to optimize PEP 563 to minimize type hinting overhead.
> So it's very sad that if I can not use type hinting when I can drop
> Python 3.9 support.

AFAIU, we're not talking about dropping PEP 563 (yet), but about making
it opt-in again.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NVJOKSXRFEUKWHF4USVOB2U3YTCL3PZL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Typing syntax and ecosystem

[Sebastian Rittau]

Am 16.04.21 um 03:21 schrieb Barry Warsaw:
Actually, I think it’s time for a comprehensive guide to type 
annotations. Just anecdotally, I was trying to annotate a library of 
mine and was having an impossible time of it, until a chat with Guido 
lead me to @typing.overload. That solved my problem intuitively and 
easily, but I just didn’t know about it. Right now, there’s 
information spread out all over the place, the stdlib documentation, 
tool documentation, StackOverflow :D etc. It’s a complicated topic 
that I think a comprehensive guide, a tutorial, etc. could really help 
with.


As a typeshed maintainer, I agree. Currently, the typing documentation 
is spread over various PEPs, the typing module documentation, and the 
mypy documentation. We also have a style guide for stub in the typeshed 
documentation. That said, at least for type stubs, we are working on a 
PEP (still looking for a sponsor) that is supposed to be a comprehensive 
document. [1][2]


 - Sebastian

[1] 
https://mail.python.org/archives/list/typing-...@python.org/message/VNMC3JPWWP3O4TNMJZMSEH6UU5BPN4ZJ/

[2] https://github.com/srittau/type-stub-pep/blob/type-stub-pep/pep-.rst


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SDEQGGWKL5MHBPZURDVQSV6VC22MWY3E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Sebastian Rittau]

On Sun, Apr 11, 2021 at 1:31 PM Barry Warsaw > wrote:

[snip]

This is something the SC has been musing about, but as it’s not a
fully formed idea, I’m a little hesitant to bring it up.  That
said, it’s somewhat relevant: We wonder if it may be time to in a
sense separate the typing syntax from Python’s regular syntax. 
TypeGuards are a case where if typing had more flexibility to
adopt syntax that wasn’t strictly legal “normal” Python, maybe
something more intuitive could have been proposed.  I wonder if
the typing-sig has discussed this possibility (in the future, of
course)?

I am strongly in favor of diverging type annotation syntax from Python 
syntax. Currently, type annotations are a very useful tool, but often 
clunky to use. Enhancements have been made, but design space is limited 
when working within existing Python syntax. Type annotations have a 
different set of rules, needs, and constraints than general-purpose 
Python code. This is similar to other domain specific languages like 
regular expressions. Ideally, Python itself would not check the syntax 
of annotations, except as needed for determining the end of an 
annotation. PEP 563 is a step in that direction.


As far as I understand the arguments against PEP 563 and in favor of PEP 
649 mostly boil down to "annotations are used outside of typing, these 
uses would need to use eval() in the future and eval() is slow". (At 
least from a user's perspective, there are more arguments from a Python 
maintainer's perspective that I can't comment on.) Are there benchmarks 
to verify that using eval() has a non-negligible effect for this use 
case? Overall, I don't find this to be a compelling argument when 
compared to the problem that PEP 649 would close all design space for 
type annotation syntax enhancements.


 - Sebastian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RWLOLMWLPZ3O5VO7OQZSHTQGR4ICXDJV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Serhiy Storchaka]

24.03.21 11:54, redrad...@gmail.com пише:
> What about of using modern C++ in developing CPython ?
> 
> With C++ we can get the following benefits:
> 1) Readability - less code, most code is hidden by abstraction without losing 
> performance
> 2) Maintainability - no code duplication in favor of using reusable classes
> 3) RAII - Resource Acquisition Is Initialization, predictable allocation and 
> free resources
> ...

The drawbacks:

1. You can only use C++ internally. All API should be in pure C. It
reduces possible benefits.
2. If you use C++, even internally, you have to link with libstdc++ and
use C++ compatible linker.
3. C++ is much much more complex than pure C. It drastically reduces the
number of available contributors and maintainers.

Also note that the C code it already written. The C++ code has yet to be
written, with chances to introduce new bugs during rewriting and
distracting resources from other tasks.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4X4QEE7IFR5WGKBSHMSNVVYWCS3VX4JW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Tin Tvrtković]

Guido:
> It seems a little disingenuous to claim discussions about annotations
don’t
> concern you when you’re actively using them (for typing, no less, in the
> case of pydantic). And I am sure a project as popular (by their own
> description) as pydantic will find a way forward if PEP 649 is rejected,
> despite overdramatized claims.

I also maintain a library that uses type annotations in a runtime context
(cattrs), but in a slightly different way than Pydantic. My project is much
less popular than Pydantic (to be expected, since it only deals with
de/serialization and the class definition layer is attrs/dataclasses), and
the first issue filed on my bug tracker about future annotations was
created at the end of 2019, so that's when I started thinking about it.

To be honest, supporting stringified annotations was a lot of work to do in
an efficient way (and that's the main reason cattrs has two converter
classes now), but there was plenty of time and I simply did the work. The
new design is even faster than the old one.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KER3PFQO2MV3LAWI2JHIANT7UVVT6KT5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Jukka Lehtosalo]

On Fri, Apr 16, 2021 at 8:27 AM Inada Naoki  wrote:

>   After PEP 563, only `'List[int]'` is practical so we can stop
> supporting `List["int"]` and others at some point.
>

There's a lot of code written before PEP 563 was available (and code that
needs to support earlier Python versions) that uses string literal quoting
in various places. If Python stops supporting List["X"], it will be a
significant backward compatibility break. Some of these string literal
escapes are made unnecessary by PEP 563, but there are still cases where
string literals are needed that neither PEP 563 nor PEP 649 address.
Examples include type aliases, type variable definitions and base classes,
but there are others [1]. These aren't evaluated in a type annotation
context, so manual string literal escaping would still be needed.


>   So playing with runtime type will become easier in the future.
>
> Am I wrong?
>

In some cases things will become easier, but other common use cases still
seem to be unresolved. I haven't seen any proposal that can completely
replace all uses of string literal escapes. I guess one option would be to
only allow string literal escaping to be used outside type annotations in
the future. I think that this could be feasible with a suitable deprecation
period.

Also, the use of "if TYPE_CHECKING" breaks the runtime use of annotations.
PEP 563 and PEP 649 reduce the need for string literal escaping in this use
case, but it mostly helps static type checkers. This is pretty common in
codebases that use static type checking. My main issue with PEP 649 is that
it only addresses a subset of remaining issues, i.e. it doesn't go far
enough. It's also not clear to me if PEP 649 (or PEP 563) can be extended
to cover the remaining runtime issues, or if we'd need a *third* approach
to solve them.

Jukka

[1] https://www.python.org/dev/peps/pep-0563/#forward-references
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZFMXY7PLLT5OGU4HRRXSQ5GUCLCKJ7W7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Can we change python -W option and PYTHONWARNINGS to use a regex for the message?

[Victor Stinner]

Hi,

I propose to change the -W command line option and the PYTHONWARNINGS
environment variable to use the message as a regular expression in
Python 3.10. Or does anyone have a reason to keep the current behavior
as it is?

I created https://bugs.python.org/issue43862 for this change.

--

Python provides two ways to specify warnings filters:

* -W command line option: can be used multiple times
* PYTHONWARNINGS environment variable: can contain multiple options
separated by commas

While the Python API warnings.filterwarnings(action, message="", ...)
uses the message as a regular expression, -W and PYTHONWARNINGS
require to match *exactly* the *whole* message.

For example, if you only want to ignore the new distutils deprecation
warning, you must write exactly:

$ ./python -X dev -W 'ignore:The distutils package is deprecated and
slated for removal in Python 3.12. Use setuptools or check PEP 632 for
potential alternatives:DeprecationWarning' -c 'import distutils'

I use -X dev to show DeprecationWarning, or you can also use -Wdefault
if you prefer.

If the deprecation warning changes in Python or if you have a single
typo, the warning is not ignored. Example with a typo ("3.13" rather
than "3.12"):

$ ./python -X dev -W 'ignore:The distutils package is deprecated and
slated for removal in Python 3.13. Use setuptools or check PEP 632 for
potential alternatives:DeprecationWarning' -c 'import distutils'
:1: DeprecationWarning: The distutils package is deprecated
and slated for removal in Python 3.12. Use setuptools or check PEP 632
for potential alternatives

The PYTHONWARNINGS has another limitation: you cannot specify a
message if it contains a comma (","). Hopefully, Python doesn't raise
warnings containing comma, right? Well... Just one example:

Lib/distutils/sysconfig.py:554:warnings.warn('SO is
deprecated, use EXT_SUFFIX', DeprecationWarning, 2)

You cannot only ignore the message:

$ PYTHONWARNINGS='ignore:SO is deprecated, use
EXT_SUFFIX:DeprecationWarning' ./python -c 'import sys;
print(sys.warnoptions); print(len(sys.warnoptions))'
Invalid -W option ignored: invalid action: 'use EXT_SUFFIX'
['ignore:SO is deprecated', ' use EXT_SUFFIX:DeprecationWarning']
2

You can only try to use "module" and "lineno" parameters of a warning
filter, which are more fragile and hard to use to use.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JMKLA4RUJLTORBPPTM4BWL76VNNMKYVG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Can we change python -W option and PYTHONWARNINGS to use a regex for the message?

[Ivan Pozdeev via Python-Dev]

It'll probably be easier to change the warnings filter semantic 
(https://docs.python.org/3/library/warnings.html#the-warnings-filter)
to which those values are directly passed.

On 16.04.2021 16:07, Victor Stinner wrote:

Hi,

I propose to change the -W command line option and the PYTHONWARNINGS
environment variable to use the message as a regular expression in
Python 3.10. Or does anyone have a reason to keep the current behavior
as it is?

I created https://bugs.python.org/issue43862 for this change.

--

Python provides two ways to specify warnings filters:

* -W command line option: can be used multiple times
* PYTHONWARNINGS environment variable: can contain multiple options
separated by commas

While the Python API warnings.filterwarnings(action, message="", ...)
uses the message as a regular expression, -W and PYTHONWARNINGS
require to match *exactly* the *whole* message.

For example, if you only want to ignore the new distutils deprecation
warning, you must write exactly:

$ ./python -X dev -W 'ignore:The distutils package is deprecated and
slated for removal in Python 3.12. Use setuptools or check PEP 632 for
potential alternatives:DeprecationWarning' -c 'import distutils'

I use -X dev to show DeprecationWarning, or you can also use -Wdefault
if you prefer.

If the deprecation warning changes in Python or if you have a single
typo, the warning is not ignored. Example with a typo ("3.13" rather
than "3.12"):

$ ./python -X dev -W 'ignore:The distutils package is deprecated and
slated for removal in Python 3.13. Use setuptools or check PEP 632 for
potential alternatives:DeprecationWarning' -c 'import distutils'
:1: DeprecationWarning: The distutils package is deprecated
and slated for removal in Python 3.12. Use setuptools or check PEP 632
for potential alternatives

The PYTHONWARNINGS has another limitation: you cannot specify a
message if it contains a comma (","). Hopefully, Python doesn't raise
warnings containing comma, right? Well... Just one example:

Lib/distutils/sysconfig.py:554:warnings.warn('SO is
deprecated, use EXT_SUFFIX', DeprecationWarning, 2)

You cannot only ignore the message:

$ PYTHONWARNINGS='ignore:SO is deprecated, use
EXT_SUFFIX:DeprecationWarning' ./python -c 'import sys;
print(sys.warnoptions); print(len(sys.warnoptions))'
Invalid -W option ignored: invalid action: 'use EXT_SUFFIX'
['ignore:SO is deprecated', ' use EXT_SUFFIX:DeprecationWarning']
2

You can only try to use "module" and "lineno" parameters of a warning
filter, which are more fragile and hard to use to use.

Victor


--
Regards,
Ivan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ADUDHMNJIYERRA5MHF4GGB2OXV2XJC37/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Can we change python -W option and PYTHONWARNINGS to use a regex for the message?

[Serhiy Storchaka]

16.04.21 16:07, Victor Stinner пише:
> I propose to change the -W command line option and the PYTHONWARNINGS
> environment variable to use the message as a regular expression in
> Python 3.10. Or does anyone have a reason to keep the current behavior
> as it is?
> 
> I created https://bugs.python.org/issue43862 for this change.

It is known issue, but changing it is would break compatibility. The
warning message can contain characters which have special meaning in
regular expressions (e.g. "(" and "?"). Event if it can be parsed as a
regular expression, it can stop to work.

It would be more safe to add some flag which control whether the message
is a pattern or literal. For example, if it starts with "/" it is a
pattern (no warning message or module name starts with "/").

Module name also should support regular expressions.

Note also that specifying message or module name in the -W option or the
PYTHONWARNINGS environment variable causes importing the re module at
the start of the interpreter, and it nontrivially increases the starting
time.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UCFRLK3HCVLJ25VM3OL45XVXC4KMFLTR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Can we change python -W option and PYTHONWARNINGS to use a regex for the message?

[Victor Stinner]

The warnings.filterwarnings() function uses a regular expression for
message and module arguments. My request is only about the Python
command line interface.

By the way, an user requested to use a regex for the module field of
-W option and PYTHONWARNINGS env var:
https://bugs.python.org/issue34624

Victor

On Fri, Apr 16, 2021 at 3:22 PM Ivan Pozdeev via Python-Dev
 wrote:
>
> It'll probably be easier to change the warnings filter semantic 
> (https://docs.python.org/3/library/warnings.html#the-warnings-filter)
> to which those values are directly passed.
>
> On 16.04.2021 16:07, Victor Stinner wrote:
> > Hi,
> >
> > I propose to change the -W command line option and the PYTHONWARNINGS
> > environment variable to use the message as a regular expression in
> > Python 3.10. Or does anyone have a reason to keep the current behavior
> > as it is?
> >
> > I created https://bugs.python.org/issue43862 for this change.
> >
> > --
> >
> > Python provides two ways to specify warnings filters:
> >
> > * -W command line option: can be used multiple times
> > * PYTHONWARNINGS environment variable: can contain multiple options
> > separated by commas
> >
> > While the Python API warnings.filterwarnings(action, message="", ...)
> > uses the message as a regular expression, -W and PYTHONWARNINGS
> > require to match *exactly* the *whole* message.
> >
> > For example, if you only want to ignore the new distutils deprecation
> > warning, you must write exactly:
> >
> > $ ./python -X dev -W 'ignore:The distutils package is deprecated and
> > slated for removal in Python 3.12. Use setuptools or check PEP 632 for
> > potential alternatives:DeprecationWarning' -c 'import distutils'
> >
> > I use -X dev to show DeprecationWarning, or you can also use -Wdefault
> > if you prefer.
> >
> > If the deprecation warning changes in Python or if you have a single
> > typo, the warning is not ignored. Example with a typo ("3.13" rather
> > than "3.12"):
> >
> > $ ./python -X dev -W 'ignore:The distutils package is deprecated and
> > slated for removal in Python 3.13. Use setuptools or check PEP 632 for
> > potential alternatives:DeprecationWarning' -c 'import distutils'
> > :1: DeprecationWarning: The distutils package is deprecated
> > and slated for removal in Python 3.12. Use setuptools or check PEP 632
> > for potential alternatives
> >
> > The PYTHONWARNINGS has another limitation: you cannot specify a
> > message if it contains a comma (","). Hopefully, Python doesn't raise
> > warnings containing comma, right? Well... Just one example:
> >
> > Lib/distutils/sysconfig.py:554:warnings.warn('SO is
> > deprecated, use EXT_SUFFIX', DeprecationWarning, 2)
> >
> > You cannot only ignore the message:
> >
> > $ PYTHONWARNINGS='ignore:SO is deprecated, use
> > EXT_SUFFIX:DeprecationWarning' ./python -c 'import sys;
> > print(sys.warnoptions); print(len(sys.warnoptions))'
> > Invalid -W option ignored: invalid action: 'use EXT_SUFFIX'
> > ['ignore:SO is deprecated', ' use EXT_SUFFIX:DeprecationWarning']
> > 2
> >
> > You can only try to use "module" and "lineno" parameters of a warning
> > filter, which are more fragile and hard to use to use.
> >
> > Victor
>
> --
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ADUDHMNJIYERRA5MHF4GGB2OXV2XJC37/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IIWWB5IZ32H3O4QHYUGRA2D7DYMVSUIY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Typing syntax and ecosystem

[Carl Meyer]

Hi Barry & Luciano,

Barry Warsaw wrote:
> Actually, I think it’s time for a comprehensive guide to type annotations.  
> Just anecdotally, I was trying to annotate a library of mine and was having 
> an impossible time of it, until a chat with Guido lead me to 
> @typing.overload.  That solved my problem intuitively and easily, but I just 
> didn’t know about it.  Right now, there’s information spread out all over the 
> place, the stdlib documentation, tool documentation, StackOverflow :D etc.  
> It’s a complicated topic that I think a comprehensive guide, a tutorial, etc. 
> could really help with.
> One of my favorite frameworks for thinking about documentation on a topic 
> such as this is:
> https://documentation.divio.com/
> I really think that would help people get into Python type annotations, both 
> casually and deeply.
> > I volunteer to help with a "Typing HOWTO". For the next few months, I
> > can offer to review if someone else writes it. In the second semester,
> > I could write it myself, if the experts on typing and the type
> > checkers would be willing to review it.
> > I don’t know whether I’ll have time to *start* something any time soon, but 
> > I would also volunteer to be a reviewer and/or provide some content.

I'm also interested in helping with this.

I think the first question to answer is, are the current mypy docs 
(https://mypy.readthedocs.io/en/stable/) insufficient for this purpose, and 
why? They do include both tutorial-style "getting started" paths as well as 
reference documentation. If they are not serving this purpose, why not? Is it 
due to their content or structure, or just because they are framed as "the mypy 
docs" and not "a typed Python HOWTO", so they don't find the right audience?

If we do need to write something new, one resource I can offer is an "intro to 
typed Python" talk I gave at PyCon 2018: 
https://www.youtube.com/watch?v=pMgmKJyWKn8

I've received feedback from many people without previous experience with typing 
that this talk was useful to them in understanding both the why and the how. If 
it seems useful I'd potentially be willing to adapt and update the content and 
code examples from this talk into written form as a starting point.

Carl
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MEZRNWGXAQ5PCVYTOFTOLS7XHORLVMCB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Guido van Rossum]

If you look deeper, the real complaints are all about the backwards
incompatibility when it comes to locally-scoped types in annotations. I.e.

def test():
  class C: ...
  def func(arg: C): ...
  return func

typing.get_type_hints(test())  # raises NameError: name 'C' is not defined

And that is a considerable concern (we've always let backwards
compatibility count more strongly than convenience of new features). While
it was known this would change, there was no real deprecation of the old
way. Alas.

On Fri, Apr 16, 2021 at 1:51 AM Sebastian Rittau  wrote:

> On Sun, Apr 11, 2021 at 1:31 PM Barry Warsaw  wrote:
>
> [snip]
>
>> This is something the SC has been musing about, but as it’s not a fully
>> formed idea, I’m a little hesitant to bring it up.  That said, it’s
>> somewhat relevant: We wonder if it may be time to in a sense separate the
>> typing syntax from Python’s regular syntax.  TypeGuards are a case where if
>> typing had more flexibility to adopt syntax that wasn’t strictly legal
>> “normal” Python, maybe something more intuitive could have been proposed.
>> I wonder if the typing-sig has discussed this possibility (in the future,
>> of course)?
>>
> I am strongly in favor of diverging type annotation syntax from Python
> syntax. Currently, type annotations are a very useful tool, but often
> clunky to use. Enhancements have been made, but design space is limited
> when working within existing Python syntax. Type annotations have a
> different set of rules, needs, and constraints than general-purpose Python
> code. This is similar to other domain specific languages like regular
> expressions. Ideally, Python itself would not check the syntax of
> annotations, except as needed for determining the end of an annotation. PEP
> 563 is a step in that direction.
>
> As far as I understand the arguments against PEP 563 and in favor of PEP
> 649 mostly boil down to "annotations are used outside of typing, these uses
> would need to use eval() in the future and eval() is slow". (At least from
> a user's perspective, there are more arguments from a Python maintainer's
> perspective that I can't comment on.) Are there benchmarks to verify that
> using eval() has a non-negligible effect for this use case? Overall, I
> don't find this to be a compelling argument when compared to the problem
> that PEP 649 would close all design space for type annotation syntax
> enhancements.
>
>  - Sebastian
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RWLOLMWLPZ3O5VO7OQZSHTQGR4ICXDJV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QGPNAR4Q3QPWDEEY4H7IMNVBXDBBRBFT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Christopher Barker]

On Fri, Apr 16, 2021 at 1:51 AM Sebastian Rittau  wrote:

> I am strongly in favor of diverging type annotation syntax from Python
> syntax. Currently, type annotations are a very useful tool, but often
> clunky to use. Enhancements have been made, but design space is limited
> when working within existing Python syntax. Type annotations have a
> different set of rules, needs, and constraints than general-purpose Python
> code. This is similar to other domain specific languages like regular
> expressions. Ideally, Python itself would not check the syntax of
> annotations, except as needed for determining the end of an annotation.
>
Another example is a discussion a little while back on python-ideas about
extending what's allowed inside square brackets. It started with a use-case
for type specification. It turned out that there were other use cases, more
tightly tied to the original meaning of __getitem__.

Nevertheless, it struck me at the time that it would be nice if the Typing
use case could be addressed without the complication of making something
that made sense in two very different domains.

- Chris

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SG5MCSP6KQDEO2BDZ6LE76TSKGH7TD6E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 563 in light of PEP 649

[Łukasz Langa]

Hi all,
I got pinged to voice my opinion on PEP 649 as the instigator of PEP 563. I'm 
sorry, this is long, and a separate thread, because it deals with three things:
- Goals set for PEP 563 and how it did in practice;
- PEP 649 and how it addresses those same goals;
- can we cleanly adopt PEP 649?


First off, it looks like this isn't worded clearly enough in the PEP itself so 
let me summarize what the goals of PEP 563 were:

Goal 1. To get rid of the forward reference problem, e.g. when a type is 
declared lower in the file than its use. A cute special case of this is when a 
class has a method that accepts or returns objects of its own type.

Goal 2. To somewhat decouple the syntax of type annotations from the runtime 
requirements, allowing for better expressibility.

Goal 3. To make annotations affect runtime characteristics of typed modules 
less, namely import time and memory usage.


Now, did PEP 563 succeed in its goals? Well, partially at best. Let's see.

In terms of Goal 1, it turned out that `typing.get_type_hints()` has limits 
that make its use in general costly at runtime, and more importantly 
insufficient to resolve all types. The most common example deals with 
non-global context in which types are generated (e.g. inner classes, classes 
within functions, etc.). But one of the crown examples of forward references: 
classes with methods accepting or returning objects of their own type, also 
isn't properly handled by `typing.get_type_hints()` if a class generator is 
used. There's some trickery we can do to connect the dots but in general it's 
not great.

As for Goal 2, it became painfully obvious that a number of types used for 
static typing purposes live outside of the type annotation context. So while 
PEP 563 tried to enable a backdoor for more usable static typing syntax, it 
ultimately couldn't. This is where PEP 585 and later PEP 604 came in, filling 
the gap by doing the sad but necessary work of enabling this extended typing 
syntax in proper runtime Python context. This is what should have been done all 
along and it makes PEP 563 in this context irrelevant as of Python 3.9 (for PEP 
585) and 3.10 (for PEP 604). However, to the extent of types used within 
annotations, the PEP 563 future-import allows using the new cute typing syntax 
already for Python 3.7+ compatible code. So library authors can already adopt 
the lowercase type syntax of PEP 585 and the handy pipe syntax for unions of 
PEP 604. And even for non-type annotation uses that can be successfully barred 
by a `if TYPE_CHECKING` block, like type aliases, type variables, and such. Of 
course that has no chance of working with `typing.get_type_hints()`.

Now, Goal 3 is a different matter. As Inada Naoki demonstrated somewhere in the 
preceding discussion here, PEP 563 made fully type-annotationed codebase import 
pretty much as fast as non-annotated code, and through the joys of string 
interning, use relatively little extra memory. At the time PEP 563, a popular 
concern around static typing in Python was that it slows down runtime while 
it's only useful for static analysis. While we (the typing crowd) were always 
sure the "only useful as a better linter" is dismissive, the performance 
argument had to go.


So where does this leave us today?

Runtime use of types was somewhat overly optimistically treated as solvable 
with `typing.get_type_hints()`. Now Pydantic and other similar tools show that 
this isn't sadly the case. Without the future-import, they could ignore the 
problem until Python 3.10 but no longer. I was somewhat surprised this was the 
case because forward references as strings could always be used. So I guess the 
answer there was to just not use them if you want your runtime tool to work. 
Fair enough.

PEP 649 addresses this runtime usage of type annotations head on in a way that 
eluded me when I first set out to solve this problem. Back then, Larry and Mark 
Shannon did voice their opinion that through some clever frame object storage, 
"implicit lambdas", we can address the issue of forward references. This seemed 
unattractive to me at the time because it didn't deal with our Goal 2 and our 
understanding was that it actually makes Goal 3 worse by holding on to all 
frames in memory where type annotations appear, and by creating massive 
duplication of equivalent annotations in memory due to lack of a mechanism 
similar to string interning. Now those issues are somewhat solved in the final 
PEP 649 and this makes for an interesting compromise for us to make. I say 
"compromise" because as Inada Naoki measured, there's still a non-zero 
performance cost of PEP 649 versus PEP 563:

- code size: +63%
- memory: +62%
- import time: +60%


Will this hurt some current users of typing? Yes, I can name you multiple past 
employers of mine where this will be the case. Is it worth it for Pydantic? I 
tend to think that yes, it is, since it is a significant community, and the 
operations on type annota

[Python-Dev] Re: In support of PEP 649

[Danny]

> Are you sure about PEP 563 break it and discord.py can not fix it?

PEP 563 has been supported by discord.py since the PEP was introduced -- the 
change was not particularly hard to support on my end. Newer versions of the 
library are even more supportive of things like `ForwardRef` and the like.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z4FDBM2NKDSY4LKAJ7JY2ZOXJQVL22YD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Walter Dörwald]

On 16 Apr 2021, at 16:59, Guido van Rossum wrote:


If you look deeper, the real complaints are all about the backwards
incompatibility when it comes to locally-scoped types in annotations. 
I.e.


def test():
  class C: ...
  def func(arg: C): ...
  return func

typing.get_type_hints(test())  # raises NameError: name 'C' is not 
defined


Can't this be solved by wrapping the annotation in a lambda, i.e.

```

def test():

...   class C: ...
...   def func(arg: lambda: C): ...
...   return func
...

test().__annotations__['arg']()

.C'>
```

So `typing.get_type_hints()` would simply call an annotation if the 
annotation was callable and replace it with the result of the call.



And that is a considerable concern (we've always let backwards
compatibility count more strongly than convenience of new features). 
While
it was known this would change, there was no real deprecation of the 
old

way. Alas.

On Fri, Apr 16, 2021 at 1:51 AM Sebastian Rittau  
wrote:


On Sun, Apr 11, 2021 at 1:31 PM Barry Warsaw  
wrote:


[snip]

This is something the SC has been musing about, but as it’s not a 
fully
formed idea, I’m a little hesitant to bring it up.  That said, 
it’s
somewhat relevant: We wonder if it may be time to in a sense 
separate the
typing syntax from Python’s regular syntax.  TypeGuards are a case 
where if
typing had more flexibility to adopt syntax that wasn’t strictly 
legal
“normal” Python, maybe something more intuitive could have been 
proposed.
I wonder if the typing-sig has discussed this possibility (in the 
future,

of course)?

I am strongly in favor of diverging type annotation syntax from 
Python

syntax. Currently, type annotations are a very useful tool, but often
clunky to use. Enhancements have been made, but design space is 
limited

when working within existing Python syntax. Type annotations have a
different set of rules, needs, and constraints than general-purpose 
Python

code. This is similar to other domain specific languages like regular
expressions. Ideally, Python itself would not check the syntax of
annotations, except as needed for determining the end of an 
annotation. PEP

563 is a step in that direction.

As far as I understand the arguments against PEP 563 and in favor of 
PEP
649 mostly boil down to "annotations are used outside of typing, 
these uses
would need to use eval() in the future and eval() is slow". (At least 
from
a user's perspective, there are more arguments from a Python 
maintainer's
perspective that I can't comment on.) Are there benchmarks to verify 
that
using eval() has a non-negligible effect for this use case? Overall, 
I
don't find this to be a compelling argument when compared to the 
problem

that PEP 649 would close all design space for type annotation syntax
enhancements.

 - Sebastian


--
--Guido van Rossum (python.org/~guido)


Servus,
   Walter
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HXJWNS4IIAHKTOCYR7AS5DOI5JAGLRPP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[redradist]

The benefits:

1. You will link with high quality libstdc++ with lots of reusable containers 
without writing your own "buggy" one.
2. C++ is much much more maintainable than pure C. It drastically increase 
number of contributors that what like writing high quality and maintainable 
code without reinventing the wheel.

My personal stop of contributing in CPython is that it is written in pure C !!
I wrote code in both: pure C and C++, but I like writing code in C++, because 
it simplifies things without losing perfomance
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YNVO3NM7J77AHVOPGCOTWZELTTHHA7YB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Chris Angelico]

On Sat, Apr 17, 2021 at 3:20 AM  wrote:
>
> The benefits:
>
> 1. You will link with high quality libstdc++ with lots of reusable containers 
> without writing your own "buggy" one.
> 2. C++ is much much more maintainable than pure C. It drastically increase 
> number of contributors that what like writing high quality and maintainable 
> code without reinventing the wheel.
>

Citations please?

> My personal stop of contributing in CPython is that it is written in pure C !!

A lot of CPython is actually written in Python. There's plenty that
you can contribute to without writing a single line of C code, if you
so desire.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SWGOS4SA5IDUFRYSJRQQQR7RQPAP7RSK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[redradist]

Chris Angelico wrote:
> On Sat, Apr 17, 2021 at 3:20 AM redrad...@gmail.com wrote:
> > The benefits:
> > 
> > You will link with high quality libstdc++ with lots of reusable containers 
> > without writing your own "buggy" one.
> > C++ is much much more maintainable than pure C. It drastically increase 
> > number of contributors that what like writing high quality and maintainable 
> > code without reinventing the wheel.
> > 
> > Citations please?

What exactly do you what ?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZQXWFCJYSN5YZTJCELFWUKBAPMROGIJP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Samuel Colvin]

Thanks so much for this, I think it makes a lot of sense. You've saved me
by explaining why PEP 563 semantics are problematic for pydantic far more
articulately than I could have done.

I can entirely see why PEP 563 made sense at the time, myself (and others
in the "runtime use of annotations" community) should have engaged with the
PEP before it was accepted, we might have been able to smooth out these
wrinkles then.

*One other route occurs to me:*

Switch from `from __future__ import annotations` to a per module feature
flag (say, something like `from __switches__ import postponed_annotations`).

I don't know if there's a precedent for this?, but I think it has a lot of
advantages:

* those who want type annotations as strings get their way (just a switch
of statement at the beginning of files)
* those who want runtime access to type annotations (pydantic etc.) get
their way
* if you have one module in a big code base that uses pydantic or similar,
you only need to pay the price of runtime type annotations in that file
* `from __future__ import annotations` can continue to prompt PEP 563
semantics (for now or indefinitely), independently it can also continue to
work for PEP 585 and PEP 604 as I believe it does now
* there's room for PEP 649 or similar in the future to improve forward refs
and performance when the switch is not set, without the rush to get it into
3.10
* we could even allow the default to be changed with an env variable or
command line flag like -O / PYTHONOPTIMIZE - but maybe this too complicated

I understand that adding another switch to python is not ideal, but given
that we are where we are, it seems like a pragmatic solution.

For me I'd be happy if there was anyway whatsoever to keep the current (or
PEP 649) behaviour in 3.10.

Samuel

--

Samuel Colvin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XCKF3JHD4APAJPL4QEYOGZFLKBCMBYSE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Chris Angelico]

On Sat, Apr 17, 2021 at 3:32 AM  wrote:
>
> Chris Angelico wrote:
> > On Sat, Apr 17, 2021 at 3:20 AM redrad...@gmail.com wrote:
> > > The benefits:
> > >
> > > You will link with high quality libstdc++ with lots of reusable 
> > > containers without writing your own "buggy" one.
> > > C++ is much much more maintainable than pure C. It drastically increase 
> > > number of contributors that what like writing high quality and 
> > > maintainable code without reinventing the wheel.
> > >
> > > Citations please?
>
> What exactly do you what ?

Evidence for your statements. Otherwise, empty statements can be
rejected with empty rebuttals.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GEVDFX6CXJQXXZ2EZC3MZ6NV6FSFWGTH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Victor Stinner]

Hi,

I used C++ in the past to write a small game. My experience was that
the compilation was quite slow and many compiler errors were hard to
understand.

I love the fact the CPython is written in C because its build time is
under one minute for a fresh build, and way faster for an incremental
build (modifying a single C file).

I'm part of the "trial-and-error" church. I modify random parts of the
C code, build, test and repeat until it works as I want ;-)

A *fresh* build (after make clean) of CPython on my laptop (8 threads,
4 CPU cores) takes 13 seconds using make -j10 and gcc -O0. A fresh
build in release mode (make -j10 and gcc -O3) takes 44 seconds on the
same laptop.

CPython is made of around 500K lines of C code.

Victor

On Fri, Apr 16, 2021 at 7:19 PM  wrote:
>
> The benefits:
>
> 1. You will link with high quality libstdc++ with lots of reusable containers 
> without writing your own "buggy" one.
> 2. C++ is much much more maintainable than pure C. It drastically increase 
> number of contributors that what like writing high quality and maintainable 
> code without reinventing the wheel.
>
> My personal stop of contributing in CPython is that it is written in pure C !!
> I wrote code in both: pure C and C++, but I like writing code in C++, because 
> it simplifies things without losing perfomance
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/YNVO3NM7J77AHVOPGCOTWZELTTHHA7YB/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PAIR6UTGNHYKGCTU3GRZSUVECJ2L2W3X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[redradist]

Guys, the issue is that I most of the time see that somebody used C++ for one 
or two times, did not understand it and left with bad taste ...

Please, answer me question, if you will go in gym two times, will you get stop 
training and say that it does not fit in your life ?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWKVHG7CLRCSCBNR5ZKUORBMH6Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Jelle Zijlstra]

El vie, 16 abr 2021 a las 10:01, Walter Dörwald ()
escribió:

> On 16 Apr 2021, at 16:59, Guido van Rossum wrote:
>
> If you look deeper, the real complaints are all about the backwards
> incompatibility when it comes to locally-scoped types in annotations. I.e.
>
> def test():
> class C: ...
> def func(arg: C): ...
> return func
>
> typing.get_type_hints(test()) # raises NameError: name 'C' is not defined
>
> Can't this be solved by wrapping the annotation in a lambda, i.e.
>
> >>> def test():
> ...   class C: ...
> ...   def func(arg: lambda: C): ...
> ...   return func
> ...
> >>> test().__annotations__['arg']()
> .C'>
>
> So typing.get_type_hints() would simply call an annotation if the
> annotation was callable and replace it with the result of the call.
>
That sort of thing can work, but just like string annotations it's not good
for usability. Users using annotations will have to remember that in some
contexts they need to wrap their annotation in a lambda, and unless they
have a good understanding of how type annotations work under the hood, it
will feel like a set of arbitrary rules. That's what I like about PEP 649:
code like this would (hopefully!) just work without needing users to
remember to use any special syntax.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V6NQ7ZOPV5BCZ3NGZXT36WVDGJKZQS47/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[redradist]

Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU
or read some articles ... otherwise I will need to spend too many time 
providing evidences to you and after all you will probably will reject anyway 
(because lots of people is biased and they even do not understand that, it is 
not about you, it is in general)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HFCMD2GHXP6MLGSYCEPSMJKA6J6723RH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Samuel Colvin]

Thank you everyone for your responses.

I entirely accept that I should have brought this up earlier, perhaps much 
earlier.

In my defence, when PEP 563 first came on my radar I assumed that 
get_type_hint() would be improved before it became the default behaviour, AFAIK 
it hasn't really changed. In particular the subtle changes to scope described 
in detail in PEP 649 [1] are still there and are still a big headache for 
pydantic.

As Paul Ganssle says, my timing was not great, but it could also have been 
worse.

I also never meant to create an "us versus them" style discussion, I was just 
worried about the narrow time window I had and the likelihood that my voice 
would not be heard, hence sounding as impassioned as I did.

For a fairly comprehensive description of why PEP 563 represents challenges to 
pydantic, I defer to Łukasz Langa (the instigator of PEP 563) who has explained 
it very clearly in another thread "[Python-Dev] PEP 563 in light of PEP 649" 
[2].

Samuel

[1] - https://www.python.org/dev/peps/pep-0649/#id9
[2] - 
https://mail.python.org/archives/list/python-dev@python.org/message/ZBJ7MD6CSGM6LZAOTET7GXAVBZB7O77O/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJTAJJEXZXM5WKJ47POLK3YWWNPKK5NB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Christian Heimes]

On 16/04/2021 19.39, Victor Stinner wrote:
> A *fresh* build (after make clean) of CPython on my laptop (8 threads,
> 4 CPU cores) takes 13 seconds using make -j10 and gcc -O0. A fresh
> build in release mode (make -j10 and gcc -O3) takes 44 seconds on the
> same laptop.

$ make clean
$ time make -j10
...
real0m2,072s
user0m4,715s
sys 0m2,333s

./configure -C and ccache are fantastic.

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LLDVGXNH32KDVVTT7NXIG7UOXHMW4WXB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[edwin]

Anyone who has done a language change on a project knows that it is a huge 
disruption.  You need solid justification to make such a change.  All I have 
seen in this thread is personal opinion.  Since this is a personal opinion 
exchange, I am of the humble opinion that the personal opinions of core devs 
matter the most, since a language change would affect them more than anyone 
else.

April 16, 2021 1:47 PM, redrad...@gmail.com wrote:

> Guys, the issue is that I most of the time see that somebody used C++ for one 
> or two times, did not
> understand it and left with bad taste ...
> 
> Please, answer me question, if you will go in gym two times, will you get 
> stop training and say
> that it does not fit in your life ?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWKVHG7CLRCSCBNR5ZKUORBMH6Z
> Code of Conduct: http://python.org/psf/codeofconduct
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3K3FDADSIFH7WJEKFNNYWFDTCZGZFKBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Ethan Furman]

On 4/16/21 10:27 AM, redrad...@gmail.com wrote:

Chris Angelico wrote:

On Sat, Apr 17, 2021 at 3:20 AM redrad...@gmail.com wrote:

The benefits:

You will link with high quality libstdc++ with lots of reusable containers without 
writing your own "buggy" one.
C++ is much much more maintainable than pure C. It drastically increase number 
of contributors that what like writing high quality and maintainable code 
without reinventing the wheel.

Citations please?


What exactly do you what ?


A list of current Python containers that are already in C++, that work exactly as our C ones do (otherwise, we will have 
had to add code to make them work as we want, and surely that will introduce bugs).


Examples of how C++ is more maintainable.

A poll of users that love C++ enough to contribute to CPython because we also use C++, combined with a poll of users 
currently contributing C code that would stop because they don't know/like C++.


We should also have evidence that C++ is available on all supported platforms 
that we support CPython on.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DKC4UKYGLJJQX3N3EOBQYROMHV7FZKMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Ethan Furman]

On 4/16/21 10:43 AM, redrad...@gmail.com wrote:


Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU
or read some articles ... otherwise I will need to spend too many time 
providing evidences to you and after all you will probably will reject anyway 
(because lots of people is biased and they even do not understand that, it is 
not about you, it is in general)


You are the one proposing the change, so it's up to you to provide the evidence for it.  If you aren't willing to put in 
a few hours for that effort, why should we put the weeks and months to port the code over?


--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OBZU7R2ORXXF3FSJL55TX4SRR7G7KZP2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Christian Heimes]

On 16/04/2021 19.14, redrad...@gmail.com wrote:
> My personal stop of contributing in CPython is that it is written in pure C !!
> I wrote code in both: pure C and C++, but I like writing code in C++, because 
> it simplifies things without losing perfomance

There are plenty of Open Source projects that could use more capable C++
developers. :)

I'm not a fan of C++. It has its use cases, e.g. in UI. Python core
isn't the best fit. AFAIK most core devs are not fluent in C++. Despite
it's name, C++ is really a different language than C. It has a different
ABI and stdlib than C, too. In my personal opinion C++ won't give us any
net benefits. I'd much rather go for Rust than C++ to gain memory safety.

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2HTFWK3JXOTCENWOVW6TYAZPIBP3HR76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Denis Kotov]

edwin＠211mainstreet.net wrote:
> Anyone who has done a language change on a project knows that it is a huge 
> disruption.  You need solid justification to make such a change.  All I have 
> seen in this thread is personal opinion.  Since this is a personal opinion 
> exchange, I am of the humble opinion that the personal opinions of core devs 
> matter the most, since a language change would affect them more than anyone 
> else.
> April 16, 2021 1:47 PM, redrad...@gmail.com wrote:
> > Guys, the issue is that I most of the time see that somebody used C++ for 
> > one or two times, did not
> > understand it and left with bad taste ...
> > Please, answer me question, if you will go in gym two times, will you get 
> > stop training and say
> > that it does not fit in your life ?
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org
> > Message archived at
> > https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWK...
> > Code of Conduct: http://python.org/psf/codeofconduct
> >

Okay lets try to discuss one by one:
1) Readability - less code, most code is hidden by abstraction without losing 
performance
In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed 
with C++ std::shared_ptr<> (RustPython use analog Arc<>)

2) Maintainability - no code duplication in favor of using reusable classes
In CPython I saw custom lists and stacks and so on ... with C++ it could be 
switched to std::list<>, std::stack<> and so on
In CPython lots of custom implemented algorithms that could be changed by using 
C++  that support lots of types !!

3) RAII - Resource Acquisition Is Initialization, predictable allocation and 
free resources
Also reusable peaces of code will help of maintaining life-time of objects with 
RAII, not only at the end of function, but in general, because life-time 
resource will be bound to object life-time
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HCZBCLFRQGPKRGJ5JAVIMOWBBLMSNOVG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Denis Kotov]

Ethan Furman wrote:
> On 4/16/21 10:43 AM, redrad...@gmail.com wrote:
> > Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU
> > or read some articles ... otherwise I will need to spend too many time 
> > providing evidences to you and after all you will probably will reject 
> > anyway (because lots of people is biased and they even do not understand 
> > that, it is not about you, it is in general)
> > You are the one proposing the change, so it's up to you to provide the 
> > evidence for it.  If you aren't willing to put in 
> a few hours for that effort, why should we put the weeks and months to port 
> the code over?
> --
> ~Ethan~

I do not ask porting code, I ask using new code with C++ and if code was tested 
enough to reimplement it in C++ with RAII, 

Also I suggest using C++ excepting that most of the people here now it ... It 
was not intended to teach C++ here, especially in Mail List )))

And reason why you at least should try learn other languages, it is because it 
will make you better developer
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7MLCM7DKBAPMBBIRWZHHTOGL76GPIYWM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2021-04-09 - 2021-04-16)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7522 ( -2)
  closed 48086 (+78)
  total  55608 (+76)

Open issues with patches: 2993 


Issues opened (52)
==

#37251: Mocking a MagicMock with a function spec results in an AsyncMo
https://bugs.python.org/issue37251  reopened by gregory.p.smith

#40066: Enum: modify __repr__, __str__; update docs
https://bugs.python.org/issue40066  reopened by ethan.furman

#40432: Pegen regenerate project for Windows not working
https://bugs.python.org/issue40432  reopened by pablogsal

#43764: Turning off generation of __match_args__ for dataclasses
https://bugs.python.org/issue43764  reopened by eric.smith

#43802: Seg fault on macOS using multiprocessing.JoinableQueue
https://bugs.python.org/issue43802  opened by jacobtylerwalls

#43803: ctypes string_at/wstring_at - bad argument name used in docs a
https://bugs.python.org/issue43803  opened by talhayon1

#43804: Add more info about building C/C++ Extensions on Windows using
https://bugs.python.org/issue43804  opened by shreyanavigyan

#43805: multiprocessing.Queue hangs when process on other side dies
https://bugs.python.org/issue43805  opened by kormang

#43806: asyncio.StreamReader hangs when reading from pipe and other pr
https://bugs.python.org/issue43806  opened by kormang

#43807: JSONDecodeError: Extra Data Raised on Long Valid JSON
https://bugs.python.org/issue43807  opened by UnknownRetiredGuy

#43811: Run GHA CI with multiple OpenSSL versions
https://bugs.python.org/issue43811  opened by christian.heimes

#43813: Denial of service on http.server module with large request met
https://bugs.python.org/issue43813  opened by demonia

#43814: Fix the error message for disallowed __weakref__ slots
https://bugs.python.org/issue43814  opened by maggyero

#43815: documentation for types.new_class() mention misleading default
https://bugs.python.org/issue43815  opened by eric.smith

#43817: Add typing.get_annotations()
https://bugs.python.org/issue43817  opened by larry

#43818: Email does not apply policy to multipart messages with defects
https://bugs.python.org/issue43818  opened by jishac

#43819: ExtensionFileLoader Does Not Implement invalidate_caches
https://bugs.python.org/issue43819  opened by Ian.H

#43821: Undocumented behavior of sleep functions and asyncio delayed e
https://bugs.python.org/issue43821  opened by Josef Havr??nek

#43826: Resource warnings in test_subprocess
https://bugs.python.org/issue43826  opened by xtreak

#43827: abc conflicts with __init_subclass__
https://bugs.python.org/issue43827  opened by vladhoi

#43829: MappingProxyType cannot hash a hashable underlying mapping
https://bugs.python.org/issue43829  opened by andymaier

#43832: asyncio + multiprocessing = core dump in sem_trywait
https://bugs.python.org/issue43832  opened by Andrei Pozolotin

#43833: Unexpected Parsing of Numeric Literals Concatenated with Boole
https://bugs.python.org/issue43833  opened by sco1

#43834: Use context manager in StringIO example
https://bugs.python.org/issue43834  opened by John Hagen

#43835: Dataclasses don't call base class __init__
https://bugs.python.org/issue43835  opened by Paul Pinterits

#43837: Operator precedence documentation could be more clear
https://bugs.python.org/issue43837  opened by aidan.feldman

#43838: There is a way to access an underlying mapping in MappingProxy
https://bugs.python.org/issue43838  opened by serhiy.storchaka

#43843: libregrtest: mark a test as failed if a thread logs an unexpec
https://bugs.python.org/issue43843  opened by vstinner

#43845: test_concurrent_futures leaks many dangling threads on FreeBSD
https://bugs.python.org/issue43845  opened by vstinner

#43846: Control stack usage in large expressions
https://bugs.python.org/issue43846  opened by Mark.Shannon

#43847: [Windows] ntpath.realpath() of bytes root directory may raise 
https://bugs.python.org/issue43847  opened by 9001

#43848: gzip.py: explain optional argument mtime
https://bugs.python.org/issue43848  opened by jwuttke

#43851: Optimise SQLite builds on macOS and Windows
https://bugs.python.org/issue43851  opened by erlendaasland

#43852: [sqlite3] Improve tuple creation
https://bugs.python.org/issue43852  opened by erlendaasland

#43853: [sqlite3] Fix sqlite3_value_text() usage
https://bugs.python.org/issue43853  opened by erlendaasland

#43854: curses: returns incorrect chars after resuming a suspended pro
https://bugs.python.org/issue43854  opened by darrikonn

#43855: test_ssl: test_msg_callback_deadlock_bpo43577() failed on macO
https://bugs.python.org/issue43855  opened by vstinner

#43856: Docs for importlib.metadata should mention Python version
https://bugs.python.org/issue43856  opened by pitrou

#43857: Fix the AttributeError message for deletion of a missing attri
https://bugs.python.org

[Python-Dev] Re: PEP 563 in light of PEP 649

[Jukka Lehtosalo]

On Fri, Apr 16, 2021 at 5:28 PM Łukasz Langa  wrote:

> [snip] I say "compromise" because as Inada Naoki measured, there's still a
> non-zero performance cost of PEP 649 versus PEP 563:
>
> - code size: +63%
> - memory: +62%
> - import time: +60%
>
>
> Will this hurt some current users of typing? Yes, I can name you multiple
> past employers of mine where this will be the case. Is it worth it for
> Pydantic? I tend to think that yes, it is, since it is a significant
> community, and the operations on type annotations it performs are in the
> sensible set for which `typing.get_type_hints()` was proposed.
>

Just to give some more context: in my experience, both import time and
memory use tend to be real issues in large Python codebases (code size less
so), and I think that the relative efficiency of PEP 563 is an important
feature. If PEP 649 can't be made more efficient, this could be a major
regression for some users. Python server applications need to run multiple
processes because of the GIL, and since code objects generally aren't
shared between processes (GC and reference counting makes it tricky, I
understand), code size increases tend to be amplified on large servers.
Even having a lot of RAM doesn't necessarily help, since a lot of RAM
typically implies many CPU cores, and thus many processes are needed as
well.

I can see how both PEP 563 and PEP 649 bring significant benefits, but
typically for different user populations. I wonder if there's a way of
combining the benefits of both approaches. I don't like the idea of having
toggles for different performance tradeoffs indefinitely, but I can see how
this might be a necessary compromise if we don't want to make things worse
for any user groups.

Jukka
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PBJ6MBQIE3DVQUUAO764PIQ3TWGLBS3X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[edwin]

April 16, 2021 2:08 PM, "Denis Kotov"  wrote:

> edwin＠211mainstreet.net wrote:
> 
>> Anyone who has done a language change on a project knows that it is a huge 
>> disruption. You need
>> solid justification to make such a change. All I have seen in this thread is 
>> personal opinion.
>> Since this is a personal opinion exchange, I am of the humble opinion that 
>> the personal opinions of
>> core devs matter the most, since a language change would affect them more 
>> than anyone else.
>> April 16, 2021 1:47 PM, redrad...@gmail.com wrote:
>> Guys, the issue is that I most of the time see that somebody used C++ for 
>> one or two times, did not
>> understand it and left with bad taste ...
>> Please, answer me question, if you will go in gym two times, will you get 
>> stop training and say
>> that it does not fit in your life ?
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWK...
>> Code of Conduct: http://python.org/psf/codeofconduct
> 
> Okay lets try to discuss one by one:
> 1) Readability - less code, most code is hidden by abstraction without losing 
> performance
> In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed 
> with C++
> std::shared_ptr<> (RustPython use analog Arc<>)

So every single python extension library would have to be rewritten to handle 
all the new C++ apis?  Sounds like an idea everyone will be very excited about.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/52YMBBM5NWZPYZ7X57O2ETKWD22DLQ3K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Antoine Pitrou]

On Fri, 16 Apr 2021 18:08:58 -
"Denis Kotov"  wrote:
> 
> Okay lets try to discuss one by one:
> 1) Readability - less code, most code is hidden by abstraction without losing 
> performance
> In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed 
> with C++ std::shared_ptr<> (RustPython use analog Arc<>)

std::shared_ptr<> would be a bad replacement for CPython's reference
counting for two reasons:

1) the reference count is maintained in a separate memory block (unless
you were careful enough to use std::make_shared() for allocation)

2) the reference count is atomic, and this has been proven to slow down
CPython by 10-20%.

That does not mean that CPython couldn't benefit from C++-based
abstractions, but they would have to be implemented (or taken from
another project such as pybind11).

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YEXWX6NT5W533PFIJVXAKHQWJCY43BDZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Denis Kotov]

Antoine Pitrou wrote:
> On Fri, 16 Apr 2021 18:08:58 -
> "Denis Kotov" redrad...@gmail.com wrote:
> > Okay lets try to discuss one by one:
> > 
> > Readability - less code, most code is hidden by abstraction without losing 
> > performance
> > 
> > In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be 
> > fixed with C++ std::shared_ptr<> (RustPython use analog Arc<>)
> > std::shared_ptr<> would be a bad replacement for CPython's reference
> counting for two reasons:
> 1) the reference count is maintained in a separate memory block (unless
> you were careful enough to use std::make_shared() for allocation)
> 2) the reference count is atomic, and this has been proven to slow down
> CPython by 10-20%.

Rust have 2 ref count classes Rc and Arc.
First is without atomic, single threaded, second is with atomic and could be 
used in multiple thread.
It is possible to implement the same std::ref_ptr like Rc without atomic 
variables
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BOYTNQSCUXBWHUXZFE7RZSBWQQZZRZ5M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Denis Kotov]

Christian Heimes wrote:
> On 16/04/2021 19.14, redrad...@gmail.com wrote:
> > My personal stop of contributing in CPython is that it is written in pure C 
> > !!
> > I wrote code in both: pure C and C++, but I like writing code in C++, 
> > because it simplifies things without losing perfomance
> > There are plenty of Open Source projects that could use more capable C++
> developers. :)
> I'm not a fan of C++. It has its use cases, e.g. in UI. Python core
> isn't the best fit. AFAIK most core devs are not fluent in C++. Despite
> it's name, C++ is really a different language than C. It has a different
> ABI and stdlib than C, too. In my personal opinion C++ won't give us any
> net benefits. I'd much rather go for Rust than C++ to gain memory safety.
> Christian


Rust is not syntax compatible with C and that is why I suggest to use C++.
If you are not fan of C++, with Rust there are also templates and sometimes 
even harder errors for life-time than C++ error template instantiation )))

Rust does not fit in 30 years code base written in C like C++ does
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NO7EE4CQ5ALKEHTTNCDZ3B36QA7HQXGP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Larry Hastings]

Please don't confuse Inada Naoki's benchmark results with the effect PEP 
649 would have on a real-world codebase.  His artifical benchmark 
constructs a thousand empty functions that take three parameters with 
randomly-chosen annotations--the results provides some insights but are 
not directly applicable to reality.


PEP 649's effects on code size / memory / import time are contingent on 
the number of annotations and the number of objects annotated, not the 
overall code size of the module.  Expressing it that way, and suggesting 
that Python users would see the same results with real-world code, is 
highly misleading.


I too would be interested to know the effects PEP 649 had on a 
real-world codebase currently using PEP 563, but AFAIK nobody has 
reported such results.



//arry/

On 4/16/21 11:05 AM, Jukka Lehtosalo wrote:
On Fri, Apr 16, 2021 at 5:28 PM Łukasz Langa > wrote:


[snip] I say "compromise" because as Inada Naoki measured, there's
still a non-zero performance cost of PEP 649 versus PEP 563:

- code size: +63%
- memory: +62%
- import time: +60%


Will this hurt some current users of typing? Yes, I can name you
multiple past employers of mine where this will be the case. Is it
worth it for Pydantic? I tend to think that yes, it is, since it
is a significant community, and the operations on type annotations
it performs are in the sensible set for which
`typing.get_type_hints()` was proposed.


Just to give some more context: in my experience, both import time and 
memory use tend to be real issues in large Python codebases (code size 
less so), and I think that the relative efficiency of PEP 563 is an 
important feature. If PEP 649 can't be made more efficient, this could 
be a major regression for some users. Python server applications need 
to run multiple processes because of the GIL, and since code objects 
generally aren't shared between processes (GC and reference counting 
makes it tricky, I understand), code size increases tend to be 
amplified on large servers. Even having a lot of RAM doesn't 
necessarily help, since a lot of RAM typically implies many CPU cores, 
and thus many processes are needed as well.


I can see how both PEP 563 and PEP 649 bring significant benefits, but 
typically for different user populations. I wonder if there's a way of 
combining the benefits of both approaches. I don't like the idea of 
having toggles for different performance tradeoffs indefinitely, but I 
can see how this might be a necessary compromise if we don't want to 
make things worse for any user groups.


Jukka

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PBJ6MBQIE3DVQUUAO764PIQ3TWGLBS3X/
Code of Conduct: http://python.org/psf/codeofconduct/



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4OHBEX4ARPMB57MS7ICTZNS44KEORJRI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Question regarding the value of PyThreadState.thread_id

[Gabriele]

Hi all.

My apologies if this is a topic that's been discussed already, but I
wasn't able to locate anything in the archive on the subject. I was
wondering if there's a fundamental reason for using
PyThread_get_thread_ident instead of PyThread_get_thread_native_id for
the value of the thread_id field of the PyThreadState object.

The reason why I'm asking is that I would like to have easy access to
the native TID on Linux to identify the /proc/stat file associated
with the thread from an external process. At the moment I have to
resort to calling process_vm_readv to copy the struct pthread over to
local VM and then guess where the tid field might be. So, if there's
no fundamental reason for thread_id to be PyThread_get_thread_ident, I
would like to propose to change it to PyThread_get_thread_native_id
instead.

Thanks,
Gabriele

-- 
"Egli è scritto in lingua matematica, e i caratteri son triangoli,
cerchi, ed altre figure
geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola;
senza questi è un aggirarsi vanamente per un oscuro laberinto."

-- G. Galilei, Il saggiatore.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DOB6VUTAIJKK4SUGBYWL4QPWI2E5Q2T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Typing syntax and ecosystem

[Barry Warsaw]

Hi Carl,

> I think the first question to answer is, are the current mypy docs 
> (https://mypy.readthedocs.io/en/stable/) insufficient for this purpose, and 
> why?

There’s certainly lots of great documentation in the mypy docs; it’s often my 
first go-to.

> They do include both tutorial-style "getting started" paths as well as 
> reference documentation. If they are not serving this purpose, why not? Is it 
> due to their content or structure, or just because they are framed as "the 
> mypy docs" and not "a typed Python HOWTO", so they don't find the right 
> audience?

Framing is an aspect.  For example, the information on installing, configuring, 
and running mypy wouldn’t (IMHO) be appropriate for a comprehensive typing 
guide.  There’s also the fact that it lives under the mypy banner rather than 
under docs.python.org for example.  But I think mypy’s docs would make an 
excellent source for the guides I’m thinking about.  You’d want to write the 
guide being tool agnostic as much as possible (though, pointing out where 
semantics or behavior may differ), and you’d want to have a section on “Type 
Checkers” to talk about the common tools, without diving into them too much.

> If we do need to write something new, one resource I can offer is an "intro 
> to typed Python" talk I gave at PyCon 2018: 
> https://www.youtube.com/watch?v=pMgmKJyWKn8

I’ve become a huge fan of short videos as a way to help introduce and train 
developers on the tools that are available to them.  I’ve been working on “the 
documentation problem” at work for the last year+ and there’s nothing like a 
good video to engage with developers, especially as they start their journey 
into a particular topic.  A series of curated videos under the Python/PSF 
banner would be awesome.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CUR75FOV2OSD7U3LTA4PRQBQLAMR4AC3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Guido van Rossum]

On Fri, Apr 16, 2021 at 12:32 PM Larry Hastings  wrote:

>
>
> Please don't confuse Inada Naoki's benchmark results with the effect PEP
> 649 would have on a real-world codebase.  His artifical benchmark
> constructs a thousand empty functions that take three parameters with
> randomly-chosen annotations--the results provides some insights but are not
> directly applicable to reality.
>
> PEP 649's effects on code size / memory / import time are contingent on
> the number of annotations and the number of objects annotated, not the
> overall code size of the module.  Expressing it that way, and suggesting
> that Python users would see the same results with real-world code, is
> highly misleading.
>
> I too would be interested to know the effects PEP 649 had on a real-world
> codebase currently using PEP 563, but AFAIK nobody has reported such
> results.
>

I'm not going to report results, but we could use mypy itself as an example
real-world code base. Mypy is almost 100% annotated. It does not include
`from __future__ import annotations` lines but those could easily be added
mechanically for some experiment.

ISTM that the unmarshal times reported by Inada are largely proportional to
the code size numbers, so perhaps the following three-way experiment would
give an indication:

(1) Addthe sizes of all pyc files for mypy run with Python 3.9 (classic)
(2) Ditto run with Python 3.10a7 (PEP 563)
(3) Ditto run with Larry's branch (PEP 649, assuming it's on by default
there -- otherwise, modify the source by inserting the needed future import
at the top)

The repo is github.com/python/mypy, the subdirectory to look is mypy, WITH
THE EXCLUSION OF THE typeshed SUBDIRECTORY THEREOF.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OOFTM4DXUN3BEU4NTVHRDDUOJVLYCEQA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Christopher Barker]

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I did find this SO post:

https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses

which is related, but not quite the same -- THAT issue was fixed.

My issue does show up in this BPO:

https://bugs.python.org/issue39442

Somewhere in the middle of that thread, Eric wrote: "At this point, this
seems more like fodder for python-ideas."

Then there's a bit more to the thread, which peters out without resolution.

On to the issue:

dataclasses may be the only part of the stdlib that uses annotations.

dataclasses store the annotations in Field objects type attribute, in
the __dataclass_fields__ attribute.

We can see that with this code:

@dataclass
class Simple:
x : int
y : float
name : str

s = Simple(3, 4.0, "fred")

print("The field types:")
for f in s.__dataclass_fields__.values():
print(f"name: {f.name}, type: {f.type}, type of type: {type(f.type)}")

Which results in:

The field types:
name: x, type: , type of type: 
name: y, type: , type of type: 
name: name, type: , type of type: 

with:

from __future__ import annotations

The result is:

The field types:
name: x, type: int, type of type: 
name: y, type: float, type of type: 
name: name, type: str, type of type: 

This of course is completely as expected.

I have no idea how dataclasses uses the Field type attribute -- as far as I
can tell, for nothing at all. However, it is there, and it is called
"type", rather than say, "annotation".

And I have a whole pile of code that fully expects the Fields' type
attribute to be an actual type object that I can call to crate an instance
of that type (or call a method on it, which is what I am actually doing)

So my code will very much break with this change.

I fully understand that the __dataclass_fields__ attribute was probably
never intended to be part of the public API, so I get what I deserve.

However, the Field object is documented, as such:

"""
class dataclasses.Field

Field objects describe each defined field. These objects are created
internally, and are returned by the fields() module-level method (see
below). Users should never instantiate a Field object directly. Its
documented attributes are:

name: The name of the field.
type: The type of the field.
default, default_factory, init, repr, hash, compare, and metadata have the
identical meaning and values as they do in the field() declaration.

Other attributes may exist, but they are private and must not be inspected
or relied on.
"""

That last sentence implies that the type attribute CAN be inspected and
relied upon, which is what I am doing.

And I haven't tried to solve this problem in my use case, but I'm not sure
it can be done -- when I get around to inspecting the type of the Field
objects, I have no idea what namespace they are in -- so I can't
reconstruct them from the string. I really need the type object itself.

So I'll probably need to re-write much of the dataclasses decorator, to
call eval() early -- though even then I'm not sure I'll have access to the
proper namespace.

Anyway -- one more data point:  PEP 563 changes the (semi-public?) API of
dataclasses.

Though *maybe* that could be addressed with a dataclasses update -- again,
I've only started to think about it -- there was some discussion of that in
the BPO, though Eric didn't seem particularly interested.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4N4FE2Y37WZBILVUPHEHNVRE6NVU6VFO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Larry Hastings]

On 4/16/21 5:00 PM, Guido van Rossum wrote:
(3) Ditto run with Larry's branch (PEP 649, assuming it's on by 
default there -- otherwise, modify the source by inserting the needed 
future import at the top)



The co_annotations stuff in my branch is gated with "from __future__ 
import co_annotations".  Without that import my branch has stock semantics.


Also, in case somebody does do this testing: don't use my branch for 
"from __future__ import annotations" testing.  There are neat new 
optimizations for stringized annotations but my branch is too 
out-of-date to have them.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B3NWB54AADBCZROO4MIBDAJQP6Q6DXAW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Stephen J. Turnbull]

Denis,

There's a standard way of interesting the Python community in new
syntax (and C++, while not Python, is new syntax in spades to this
community ;-).

You take a hunk of the standard library (in this case it would have to
be an accelerator written in C since you want to compare C++ vs. C) or
interpreter code, and translate it to the new syntax.

Now, *INCREF and friends are frequently cited as annoyances or even
warts, so your suggestion of std::shared _ptr<> seemed plausible to
me.  But Antoine peeked at it and points out it's not that easy, for
performance reasons you can't use it "as is".  It's true that you
could reimplement it, but then it's not std::shared_ptr<> any more and
the only benefit left is that it looks familiar to C++ programmers
(and may mislead *them* into thinking about it as if it were exactly
std::shared_ptr<>).

And that's why you need to do more work than arguing that in principle
C++ is just a better language than C.  We've been hearing that for 4
decades now (at least we greybeards have), and we've discovered that
for many existing applications, C++ may be better but the cost of
converting large swaths of C code to equivalent C++ that passes all
tests is too big.  Python may very well be one of them.

So if you're not going to do the work to demonstrate big wins from
using C++ instead of C in actual Python implementation code, I think
you're wasting your posts.

Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WBA5PT6DIMRC6IPMAPAO5VALDVIGRPXX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Inada Naoki]

On Sat, Apr 17, 2021 at 1:38 PM Guido van Rossum  wrote:
>
> I'm not going to report results, but we could use mypy itself as an example 
> real-world code base. Mypy is almost 100% annotated. It does not include 
> `from __future__ import annotations` lines but those could easily be added 
> mechanically for some experiment.
>
> ISTM that the unmarshal times reported by Inada are largely proportional to 
> the code size numbers, so perhaps the following three-way experiment would 
> give an indication:
>
> (1) Addthe sizes of all pyc files for mypy run with Python 3.9 (classic)
> (2) Ditto run with Python 3.10a7 (PEP 563)
> (3) Ditto run with Larry's branch (PEP 649, assuming it's on by default there 
> -- otherwise, modify the source by inserting the needed future import at the 
> top)
>

Please don't use 3.10a7, but latest master branch.
CFG optimizer broke some PEP 563 optimization and I fixed it yesterday.
https://github.com/python/cpython/pull/25419

> The repo is github.com/python/mypy, the subdirectory to look is mypy, WITH 
> THE EXCLUSION OF THE typeshed SUBDIRECTORY THEREOF.
>

I want to measure import time and memory usage. Will `import
mypy.main` import all important modules?

This is my quick result of (1) and (2). I can not try (3) because of
memory error. (see later).

## memory usage

```
$ cat a.py
import tracemalloc
tracemalloc.start()
import mypy.main
print("memory:", tracemalloc.get_traced_memory()[0])

# (1)
$ python3 a.py
memory: 8963137
$ python3 -OO a.py
memory: 8272848

# (2)
$ ~/local/python-dev/bin/python3 a.py
memory: 8849216
$ ~/local/python-dev/bin/python3 -OO a.py
memory: 8104730

>>> (8963137-8849216)/8963137
0.012709947421310196
>>> (8272848-8104730)/8272848
0.020321659481716575
```

PEP 563 saved 1~2% memory.

## GC time

```
$ pyperf timeit -s 'import mypy.main, gc' -- 'gc.collect()'

3.9: . 2.68 ms +- 0.02 ms
3.10: . 2.23 ms +- 0.01 ms

Mean +- std dev: [3.9] 2.68 ms +- 0.02 ms -> [3.10] 2.23 ms +- 0.01
ms: 1.20x faster
```

PEP 563 is 1.2x faster!

## import time

```
$ python3 -m pyperf command python3 -c 'import mypy.main'

(1) command: Mean +- std dev: 99.6 ms +- 0.3 ms
(2) command: Mean +- std dev: 93.3 ms +- 1.2 ms

>>> (99.6-93.3)/99.6
0.06325301204819275
```

PEP 563 reduced 6% importtime.

## memory error on co_annotations

I modifled py_compile to add `from __future__ import co_annotations`
automatically.

```
$ ../co_annotations/python -m compileall mypy
Listing 'mypy'...
Compiling 'mypy/checker.py'...
free(): corrupted unsorted chunks
Aborted

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x77c73859 in __GI_abort () at abort.c:79
#2  0x77cde3ee in __libc_message
(action=action@entry=do_abort, fmt=fmt@entry=0x77e08285 "%s\n") at
../sysdeps/posix/libc_fatal.c:155
#3  0x77ce647c in malloc_printerr
(str=str@entry=0x77e0a718 "free(): corrupted unsorted chunks") at
malloc.c:5347
#4  0x77ce81c2 in _int_free (av=0x77e39b80 ,
p=0x55d1db30, have_lock=) at malloc.c:4356
#5  0x55603906 in PyMem_RawFree (ptr=) at
Objects/obmalloc.c:1922
#6  _PyObject_Free (ctx=, p=) at
Objects/obmalloc.c:1922
#7  _PyObject_Free (ctx=, p=) at
Objects/obmalloc.c:1913
#8  0x5567caa9 in compiler_unit_free (u=0x55ef0fd0) at
Python/compile.c:583
#9  0x5568aea5 in compiler_exit_scope (c=0x7fffc3d0) at
Python/compile.c:760
#10 compiler_function (c=0x7fffc3d0, s=,
is_async=0) at Python/compile.c:2529
#11 0x5568837d in compiler_visit_stmt (s=,
c=0x7fffc3d0) at Python/compile.c:3665
#12 compiler_body (c=c@entry=0x7fffc3d0, stmts=0x56222450) at
Python/compile.c:1977
#13 0x55688e51 in compiler_class (c=c@entry=0x7fffc3d0,
s=s@entry=0x56222a60) at Python/compile.c:2623
#14 0x55687ce3 in compiler_visit_stmt (s=,
c=0x7fffc3d0) at Python/compile.c:3667
#15 compiler_body (c=c@entry=0x7fffc3d0, stmts=0x563014c0) at
Python/compile.c:1977
#16 0x5568db00 in compiler_mod (filename=0x772e6770,
mod=0x563017b0, c=0x7fffc3d0) at Python/compile.c:2001
```

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2UON4FZ5UJ3RYE3ZO5Q445RVPMFAR2SW/
Code of Conduct: http://python.org/psf/codeofconduct/