[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Luciano Ramalho]

On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum  wrote:
> Can you try this?
>
> async def __sleep(self):
> await None

Perhaps you meant this?

async def __sleep0():
  await None

Either way, `await None` raises "TypeError: object NoneType can't be
used in 'await' expression".

Maybe I misunderstood your suggestion?



-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6VJEHRLKZPYNASAUU2KVXWLA6T6I2H5A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597 bikeshedding: envvar / option name.

[Inada Naoki]

On Mon, Feb 15, 2021 at 3:00 PM Paul Bryan  wrote:
>
> Let the bikeshedding begin. How about with the underscores in place? More 
> readable to my eyes.
>

I agree with you. Although it is not consistent with existing many
option names, it is much more readable.

Ivan, Victor, what do you think? about PYTHON_WARN_DEFAULT_ENCODING?

---
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TN4UE2WRKA3F6BBS7TYEYMBRSARYIUQM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Luciano Ramalho]

On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum  wrote:
> Can you try this?
>
> async def __sleep(self):
> await None

That didn't work*, but this does:

async def __sleep():
return None

Was that the idea?

(*) TypeError: object NoneType can't be used in 'await' expression

>
> On Thu, Feb 18, 2021 at 22:31 Luciano Ramalho  wrote:
>>
>> Follow up question: what's the plan to replace this use of
>> `@types.coroutine` in `asyncio/tasks.py`? [1]
>>
>> @types.coroutine
>> def __sleep0():
>>  ""
>>  yield
>>
>> [1] https://github.com/python/cpython/blob/master/Lib/asyncio/tasks.py#L585
>>
>> Best,
>>
>> Luciano
>>
>> On Fri, Feb 19, 2021 at 2:31 AM Luciano Ramalho  wrote:
>> >
>> > On Fri, Feb 19, 2021 at 1:59 AM Guido van Rossum  wrote:
>> > >> 1) What Python construct is to be used at the end of a chain of await
>> > >> calls, if not of a generator-based coroutine decorated with
>> > >> `@types.coroutine` and using a `yield` expression in its body?
>> >
>> > > At the end of the chain you can call the __await__() method which gives 
>> > > an iterator, and then you call next() or send() on that iterator. Each 
>> > > next()/send() call then represents an await step, and send() in general 
>> > > is used to provide an awaited result. Eventually this will raise 
>> > > StopIteration with a value indicating the ultimate result (the return 
>> > > value of the top-level async def).
>> >
>> > All right, that made sense to me. Thank you so much, Guido.
>> >
>> > Thanks for the clarification about `@types.coroutine` as well.
>> >
>> > Take care,
>> >
>> > Luciano
>> >
>> >
>> > --
>> > Luciano Ramalho
>> > |  Author of Fluent Python (O'Reilly, 2015)
>> > | http://shop.oreilly.com/product/0636920032519.do
>> > |  Technical Principal at ThoughtWorks
>> > |  Twitter: @ramalhoorg
>>
>>
>>
>> --
>> Luciano Ramalho
>> |  Author of Fluent Python (O'Reilly, 2015)
>> | http://shop.oreilly.com/product/0636920032519.do
>> |  Technical Principal at ThoughtWorks
>> |  Twitter: @ramalhoorg
>
> --
> --Guido (mobile)



-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4YLGMUXFPWKHQOQI4SEOM4UR2EN5MUNN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Luciano Ramalho]

On Fri, Feb 19, 2021 at 6:29 AM Luciano Ramalho  wrote:
> async def __sleep():
> return None

Sorry, I meant to write:

async def __sleep0():
return None

Since the idea is to replace the generator-based coroutine `__sleep0`
in tasks.py [1] with a native coroutine.

[1] 
https://github.com/python/cpython/blob/e92d67dfbb4790df37aa6a0961fb6dc7e8d2fbbf/Lib/asyncio/tasks.py#L585

-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TUQPYNWLCFM66G7B3C26WFIDGL7Y3O5G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Paul Sokolovsky]

Hello,

On Fri, 19 Feb 2021 06:29:35 -0300
Luciano Ramalho  wrote:

> On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum 
> wrote:
> > Can you try this?
> >
> > async def __sleep(self):
> > await None  
> 
> That didn't work*, but this does:
> 
> async def __sleep():
> return None
> 
> Was that the idea?

No. That doesn't return control to the event loop, like the original
"yield" does.

The purpose of all these shaman dances to obfuscate the relationship
between generators and async functions is unclear. (Well, it's kinda
clear - to allow "async generators", but ughh, don't let me start on
that.) The best solution now indeed appears to allow an "await None"
special construct, to have the same meaning as "yield", but only for
async functions. Any other solution is dirty (and inefficient!)
workaround.


> 
> (*) TypeError: object NoneType can't be used in 'await' expression
> 
> >
> > On Thu, Feb 18, 2021 at 22:31 Luciano Ramalho 
> > wrote:  
> >>
> >> Follow up question: what's the plan to replace this use of
> >> `@types.coroutine` in `asyncio/tasks.py`? [1]
> >>
> >> @types.coroutine
> >> def __sleep0():
> >>  ""
> >>  yield
> >>
> >> [1]
> >> https://github.com/python/cpython/blob/master/Lib/asyncio/tasks.py#L585
> >>
> >> Best,
> >>
> >> Luciano
> >>
> >> On Fri, Feb 19, 2021 at 2:31 AM Luciano Ramalho
> >>  wrote:  
> >> >
> >> > On Fri, Feb 19, 2021 at 1:59 AM Guido van Rossum
> >> >  wrote:  
> >> > >> 1) What Python construct is to be used at the end of a chain
> >> > >> of await calls, if not of a generator-based coroutine
> >> > >> decorated with `@types.coroutine` and using a `yield`
> >> > >> expression in its body?  
> >> >  
> >> > > At the end of the chain you can call the __await__() method
> >> > > which gives an iterator, and then you call next() or send() on
> >> > > that iterator. Each next()/send() call then represents an
> >> > > await step, and send() in general is used to provide an
> >> > > awaited result. Eventually this will raise StopIteration with
> >> > > a value indicating the ultimate result (the return value of
> >> > > the top-level async def).  
> >> >
> >> > All right, that made sense to me. Thank you so much, Guido.
> >> >
> >> > Thanks for the clarification about `@types.coroutine` as well.
> >> >
> >> > Take care,
> >> >
> >> > Luciano
> >> >
> >> >
> >> > --
> >> > Luciano Ramalho
> >> > |  Author of Fluent Python (O'Reilly, 2015)
> >> > | http://shop.oreilly.com/product/0636920032519.do
> >> > |  Technical Principal at ThoughtWorks
> >> > |  Twitter: @ramalhoorg  
> >>
> >>
> >>
> >> --
> >> Luciano Ramalho
> >> |  Author of Fluent Python (O'Reilly, 2015)
> >> | http://shop.oreilly.com/product/0636920032519.do
> >> |  Technical Principal at ThoughtWorks
> >> |  Twitter: @ramalhoorg  
> >
> > --
> > --Guido (mobile)  
> 
> 
> 
> -- 
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> |  Technical Principal at ThoughtWorks
> |  Twitter: @ramalhoorg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4YLGMUXFPWKHQOQI4SEOM4UR2EN5MUNN/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EM6VKVKSIHZTYFKTMZL4NV6HCDQ45N3L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Michał Górny]

On Tue, 2021-02-16 at 00:53 +0100, Victor Stinner wrote:
> Hi Michał,
> 
> I created https://python-security.readthedocs.io/ website to track
> known Python vulnerabilities to help me checking if fixes are
> backported to all supported Python branches. I'm maintaing this list
> manually, it's far from being complete, and likely outdated.
> 
> I also created https://github.com/vstinner/check_python_vuln project
> which implements runtime checks for a few Python vulnerabilities. It
> should help users and Linux packagers to check if their Python has
> known vulnerabilities or not.

Thank you.

> On Thu, Feb 11, 2021 at 9:44 PM Michał Górny  wrote:
> > I feel that vulnerability fixes do not make it to end users fast enough.
> > For example, according to the current release schedules for 3.9 and 3.8,
> > the bugfix releases are planned two months apart. While the release is
> > expected to happen in the next few days, both versions are known to be
> > vulnerable for almost a month!
> 
> We are doing our best to fix all known security vulnerabilities in all
> maintained Python versions (3.6, 3.7, 3.8, 3.9 and master: 5
> branches). But we have no policy to define the severity of these
> vulnerabilities. IMO sometimes we are a little bit too conservative
> when marking some issues are "security" issues.

I generally try to avoid making assumptions about severity of security
bugs, and treat all of them the same.

> For example, the https://bugs.python.org/issue41944 "CJK codecs tests
> call eval() on content received via HTTP" issue only impacts users
> running the Python test suite.
> https://python-security.readthedocs.io/vuln/cjk-codec-download-eval.html
> gives the timeline, extract:
> 
> 2020-10-05 (+0 days): Reported (email sent to the PSRT list)
> (...)
> 2020-10-22 (+17 days): CVE-2020-27619 published
> 2020-12-07 (+63 days): Python 3.9.1 released
> 2020-12-21 (+77 days): Python 3.8.7 released
> 
> IMO here the delay is reasonable (~2 months for 3.8 and 3.9 versions)
> for this kind of vulnerability.

I have ignored this one as well, since we are running the test suite
with network access disabled.  I can't complain for this one but I don't
know whether it doesn't impact other people's workflows.  Please correct
me if I'm missing something but if a distro packager uses a workflow
like 'build, run tests, install' (this is what we do), I believe that
this vulnerability could be used to inject malware into Python packages
distributed with a lot of distributions.

> HTTP header and email header injections vulnerabilities are bad, but
> they can be prevented by applications. For example, good user inputs
> validation should prevent such vulnerability. Example of HTTP header
> injection timeline:
> https://python-security.readthedocs.io/vuln/http-header-injection-method.html
> 
> Do you have a specific kind of vulnerability in mind that would
> require a faster release?

As I said above, I try not to make assumptions about severity.  However,
I personally wouldn't rely on people having their scripts fully
protected or actually doing anything to workaround vulnerabilities
in CPython.  If we have a fix handy, it is our responsibility to deploy
it.

> > 2. When releases happen, security fixes are often combined with many
> > other changes. This causes problems for distribution maintainers who, on
> > one hand, would like to deploy the security fixes to production versions
> > ASAP, and on the other, would prefer that the new version remained in
> > testing for some time due to the other changes.
> 
> We attempt to avoid incompatible changes in 3.x.y bug fix releases. If
> it happens, we can consider to revert it on a case by case basis.
> Usually, it is discussed before merging a change.
> 
> Usually, the incompatible changes that we allow are justified... to
> fix security issues :-)
> 
> Do you have a specific example of problematic incompatible change in
> mind? For me, the largest change was a Python 2.7 release which
> started to check TLS certificates on HTTP... It was to make Python
> more secure to protect users :-)

I didn't mean intended incompatibilities.  I meant the possibility of
new bugs.  I can't say I recall such a problem with CPython but that
doesn't mean that there weren't sneaky issues in the past.  We have
a testing process in place to give braver users a chance to discover
such problems.

> > Furthermore, I think that at least for branches that are in higher level
> > of maintenance than security, it could make sense to actually make
> > security releases (e.g. 3.9.1.x) that would include only security fixes
> > without other changes.
> 
> IMO that's too much work for everybody. developers, testers and packagers.

I do realize that's extra work for people responsible for making
the release but packagers are doing the work already (or should be doing
it).

-- 
Best regards,
Michał Górny

___
Python-Dev mailing list -- python-dev@python.org
To unsubs

[Python-Dev] Re: Security releases of CPython

[Michał Górny]

On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
> On 2/11/2021 3:23 PM, Michał Górny wrote:
> > Hello,
> > 
> > I'm the primary maintainer of CPython packages in Gentoo. I would like
> > to discuss possible improvement to the release process in order to
> > accelerate releasing security fixes to users.
> > 
> > I feel that vulnerability fixes do not make it to end users fast enough.
> 
> When we introduce a bad regression in a release, including a 
> severe-enough security vulnerability, and discover it soon enough, we 
> have sometimes made immediately releases.
> 
> Beyond that, your proposal to add several releases per Python version, 
> perhaps double what we do now, runs up against the limits of volunteer 
> time and effort.  As it is now, becoming a release manager is a 7 year 
> commitment, with at least one release about every other month for the 
> first 4.  There are also the 2 people who produce the Windows and macOS 
> installers.  I believe each has done it for at least 7 or 8 years 
> already.  Releases are not just a push of a button.  Make the release 
> job too onerous, and there might not be any more volunteers.

While I understand your concerns and sympathize with them, I don't think
it's fair to play the volunteer card here.  After all, I am a volunteer
too, and many distribution packagers are as well.  We also have limited
time and many other duties, and it just feels unfair for you to be
dumping the security work on us.

The thing is, CPython upstream has the opportunity to do the central
work that benefits everyone.  You get the first indication
of the vulnerability, you have the first fix and backport to all
branches.  It is only the natural next step to make a release with it. 
While I realize it means more work, not doing that means more work for
a number of other people who are impacted by your prior work.

That said, if the regular release process is too much work, then maybe
a process similar to security-supported branches would work?  After all,
I think it reasonable to assume that a large number of CPython users are
using it via distribution packages and having Python 3.8.x.y source-only
release will be already a big help to people making these packages.

> > For example, according to the current release schedules for 3.9 and 3.8,
> > the bugfix releases are planned two months apart. While the release is
> > expected to happen in the next few days, both versions are known to be
> > vulnerable for almost a month!
> > 
> > Ironically, things look even worse for security-supported versions.
> > Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
> > schedule (planned for Jan 15th), and they are known to be vulnerable
> > since mid-October.
> > 
> > In my opinion, this causes three issues:
> > 
> > 1. Users using official releases are exposed to security vulnerabilities
> > for prolonged periods of time.
> > 
> > 2. When releases happen, security fixes are often combined with many
> > other changes. This causes problems for distribution maintainers who, on
> > one hand, would like to deploy the security fixes to production versions
> > ASAP, and on the other, would prefer that the new version remained in
> > testing for some time due to the other changes.
> > 
> > 3. Effectively, it means that distribution developers need to track
> > and backport security fixes themselves. In the end, this means a lot of
> > duplicate work.
> 
> Perhaps people doing duplicate work could get together to eliminate some of
> the duplication.  It should be possible to filter security commits from the
> python-checkins list by looking at the news entries and if need be, the bpo
> issues.

This makes me think of the 'systemd-stable' repository (where another
maintainer is making bugfix releases to previous versions of systemd). 
I suppose this could work as a last resort but only if it was officially
supported and endorsed by CPython.  However, at this point it's not
really clear why not go the one step further and make the official
releases.

-- 
Best regards,
Michał Górny

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I52QGQVA55BXF425V5IZLF3GYYPWLOKW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597 bikeshedding: envvar / option name.

[Victor Stinner]

Environment variable names with underscore would be more readable, but
IMO consistency with all existing names matters more:
https://docs.python.org/dev/using/cmdline.html#environment-variables

So I prefer PYTHONWARNDEFAULTENCODING env var and
sys.flags.warn_default_encoding attribute names.

Victor

On Fri, Feb 19, 2021 at 9:48 AM Inada Naoki  wrote:
>
> On Mon, Feb 15, 2021 at 3:00 PM Paul Bryan  wrote:
> >
> > Let the bikeshedding begin. How about with the underscores in place? More 
> > readable to my eyes.
> >
>
> I agree with you. Although it is not consistent with existing many
> option names, it is much more readable.
>
> Ivan, Victor, what do you think? about PYTHON_WARN_DEFAULT_ENCODING?
>
> ---
> Inada Naoki  



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SVSPIQFLTLDAYXP3PUXI3F4JZYOB6472/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Tobias Kohn]

 Hi Mark,

Thank you for your proposal to try and have more precise semantics for  
pattern matching.  Of course, the proposal primarily introduces a new  
and extended protocol for pattern matching, upon which the 'semantics'  
is then based.  I think it is important to recognise and discuss your  
work on this basis that it proposes a different protocol of how  
pattern matching works, before we consier whether your semantics  
really is 'more precise'.


As I understand it, your PEP proposes two major differences: control  
over the matching process moves from patterns to the objects and a  
custom protocol is introduced (via the `__deconstruct__` method).   
There is also the matter of the `__match_kind__` with the intention of  
improving performance, which I find of minor importance here.


If you go back and look at the history and development of PEP 634, you  
will find that we started off with a fully customisable method that we  
called `__match__` instead of `__deconstruct__`.  The idea of more  
flexible and customisable pattern matching is thus certainly in the  
spirit of our proposal as well.  However, we finally removed this part  
from our proposal due to strong concerns from the community that the  
match protocol is too complex, particularly for an initial version of  
pattern matching.  Should the need arise, it would still be possible  
to later add such a customisation protocol in an upcoming version of  
Python.


Given these concerns with respect to complexity and our concession to  
remove the `__match__` method, I am wondering: do you have strong  
arguments that would speak for inclusion of this feature in the first  
version of pattern matching after all?  I would love to have this  
feature and am truly interested in any arguments in its favour.


When pattern matching meets OOP, there is indeed some discussion that  
can be had on where the matching itself is to happen.  Simply put:  
should an object recognise that it is an instance of a given class, or  
should the class recognise that an object is an instance of it?


In the spirit of Python's `instancecheck` method, we opted for  
classes/patterns recognising whether an object matches, where the  
object itself is 'passive'.  In other words, `case C()` would match  
any object that is an instance of a subclass of `C`, in line with  
usual OOP principles (where any instance of a subclass of `C` is still  
a valid instance of `C` itself).


Your design turns this around and has the object take on the  
responsibility of checking whether it wants to match a specific  
pattern.  This has the advantage that the object itself can morph into  
anything it wants to—as long as it is aware of the pattern.  And  
therein lies the problem that IMHO fully negates one of the major  
advantages of pattern matching: that the patterns and the objects that  
are to be matched are de-coupled and fully independent.  In other  
words: in your design it is not she who writes the patterns that  
controls what objects match, but he who designs the class hierarchy of  
the objects used.


One of the great traits of pattern matching is that it works  
out-of-the-box even with objects from ancient or alien code bases.   
Pattern matching according to PEP 634 is isolated, it happens only  
where pattern matching is explicitly used.  When writing classes, you  
do not have to worry about pattern matching at all, unless you want  
your classes to be used as patterns.  And should we decide some time  
in the future that we want to introduce new kinds of patterns, we  
still do not have to change a single object because the matching  
happens in the patterns and not in the objects, anyway.  That his  
would not be the case with your design is already obvious with the  
list of `__match_kind__` for common builtin classes, for instance.


Although there are some other programming languages that chose to  
follow this route, I think it goes against the spirit of what is  
already there in Python, it violates the principle of separated  
concerns and is thus a rather ill-advised modification of the original  
protocol.


Kind regards,
Tobias
 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7UIKPFTOU6APJGHW35A3P4JV4N6PL7K7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Guido van Rossum]

I think that for this functionality (force the event loop to run), the
trampoline just needs to define a function that returns a magic value and
special-case that. The helper then becomes

def __sleep0():
await 

The magic object may have to be something with an __await__() method.

On Fri, Feb 19, 2021 at 01:34 Luciano Ramalho  wrote:

> On Fri, Feb 19, 2021 at 6:29 AM Luciano Ramalho 
> wrote:
> > async def __sleep():
> > return None
>
> Sorry, I meant to write:
>
> async def __sleep0():
> return None
>
> Since the idea is to replace the generator-based coroutine `__sleep0`
> in tasks.py [1] with a native coroutine.
>
> [1]
> https://github.com/python/cpython/blob/e92d67dfbb4790df37aa6a0961fb6dc7e8d2fbbf/Lib/asyncio/tasks.py#L585
>
> --
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> |  Technical Principal at ThoughtWorks
> |  Twitter: @ramalhoorg
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IB5E552MRHZTGX3IQN6OKSDY3DDGA4D3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Tobias Kohn]

 Hi Oscar,

Quoting Oscar Benjamin :

Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but  
I can see some parts that I definitely like. [...]


As I have just noted in my response to Mark, the aspect with the  
"deconstructor" (or `__match__` protocol as we called it) is  
definitely something that I like, too.  Moreover, I think that  
packages like sympy might make a strong argument for it.


That being said, perhaps we would have to start thinking about  
concrete use cases first and then consider how to best provide for  
those with an extended match protocol.  Even though I still like the  
idea of a flexible match method, there might be aspects that I/we  
overlook without knowing exactly where we want to go with this.


I'm not entirely sure but I think that with PEP 653 you can  
implement this like:


   def __deconstruct__(obj):
       if obj.step != 1:
           return obj.start, obj.stop, obj.step
       elif obj.start != 0:
           return obj.start, obj.stop
       else:
           return obj.stop

I think that would then mean you could use a pattern range(10) to  
unambiguously match range(10) without matching range(10, 20) etc.


This is certainly more of an anti-pattern (no pun intended) than an  
argument for the match protocol.  Shuffling positional arguments  
around like this seems like a rather bad idea, particularly in the  
context of pattern matching.  I'd rather write `case range(_, stop,  
_)` or `case range(0, stop, 1)` and be explicit here.


Kind regards,
Tobias

 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PWJSQIQ7XHJRD7ZN7QWT6HF5MGMG47GN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.9.2 and 3.8.8 are now available

[Łukasz Langa]

Convinced of the wonders of free two-day deliveries, I’m pleased to present you 
Python 3.9.2 and 3.8.8. Get them from:

https://www.python.org/downloads/release/python-392/ 


https://www.python.org/downloads/release/python-388/ 


Next up, the last full regular maintenance release of Python 3.8 is planned for 
May 3rd 2021, after which it will shift to source releases only for security 
bug fixes only. Maintenance releases for the 3.9 series will continue at 
regular bi-monthly intervals, with 3.9.3 planned for early May 2021.

Why the expedited final release?

This release, just as the candidate before it, contains two security fixes:

bpo-42938 : Avoid static buffers when 
computing the repr of ctypes.c_double and ctypes.c_longdouble values. This 
issue was assigned CVE-2021-3177 
.

bpo-42967 : Fix web cache poisoning 
vulnerability by defaulting the query args separator to &, and allowing the 
user to choose a custom separator. This issue was assigned CVE-2021-23336 
.

Since the announcement of the release candidates for 3.9.2 on 3.8.8, we 
received a number of inquiries from end users urging us to expedite the final 
releases due to the security content, especially CVE-2021-3177 
.

This took us somewhat by surprise since we believed security content is 
cherry-picked by downstream distributors from source either way, and the RC 
releases provide installers for everybody else interested in upgrading in the 
meantime. It turns out that release candidates are mostly invisible to the 
community and in many cases cannot be used due to upgrade processes which users 
have in place.

In turn, the other active release managers and I decided to stop providing 
release candidates for bugfix versions. Starting from now on after the initial 
3.x.0 final release, all subsequent releases are going to be provided as is in 
bi-monthly intervals. The release calendar PEPs for 3.8 and 3.9 have been 
updated accordingly.

On the severity of CVE-2021-3177

We recommend you upgrade your systems to Python 3.8.8 or 3.9.2.

Our understanding is that while the CVE is listed as “remote code execution”, 
practical exploits of this vulnerability as such are very unlikely due the 
following conditions needing to be met for successful RCE:

pass an untrusted floating point number from a remote party to 
ctypes.c_double.from_param (note: Python floating point numbers were not 
affected);

have that object be passed to repr() (for instance through logging);

have that float point number be valid machine code;

have the buffer overflow overwrite the stack at exactly the right place for the 
code to get executed.

In fact, Red Hat’s evaluation of the vulnerability was consistent with ours. 
They write: “the highest threat from this vulnerability is to system 
availability .”

To be sure, denial of service through malicious input is also a serious issue. 
Thus, to help the community members for whom the release candidate was 
insufficient, we are releasing the final versions of 3.9.2 and 3.8.8 today.

What’s new?

The Python 3.9 series contains many new features and optimizations over 3.8. 
See the “What’s New in Python 3.9 2 
” document for more information 
about features included in the 3.9 series. We also have a detailed change log 
for 3.9.2rc1 

 specifically. The final release only contains a single bugfix 
 over the release candidate.

Detailed information about all changes made in version 3.8.8rc1 specifically 
can be found in its respective changelog 
.
 The final version contains no changes over the release candidate.

We hope you enjoy those new releases!

Thanks to all of the many volunteers who help make Python Development and these 
releases possible! Please consider supporting our efforts by volunteering 
yourself or through organization contributions to the Python Software 
Foundation.

Your friendly release team,
Ned Deily @nad 
Steve Dower @steve.dower 
Łukasz Langa @ambv ___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https

[Python-Dev] Re: PEP 597 bikeshedding: envvar / option name.

[Paul Bryan]

As long as we don't encounter sys.flags.pen_is_mightier type of cases,
I guess it'll be OK. :-)

On Fri, 2021-02-19 at 12:08 +0100, Victor Stinner wrote:
> Environment variable names with underscore would be more readable,
> but
> IMO consistency with all existing names matters more:
> https://docs.python.org/dev/using/cmdline.html#environment-variables
> 
> So I prefer PYTHONWARNDEFAULTENCODING env var and
> sys.flags.warn_default_encoding attribute names.
> 
> Victor
> 
> On Fri, Feb 19, 2021 at 9:48 AM Inada Naoki 
> wrote:
> > 
> > On Mon, Feb 15, 2021 at 3:00 PM Paul Bryan  wrote:
> > > 
> > > Let the bikeshedding begin. How about with the underscores in
> > > place? More readable to my eyes.
> > > 
> > 
> > I agree with you. Although it is not consistent with existing many
> > option names, it is much more readable.
> > 
> > Ivan, Victor, what do you think? about
> > PYTHON_WARN_DEFAULT_ENCODING?
> > 
> > ---
> > Inada Naoki  
> 
> 
> 

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6NJ3E2PNF3QZBYH57UOTCHQL4IWGTLNP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi,

I wish I'd read this before replaying your last email

On 18/02/2021 6:49 pm, Brandt Bucher wrote:

Brandt Bucher wrote:

For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, 
y=y, z=z)`: ...



It also appears that we lose a lot of expressive "idioms" by requiring 
`__attributes__` to be complete.



This also means that matching classes like `types.SimpleNamespace` are much less powerful 
under PEP 653, since the class must know which attributes are "allowed" to be 
looked up.


Never mind these three points... I *think* setting `__match_kind__ = 
MATCH_DEFAULT` allows for arbitrary attribute extraction like this. Perhaps 
make it bit clearer?


Not quite. It uses the instance dictionary to avoid capturing bound 
methods and other computed attributes.


I'll make this more prominent.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QNASJRQ63QV5DML3LL4WIV34NKRHLWGL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi Brandt,

On 18/02/2021 5:32 pm, Brandt Bucher wrote:

Thanks for taking the time to work on this, Mark.

Overall, I'm skeptical of this proposal. It seems like it takes a lot of "simple" things 
and makes them quite complex, and takes many "static" things and makes them quite 
dynamic. I feel that it also misrepresents certain aspects of PEP 634.

Here's a more detailed commentary:


Pattern matching will be more usable for complex classes, by allowing classes 
more control over which patterns they match.


I fear this is at the expense of most simple classes, which currently "just 
work" with PEP 634. I'm not convinced that making it easier for very complicated 
classes (like those in `sympy`) to participate at the expense of everyday classes is a 
win.


In my experience, nothing "just works".
It means "mostly works most of the time, and occasionally goes horribly 
wrong"





For comparison, here is what (as I understand it) each PEP requires for a class 
to be used in a pattern such as `C(x, y, z)`:

```py
class C:
 """A PEP 634 matchable class."""
 __match_args__ = ...

class C:
 """A PEP 653 matchable class."""
 __match_kind__ = MATCH_CLASS
 __attributes__ = ...
 def __deconstruct__(self):
 ...
```


True, but most pure-Python classes don't have a natural order of 
attributes, so the pattern is likely to be `C(x=x,y=y,z=z)`.


In which case no extra work is required for either PEP to pattern match.



For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, 
y=y, z=z)`:

```py
class C:
 """A PEP 634 matchable class (all classes work)."""

class C:
 """A PEP 653 matchable class."""
 __match_kind__ = MATCH_CLASS
 __attributes__ = ...
 def __deconstruct__(self):
 ...
```


This is not true. MATCH_DEFAULT does what you want.



It also appears that we lose a lot of expressive "idioms" by requiring 
`__attributes__` to be complete. For example, the following attribute extractions work 
with PEP 634, but not PEP 653.

```py
match subject:
 case object(a=_):
 # Match any object with an "a" attribute.
 ...
 case object(x=x, y=y, z=z):
 # Match any object with "x", "y", and "z" attributes, extracting them.
 ...
```


You need to define what you mean by "work" :)

This behaves the same for both PEPs:

class C:
__init__(self, x, y, z):
...

match C(1,2,3):
case object(x=x, y=y, z=z):
...

This also works the same for both PEPs, albeit with different mechanisms:

match namedtuple("n", "x, y, z")(1,2,3):
case object(x=x, y=y, z=z):
...


But, with PEP 634:

>>> match(sympy.cos(x)):
>>>case object(n):
>>>...
>>> n


Which seem like "not working" to me.



This also means that matching classes like `types.SimpleNamespace` are much less powerful 
under PEP 653, since the class must know which attributes are "allowed" to be 
looked up.


Why do you say this?

This should work the same for PEP 634 and 653:

>>> match types.SimpleNamespace(a = 1):
>>> case object(a):
>>>print(a)
1



Further, this whole specification shifts most of the matching logic from the 
class in the pattern to the subject itself:

```py
match ChildA(), ChildB():
 case Parent(a, b), Parent(x, y):
 ...
```

The above pattern could have completely different rules for extracting `a` and 
`b` vs `x` and `y` if either child overrides `__deconstruct__`.  I think that 
is a mistake.


Inheritance allows sub-classes to change the behavior of their 
super-classes. Yes, it can produce weird effects if they ignore Liskov's 
substitution principle. That why the modern advice is "Favour 
composition over inheritance".


You can't prevent people writing bad code.




PEP 634 also privileges some builtin classes with a special form of matching, the 
"self" match. For example the pattern `list(x)` matches a list and assigns the 
list to `x`. By allowing classes to choose which kinds of pattern they match, other 
classes can use this form as well.


This is already fairly trivial to implement:

```py
class C:
 __match_args__ = ("_self",)
 _self = property(lambda s: s)
```


That's pretty obscure, IMO. You can't claim that it is trivial compared to:

class C:
__match_kind__ = MATCH_SELF




You can even avoid adding a `_self` attribute if you do some magic with 
descriptors... ;). We could consider provide a decorator for this somewhere in 
the stdlib if there's demonstrated need (but I'm not sure there will be).


All classes should ensure that the the value of `__match_kind__` follows the 
specification. Therefore, implementations can assume, without checking, that 
all the following are false:
`(__match_kind__ & (MATCH_SEQUENCE | MATCH_MAPPING)) == (MATCH_SEQUENCE | 
MATCH_MAPPING)`
`(__match_kind__ & (MATCH_SELF | MATCH_CLASS)) == (MATCH_SELF | MATCH_CLASS)`
`(__match_kind__ & (MATCH_SELF | MATCH_DEFAULT)) == (MATCH_SELF | 
MATCH_DEFAULT)`
`(__match_kind__ 

[Python-Dev] Re: Python 0.9.1

[David Mertz]

In conversation with Dan, I have fixed my conda package (but overwritten
the same version).  I needed to add this to the build:

# sudo apt-get install gcc-multilib
CC='gcc -m32' make python

I don't have 32-bit headers by default anymore on my distro.  With that
change, I can run:

% conda install -c davidmertz python=0.9
% python
>>> 2*2
4

Dan sent me links to some patches he developed himself, but I have not
applied them.  I'm not doubting that they are good and useful, but I'm
trying to make something as close as possible to what was actually first
released.

-- 
The dead increasingly dominate and strangle both the living and the
not-yet born.  Vampiric capital and undead corporate persons abuse
the lives and control the thoughts of homo faber. Ideas, once born,
become abortifacients against new conceptions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3B4EW5IF6OQ33CQRGXC3H57LG3KSQ6F5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Tobias Kohn]

 Hi Mark,

Quoting Mark Shannon :


[...]

If you had published these "more complicated, powerful protocols",  
you might be able to claim that this is a "rehash".

But you didn't.
 


I would say that these ideas have been quite prominently published:
https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol

https://dl.acm.org/doi/10.1145/3426422.3426983

Moreover, much of the discussion is publicly available, for instance here:
https://github.com/gvanrossum/patma/issues/8

Cheers,
Tobias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WS2DFBMBRTF5YXZS6IG42QNJNDJHPBYN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2021-02-12 - 2021-02-19)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7428 (+13)
  closed 47578 (+49)
  total  55006 (+62)

Open issues with patches: 2943 


Issues opened (43)
==

#37146: opcode cache for LOAD_GLOBAL emits false alarm in memory leak 
https://bugs.python.org/issue37146  reopened by vstinner

#42990: Improve the C code for calling Python code: _PyEval_EvalCode()
https://bugs.python.org/issue42990  reopened by vstinner

#43209: system cannot find the file specified in subprocess.py
https://bugs.python.org/issue43209  opened by robertjrichardson

#43211: Python is not responding after running program
https://bugs.python.org/issue43211  opened by diasybarrett

#43212: Link to class and function source code on GitHub
https://bugs.python.org/issue43212  opened by boris

#43214: site: Potential UnicodeDecodeError when handling pth file
https://bugs.python.org/issue43214  opened by methane

#43215: Document Happy Eyeballs arguments of asyncio.open_connection
https://bugs.python.org/issue43215  opened by illia-v

#43216: Removal of @asyncio.coroutine in Python 3.10
https://bugs.python.org/issue43216  opened by illia-v

#43219: shutil.copy raises IsADirectoryError when the directory does n
https://bugs.python.org/issue43219  opened by jerpint

#43220: Argparse: Explicit default required arguments with add_mutuall
https://bugs.python.org/issue43220  opened by keith

#43223: [security] http.server: Open Redirection if the URL path start
https://bugs.python.org/issue43223  opened by hamzaavvan

#43224: Add support for PEP 646 (Variadic Generics) to typing.py
https://bugs.python.org/issue43224  opened by mrahtz

#43225: Add missing value returned by methods on cookiejar
https://bugs.python.org/issue43225  opened by eamanu

#43226: build from 3.9.1 tarball on WSL failed
https://bugs.python.org/issue43226  opened by DoctorDan

#43228: Regression in function builtins
https://bugs.python.org/issue43228  opened by vstinner

#43229: freeze searches libpython3.9.so in /usr/lib instead /usr/lib/x
https://bugs.python.org/issue43229  opened by chba

#43232: Prohibit previously deprecated operations on asyncio.trsock.Tr
https://bugs.python.org/issue43232  opened by illia-v

#43233: test_os: test_copy_file_range_offset fails on FreeBSD CURRENT
https://bugs.python.org/issue43233  opened by pablogsal

#43234: Prohibit passing non-ThreadPoolExecutor executors to loop.set_
https://bugs.python.org/issue43234  opened by illia-v

#43235: Tools/scripts/stable_abi.py should also check PC/python3dll.c 
https://bugs.python.org/issue43235  opened by vstinner

#43236: Windows IDLE taskbar icon jump list fails to open recent files
https://bugs.python.org/issue43236  opened by veganaiZe

#43239: PyCFunction_New is not exported with PyAPI_FUNC
https://bugs.python.org/issue43239  opened by petr.viktorin

#43242: Add Void type to typing library
https://bugs.python.org/issue43242  opened by ocket

#43244: Move PyArena C API to the internal C API
https://bugs.python.org/issue43244  opened by vstinner

#43245: Add keyword argument support to ChainMap.new_child()
https://bugs.python.org/issue43245  opened by rhettinger

#43246: Dict copy optimization violates subclass invariant
https://bugs.python.org/issue43246  opened by jab

#43247: How to search for a __main__ module using pyclbr in Python3?
https://bugs.python.org/issue43247  opened by kebab-mai-haddi

#43248: platform.libc_ver() returns no information on Alpine Linux
https://bugs.python.org/issue43248  opened by Omer.Katz

#43250: [C API] Depreate or remove PyFPE_START_PROTECT() and PyFPE_END
https://bugs.python.org/issue43250  opened by vstinner

#43251: sqlite3_column_name() failures should raise MemoryError
https://bugs.python.org/issue43251  opened by erlendaasland

#43252: deepcopy of weakref proxies
https://bugs.python.org/issue43252  opened by konrad.schwarz

#43253: asyncio open_connection fails when a socket is explicitly clos
https://bugs.python.org/issue43253  opened by danielen1337

#43256: Document that PyErr_SetInterrupt is async-signal-safe
https://bugs.python.org/issue43256  opened by pitrou

#43257: get_type_hints evaluates class variables before type hints
https://bugs.python.org/issue43257  opened by xirdneh

#43259: argparse: allow add_mutually_exclusive_group on add_argument_g
https://bugs.python.org/issue43259  opened by calestyo

#43260: Never release buffer when MemoryError in print()
https://bugs.python.org/issue43260  opened by Ramin Farajpour Cami

#43261: Python 3.9 repair on Windows do not recover pip module
https://bugs.python.org/issue43261  opened by Prometheus3375

#43262: "_weakrefset.py", line 56: IndexError: pop from empty list
https://bugs.python.org/issue43262  opened by jhshi89

#43263: threading module unable to run
https://bugs.python.org/issue43263  opened by tw

[Python-Dev] Re: Python 0.9.1

[Skip Montanaro]

> In conversation with Dan, I have fixed my conda package (but overwritten
the same version).  I needed to add this to the build:
>
> # sudo apt-get install gcc-multilib
> CC='gcc -m32' make python

Thanks. That fixes it for me as well. I never even looked at intobject.c,
since it compiled out of the box, and didn't dig into it when I saw the
error. Looking now, I see a 32-bit assumption:

if (x > 0x7fff || x < (double) (long) 0x8000)
return err_ovf();


With the -m32 flag, running lib/testall.py runs to completion.

Skip
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S3SIGIAWOORGVDM6ZOR6UOG3T2RUM5TX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi Tobias,

On 19/02/2021 5:57 pm, Tobias Kohn wrote:
> Hi Mark,
>
> Quoting Mark Shannon mailto:m...@hotpy.org>>:
>
>> [...]
>>
>> If you had published these "more complicated, powerful protocols", you
>> might be able to claim that this is a "rehash".
>> But you didn't.
>>
> I would say that these ideas have been quite prominently published:
> https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol

But they are not referenced in PEP 634. I shouldn't have to trawl the 
internet to find the rejected ideas section.


>
> https://dl.acm.org/doi/10.1145/3426422.3426983

That paper describes a `__match__` method, which is absent from PEP 634.
Why?


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L7REZUBVMDRPRXX2TMWYMOWGISRGDHJR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Tobias Kohn]

 Hi Mark,

Quoting Mark Shannon :


Hi Tobias,

[...]

But they are not referenced in PEP 634. I shouldn't have to trawl  
the internet to find the rejected ideas section.



https://dl.acm.org/doi/10.1145/3426422.3426983


That paper describes a `__match__` method, which is absent from PEP 634.
Why?

Cheers,
Mark.



You are right: you should not have to trawl the internet to get these  
information.  However, given the history of the Pattern Matching PEPs  
and that PEP 622 is linked directly from PEP 634, I would think it is  
not that far a journey—even though there have been so many discussions  
that it has definitely become unwieldy to retain an overview...


Anyway, the answer to both your questions lies in that the Pattern  
Matching feature itself is rather complex and the PEPs ended up being  
huge and hard to read and understand as it is.  We therefore refrained  
from long lists of rejected ideas in PEP 634, since that has already  
been done in PEP 622.  Moreover, the `__match__` method and protocol  
were removed from the PEPs to focus on a core infrastructure and keep  
it as simple as possible.


I think such a customisable protocol will eventually add great value  
to pattern matching.  However, one of the main arguments was to wait  
until we have more experience with pattern matching in Python and can  
give specific use cases for this extended protocol, which allow us to  
guide the design of the protocol.


At the end of the day, I guess we quite agree that we would like to  
have `__match__`, `__deconstruct__` or whatever you want to name it.   
But the slight variations in design demonstrate that it might also be  
a good idea to carefully lay it out first, before adding it to the  
pattern matching engine.  Hence, separating this from the core  
implementation seems like a good idea that I still fully support  
(despite also looking forward to having it some day in the future ^_^).


Cheers,
Tobias
 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T6Y7DCKIXFYV3UQY7BHTARUBPUQ6BI2C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Brett Cannon]

On Fri, Feb 19, 2021 at 2:16 AM Michał Górny  wrote:

> On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
> > On 2/11/2021 3:23 PM, Michał Górny wrote:
> > > Hello,
> > >
> > > I'm the primary maintainer of CPython packages in Gentoo. I would like
> > > to discuss possible improvement to the release process in order to
> > > accelerate releasing security fixes to users.
> > >
> > > I feel that vulnerability fixes do not make it to end users fast
> enough.
> >
> > When we introduce a bad regression in a release, including a
> > severe-enough security vulnerability, and discover it soon enough, we
> > have sometimes made immediately releases.
> >
> > Beyond that, your proposal to add several releases per Python version,
> > perhaps double what we do now, runs up against the limits of volunteer
> > time and effort.  As it is now, becoming a release manager is a 7 year
> > commitment, with at least one release about every other month for the
> > first 4.  There are also the 2 people who produce the Windows and macOS
> > installers.  I believe each has done it for at least 7 or 8 years
> > already.  Releases are not just a push of a button.  Make the release
> > job too onerous, and there might not be any more volunteers.
>
> While I understand your concerns and sympathize with them, I don't think
> it's fair to play the volunteer card here.  After all, I am a volunteer
> too, and many distribution packagers are as well.  We also have limited
> time and many other duties, and it just feels unfair for you to be
> dumping the security work on us.
>

I also don't think it's fair to say anyone is "dumping security work" on
you. Volunteerism is about accepting that I can't ask anything of you while
you can't ask anything of me. We all do our best here, but it's making
choices with very finite resources and our release and security approach is
the one we have made based on what we have to work with. If people would
like to see more frequent releases then it will most likely require people
volunteering to help with the security workload as well as with the release
process, both of which are rather thankless jobs unfortunately. But simply
because we have differing views on how to handle security doesn't mean
anyone is actively "dumping" anything on anyone.


>
> The thing is, CPython upstream has the opportunity to do the central
> work that benefits everyone.  You get the first indication
> of the vulnerability, you have the first fix and backport to all
> branches.  It is only the natural next step to make a release with it.
> While I realize it means more work, not doing that means more work for
> a number of other people who are impacted by your prior work.
>
> That said, if the regular release process is too much work, then maybe
> a process similar to security-supported branches would work?  After all,
> I think it reasonable to assume that a large number of CPython users are
> using it via distribution packages and having Python 3.8.x.y source-only
> release will be already a big help to people making these packages.
>

But a very large number of people get their releases from python.org, so if
we do source-only releases without pushing out an install that then causes
a schism between what users get on one platform versus another that goes
beyond simply the delivery mechanism (and this isn't hypothetical;
downstream packages by Linux distros have caused issues for users due to
differing from what python.org pushes out, so we do need to consider this
impact).

-Brett


>
> > > For example, according to the current release schedules for 3.9 and
> 3.8,
> > > the bugfix releases are planned two months apart. While the release is
> > > expected to happen in the next few days, both versions are known to be
> > > vulnerable for almost a month!
> > >
> > > Ironically, things look even worse for security-supported versions.
> > > Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
> > > schedule (planned for Jan 15th), and they are known to be vulnerable
> > > since mid-October.
> > >
> > > In my opinion, this causes three issues:
> > >
> > > 1. Users using official releases are exposed to security
> vulnerabilities
> > > for prolonged periods of time.
> > >
> > > 2. When releases happen, security fixes are often combined with many
> > > other changes. This causes problems for distribution maintainers who,
> on
> > > one hand, would like to deploy the security fixes to production
> versions
> > > ASAP, and on the other, would prefer that the new version remained in
> > > testing for some time due to the other changes.
> > >
> > > 3. Effectively, it means that distribution developers need to track
> > > and backport security fixes themselves. In the end, this means a lot of
> > > duplicate work.
> >
> > Perhaps people doing duplicate work could get together to eliminate some
> of
> > the duplication.  It should be possible to filter security commits from
> the
> > python-checkins list by looking at t

[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Brett Cannon]

On Thu, Feb 18, 2021 at 9:04 PM Guido van Rossum  wrote:

> On Thu, Feb 18, 2021 at 8:23 PM Luciano Ramalho 
> wrote:
>
>> Thanks for your reply, Guido.
>>
>> On Fri, Feb 19, 2021 at 12:07 AM Guido van Rossum 
>> wrote:
>> > Reading the doc section you link to, it's pretty clear that
>> `@asyncio.coroutine` will be removed.
>>
>> The *Note* right at the top of [1] says "Support for generator-based
>> coroutines is deprecated and is scheduled for removal in Python 3.10."
>>
>> [1]
>> https://docs.python.org/3/library/asyncio-task.html#generator-based-coroutines
>>
>> But PEP 492 [2] says:
>>
>> "Since, internally, coroutines are a special kind of generators, every
>> await is suspended by a yield somewhere down the chain of await calls"
>>
>> [2] https://www.python.org/dev/peps/pep-0492/#await-expression
>>
>> If that part of PEP 492 is no longer accurate, then I have a couple of
>> questions:
>>
>> 1) What Python construct is to be used at the end of a chain of await
>> calls, if not of a generator-based coroutine decorated with
>> `@types.coroutine` and using a `yield` expression in its body?
>>
>> 2) Given that the sole purpose of `@types.coroutine` is to decorate
>> generator-based coroutines to become awaitable, will that decorator
>> also be removed, along with "support for generator-based coroutines"?
>>
>> Best,
>>
>> Luciano
>>
>
> It looks like types.coroutine will remain (it does not contain code to
> warn about deprecation like asyncio.coroutine does), but I don't think it
> is required to end a chain of coroutines -- it may have been an oversight
> that we did not start deprecating it. (But in any case it won't be
> supported by asyncio.)
>
> At the end of the chain you can call the __await__() method which gives an
> iterator, and then you call next() or send() on that iterator. Each
> next()/send() call then represents an await step, and send() in general is
> used to provide an awaited result. Eventually this will raise StopIteration
> with a value indicating the ultimate result (the return value of the
> top-level async def).
>
> The code used to "drive" a chain of await calls is called a trampoline.
> But writing a trampoline is not easy, and the only example I know of is
> asyncio's Task class, in particular its __step() method, which of course is
> beyond complicated because it has to handle so many special cases.
>

I've found https://github.com/dabeaz/curio to be the simplest,
full-featured Python event loop to read.

-Brett


>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CEQQLON7A5D64V7VAPH7OCTULG2HZPHQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CSRXFPGEZO6LFDHM2Q5USMVWXIIDK42R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification on the removal of generator-based coroutines in 3.10

[Abdur-Rahmaan Janhangeer]

Thanks for Mr David beazley's materials online, a beginner like me was able
to understand this thread from the beginning. Rarely do i get to understand
what the folks in here are rumbling about.

He was also teaching compiler theory at university. Don't know if he ever
contributed to CPython. Here's a direct effect of him on me:
https://www.pythonkitchen.com/python-generators-in-depth/

Though i consider Mr Paul's comments as near trolling, i was able to
appreciate his perspective this time

If ever there is a newbie like me on this list, i'd recommend to vaccum in
as much as possible from mr Beazley ^^
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZBIVNECI36OFZ3H6WYJTJ74OWTRFNUF2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Terry Reedy]

On 2/19/2021 5:11 AM, Michał Górny wrote:

On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:



Releases are not just a push of a button.  Make the release
job too onerous, and there might not be any more volunteers.


While I understand your concerns and sympathize with them,


Your accusations of unfairness in response to my polite, volunteered 
response, suggest otherwise.



I don't think it's fair to play the volunteer card here.

...

it just feels unfair for you to be dumping the security work on us.


Which I did not do.  Bye.

--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UOWWSWZ3O4NV6PIQBMF2KKBVSTK6HXL3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Stestagg]

The thing that stood out from this conversation, for me, is: Releases are
too hard, and there’s a risk of not having enough volunteers as a result.

How hard is it to fix that?

Steve

On Fri, 19 Feb 2021 at 22:05, Terry Reedy  wrote:

> On 2/19/2021 5:11 AM, Michał Górny wrote:
> > On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
>
> >> Releases are not just a push of a button.  Make the release
> >> job too onerous, and there might not be any more volunteers.
> >
> > While I understand your concerns and sympathize with them,
>
> Your accusations of unfairness in response to my polite, volunteered
> response, suggest otherwise.
>
> > I don't think it's fair to play the volunteer card here.
> ...
> > it just feels unfair for you to be dumping the security work on us.
>
> Which I did not do.  Bye.
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UOWWSWZ3O4NV6PIQBMF2KKBVSTK6HXL3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2OXDTSZENCMKBOPWXW5E77LM7ENC6PBU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Christian Heimes]

On 19/02/2021 23.22, Stestagg wrote:
> The thing that stood out from this conversation, for me, is: Releases
> are too hard, and there’s a risk of not having enough volunteers as a
> result.
> 
> How hard is it to fix that? 

Actually it's easy to fix!

The PSF needs needs sufficient money to hire a couple of people, so the
PSF can turn release management and security maintenance from unpaid
volunteers into paid fulltime jobs.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4INCHKPOGKMTDQ4IDS4RDR33KJ4RR2H5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Stestagg]

On Fri, 19 Feb 2021 at 22:50, Christian Heimes  wrote:

> On 19/02/2021 23.22, Stestagg wrote:
> > The thing that stood out from this conversation, for me, is: Releases
> > are too hard, and there’s a risk of not having enough volunteers as a
> > result.
> >
> > How hard is it to fix that?
>
> Actually it's easy to fix!
>
> The PSF needs needs sufficient money to hire a couple of people, so the
> PSF can turn release management and security maintenance from unpaid
> volunteers into paid fulltime jobs.
>
That’s certainly one option. I’d be super surprised if that were the only
one.

What were the budget requirements for this? How far from the target is the
PSF?  Is there a plan to hit it?

Are there no technical solutions that might help reduce the cost?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/55GFSMWJOG7P6NWU4KDHMKJN2OGX5YSG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Oscar Benjamin]

On Fri, 19 Feb 2021 at 16:27, Tobias Kohn  wrote:
>
> Quoting Oscar Benjamin :
>
> > I'm not entirely sure but I think that with PEP 653 you can implement this 
> > like:
> >
> >   def __deconstruct__(obj):
> >   if obj.step != 1:
> >   return obj.start, obj.stop, obj.step
> >   elif obj.start != 0:
> >   return obj.start, obj.stop
> >   else:
> >   return obj.stop
> >
> > I think that would then mean you could use a pattern range(10) to 
> > unambiguously match range(10) without matching range(10, 20) etc.
>
> This is certainly more of an anti-pattern (no pun intended) than an argument 
> for the match protocol.  Shuffling positional arguments around like this 
> seems like a rather bad idea, particularly in the context of pattern 
> matching.  I'd rather write `case range(_, stop, _)` or `case range(0, stop, 
> 1)` and be explicit here.

Good point! I was trying to reason out the mechanism but the
alternative approach as you say would be to always use all 3
arguments. PEP 653 here actually makes it possible for the range class
to affect matching so that it can only succeed against a pattern that
uses all 3 arguments.

The significance of range as an example is basically that it has a
varying number of positional only arguments to its constructor. The
most important feature of the example is not the reordering of
arguments: it is the fact that with PEP 634 a pattern range(x) would
have to match range(x, y) which is very unlikely to be useful in the
case of range but also not generally applicable for other types
either. There is no way for range (or other types) to prevent that
while supporting positional argument matching through __match_args__.

Under PEP 634 in general, for any class C, a pattern C(x) matches an
object C(x, y) and there's no way for C to override that. To me that
is sufficiently unintuitive in the abstract that no example is really
needed to see where there is room for improvement.

--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/43UTGYJHOEZSMOFA7WHCJ6ERR2JFQPUE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Brandt Bucher]

Oscar Benjamin wrote:
> Under PEP 634 in general, for any class C, a pattern C(x) matches an object 
> C(x, y) and there's no way for C to override that. To me that is sufficiently 
> unintuitive in the abstract that no example is really needed to see where 
> there is room for improvement.

We originally kicked around (and at one point even implemented) a 
`__match_args_required__` attribute, which is an integer specifying a minimum 
required number of positional sub-patterns. For reasons I can't recall, though, 
it was eventually dropped. It would take care of this, and I imagine it could 
be quite painless to add it back if there was enough support.

Here's a query for related discussions: 
https://github.com/gvanrossum/patma/issues?q=__match_args_required__
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKZUE2BUDZQQHQLZK3IYBUNHYQWYMVA6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Oscar Benjamin]

On Fri, 19 Feb 2021 at 15:41, Tobias Kohn  wrote:
>
> Hi Mark,
>
> Thank you for your proposal to try and have more precise semantics for 
> pattern matching.  Of course, the proposal primarily introduces a new and 
> extended protocol for pattern matching, upon which the 'semantics' is then 
> based.  I think it is important to recognise and discuss your work on this 
> basis that it proposes a different protocol of how pattern matching works, 
> before we consier whether your semantics really is 'more precise'.
...
>
> If you go back and look at the history and development of PEP 634, you will 
> find that we started off with a fully customisable method that we called 
> `__match__` instead of `__deconstruct__`.  The idea of more flexible and 
> customisable pattern matching is thus certainly in the spirit of our proposal 
> as well.  However, we finally removed this part from our proposal due to 
> strong concerns from the community that the match protocol is too complex, 
> particularly for an initial version of pattern matching.  Should the need 
> arise, it would still be possible to later add such a customisation protocol 
> in an upcoming version of Python.

It's not easy now to look back over the history of all of this. My
recollection of the original version of PEP 622 (if that's the right
one...) was that it had an overcomplicated protocol for __match__. It
needed to be simplified but in the end it was dropped.

The question now is if it will be straight-forward to retrofit a
protocol to PEP 634 after it is released and when backward
compatibility constraints kick in. PEP 653 (as discussed here) is
precisely an attempt to retrofit a protocol to PEP 634. I think the
difficulties involved in achieving that will become harder rather than
easier in future.

--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QXEA5OQ5P75TS22ZQWYRNMVGT5GEFKPE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Brett Cannon]

On Fri, Feb 19, 2021 at 3:10 PM Stestagg  wrote:

>
>
> On Fri, 19 Feb 2021 at 22:50, Christian Heimes 
> wrote:
>
>> On 19/02/2021 23.22, Stestagg wrote:
>> > The thing that stood out from this conversation, for me, is: Releases
>> > are too hard, and there’s a risk of not having enough volunteers as a
>> > result.
>> >
>> > How hard is it to fix that?
>>
>> Actually it's easy to fix!
>>
>> The PSF needs needs sufficient money to hire a couple of people, so the
>> PSF can turn release management and security maintenance from unpaid
>> volunteers into paid fulltime jobs.
>>
> That’s certainly one option. I’d be super surprised if that were the only
> one.
>
> What were the budget requirements for this? How far from the target is the
> PSF?  Is there a plan to hit it?
>

There is no specific drive to hire someone to target security and/or
release management at the moment. We just got enough funding for the first
time to hire a dev-in-residence for Python itself to try to help tackle our
1.4K PR backlog in some fashion that they won't be dedicated to security or
release management.


>
> Are there no technical solutions that might help reduce the cost?
>

If you would like to help out, you can speak with the release managers to
see if they could use help in some way; same goes for the security team.

-Brett


> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/55GFSMWJOG7P6NWU4KDHMKJN2OGX5YSG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EYP6H3HNLZN4DQ3Y3ROJ63LSMUX3LT7Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Steve Holden]

On Fri, Feb 19, 2021 at 10:53 PM Christian Heimes 
wrote:

> On 19/02/2021 23.22, Stestagg wrote:
> > The thing that stood out from this conversation, for me, is: Releases
> > are too hard, and there’s a risk of not having enough volunteers as a
> > result.
> >
> > How hard is it to fix that?
>
> Actually it's easy to fix!
>
> For some value of "easy"  that involves raising a six-figure dollar sum
annually ...

> The PSF needs needs sufficient money to hire a couple of people, so the
> PSF can turn release management and security maintenance from unpaid
> volunteers into paid fulltime jobs.
>

Oh, is that all? Sustainability of the PSF, as has been shown over its 20
years of existence, is not an easy thing to achieve. It was hit by
the financial crisis in 2008 and again by the coronavirus crisis last year,
both things that affected all foundations.

If you plan to bring this kind of money in and rely on it, you have to
ensure it comes from sources that can't just be switched off when budgets
tighten. It could be done, but "easy" sounds like exaggeration to me.
Unless your suggestions were joking, but I saw no smiley ...

Kind regards,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PUF4BTHGDFRV3P4WJ7EYWB5OTRXCTLR4/
Code of Conduct: http://python.org/psf/codeofconduct/