[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Ronald Oussoren via Python-Dev]

> On 19 Jan 2021, at 17:15, Antoine Pitrou  wrote:
> 
> On Tue, 19 Jan 2021 15:54:39 +
> Mark Shannon mailto:m...@hotpy.org>> wrote:
>> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
>>> On Tue, 19 Jan 2021 13:31:45 +
>>> Mark Shannon  wrote:  
 Hi everyone,
 
 It's time for yet another PEP :)
 
 Fortunately, this one is a small one that doesn't change much.
 It's aim is to make the VM more robust.  
>>> 
>>> On the principle, no objection.
>>> 
>>> In practice, can you show how an implementation of Py_CheckStackDepth()
>>> would look like?  
>> 
>> It would depend on the platform, but a portable-ish implementation is here:
>> 
>> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71
> 
> This doesn't tell me how `stack_limit_pointer` is computed or estimated
> :-)

There already is an implementation of this for Windows (``PyOS_CheckStack``). 
For other platforms there will 
have to be a different implementation of this function.  I’ve looked into this 
in the past for macOS, and that platform
has an API to retrieve the size of the stack as well as a pointer to the start 
of the stack (AFAIK stacks aren’t auto-growing
on macOS)

Ronald
—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F62XDAI3NF5YXMQXWEIIIDKY3M6KZ64O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improve CPython tracing performance

[Victor Stinner]

Reply to an old thread.

On Sat, Oct 31, 2020 at 8:02 AM Nick Coghlan  wrote:
> > Debuggers and profilers usually only care of specific frames or
> > function calls (ex: 10% of function calls or even a single function
> > call in a whole application). The problem is how to make them as
> > efficient as possible for "no operation" calls, when they don't care
> > about the current frame. Avoiding PyFrame_FastToLocalsWithError() to
> > enter the debugger/profile and avoiding PyFrame_LocalsToFast() on exit
> > sounds a simple and practical solution.
>
> Aye, I agree. I just don't think we can remove those implicit calls
> without preparing a replacement API first.

Again, I don't think that it's incompatible. We can enforce calling
PyFrame_FastToLocalsWithError() at enter and PyFrame_LocalsToFast() at
exit for now, and enhance the API later.


> Unfortunately, cell variables mean that there's no way to make
> snapshot-with-writeback logic consistently correct in the presence of
> generators and coroutines (...)

I do understand that there are corner cases where it doesn't work, ok.

But https://bugs.python.org/issue42197 is a simple and ready to merge
solution which should optimize profilers and debuggers for the most
common case.

To be clear: currently, PyFrame_FastToLocalsWithError() is called at
enter and PyFrame_LocalsToFast() is called at exit. So asking
debuggers/profilers to call them explicitly doesn't make the situation
worse (nor better ;-)) for generators/coroutines, it would be exactly
the same behavior. It's just an optimization.

The PEP 558 is being discussed for 5 years and still a draft. I don't
think that it should hold bpo-42197 optimization.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MV2723DT7BLHBJPNM6K4WHLTWSRMHZLN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Upcoming 3.7.10 and 3.6.13 Security Releases

[Miro Hrončok]

On 10. 01. 21 21:15, Ned Deily wrote:

We are planning to produce the next security-fix rollup releases for Python 
3.7.x and 3.6.x on 2021-01-15. The most recent releases for these versions were 
on 2020-08-17.  There has not been a lot of activity for either branch since 
then.

Core developers: if you know of any additional security issues that should be addressed 
in these releases, please mark the relevant bpo issues as "release blocker" 
and, if possible, submit PRs for review prior to the end of 2021-01-14 AOE.


Hi Ned. I am not a core developer, but can https://bugs.python.org/issue42938 
please be included? I know it is past the deadline, but I have not seen the 
releases.


Thanks,
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JUJAS2O2DY3TUPITDY7VQYEZXDVVMV7N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Upcoming 3.7.10 and 3.6.13 Security Releases

[Christian Heimes]

On 20/01/2021 13.06, Miro Hrončok wrote:
> On 10. 01. 21 21:15, Ned Deily wrote:
>> We are planning to produce the next security-fix rollup releases for
>> Python 3.7.x and 3.6.x on 2021-01-15. The most recent releases for
>> these versions were on 2020-08-17.  There has not been a lot of
>> activity for either branch since then.
>>
>> Core developers: if you know of any additional security issues that
>> should be addressed in these releases, please mark the relevant bpo
>> issues as "release blocker" and, if possible, submit PRs for review
>> prior to the end of 2021-01-14 AOE.
> 
> Hi Ned. I am not a core developer, but can
> https://bugs.python.org/issue42938 please be included? I know it is past
> the deadline, but I have not seen the releases.

Ned has postponed the upcoming security releases for the issue.
Benjamin's fix has landed two days ago. The fixes will be included in
3.7.10 and 3.6.13.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EMSDC5PS5EU2W77SRJAON4BOHNCD7N76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Upcoming 3.7.10 and 3.6.13 Security Releases

[Miro Hrončok]

On 20. 01. 21 13:43, Christian Heimes wrote:

On 20/01/2021 13.06, Miro Hrončok wrote:

On 10. 01. 21 21:15, Ned Deily wrote:

We are planning to produce the next security-fix rollup releases for
Python 3.7.x and 3.6.x on 2021-01-15. The most recent releases for
these versions were on 2020-08-17.  There has not been a lot of
activity for either branch since then.

Core developers: if you know of any additional security issues that
should be addressed in these releases, please mark the relevant bpo
issues as "release blocker" and, if possible, submit PRs for review
prior to the end of 2021-01-14 AOE.


Hi Ned. I am not a core developer, but can
https://bugs.python.org/issue42938 please be included? I know it is past
the deadline, but I have not seen the releases.


Ned has postponed the upcoming security releases for the issue.
Benjamin's fix has landed two days ago. The fixes will be included in
3.7.10 and 3.6.13.


Thanks!

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T544X74NME4BP2OFROK6UXQGRBPJVSL5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

Hi Pablo,

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:

Hi Mark,

Thanks for gathering this proposal! Looks very interesting. I have some 
preliminary questions: how is this going to affect
the "py-bt" command of the gdb helpers 
(https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
and other similar tools that produce a unified Python-C backtrace? There are 
several debuggers that
rely on the fact that they can merge the C stack and the Python stack by substituting 
every call to "_PyEval_EvalFrameDefault"
and friends for the respective Python frame obtained separately or by 
inspecting the frame object in the stack (like gdb does).


I don't see any problem fixing up the gdb helpers. The code will need to 
be aware that C frames will be one-to-many to Python frames, but there's 
nothing tricky there.




Is this change going to affect these tools or they will continue working as 
before? In case this change will affect this tools,
is there any workaround to produce the unified C/Python call stack given the 
Python stack and the C stack?


It depends on what you mean by "similar tools".
For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it is at 
the moment :)


Cheers,
Mark.



Kind regards,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y3NKLDE4ZL7SQQ6UWPHR73KA5RUY4BQI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Pablo Galindo Salgado]

>It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There are
many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)

> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it is at
> the moment :)

I agree with that, but we need to have in mind that this will break
these tools and therefore we need to be aware on how difficult will
be to overcome the new situation. Is not the same that a small or
medium change is required than the case where is virtually impossible
to overcome. If we were designing the interpreter from scratch that would
not matter, but with an existing ecosystem of tools, it does. Breaking tools
in an established ecosystem has always been a major complaint of users when
we do major releases.

Notice as well that the gdb helpers *are* an out-of-process tool.

Of course, I don't imply that this should be a show stopper or similar, but
it should be taken into consideration along all the other pros and cons.

Regards from cloudy London,
Pablo Galindo Salgado

On Wed, 20 Jan 2021 at 15:12, Mark Shannon  wrote:

> Hi Pablo,
>
> On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> > Hi Mark,
> >
> > Thanks for gathering this proposal! Looks very interesting. I have some
> preliminary questions: how is this going to affect
> > the "py-bt" command of the gdb helpers (
> https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897
> )
> > and other similar tools that produce a unified Python-C backtrace? There
> are several debuggers that
> > rely on the fact that they can merge the C stack and the Python stack by
> substituting every call to "_PyEval_EvalFrameDefault"
> > and friends for the respective Python frame obtained separately or by
> inspecting the frame object in the stack (like gdb does).
>
> I don't see any problem fixing up the gdb helpers. The code will need to
> be aware that C frames will be one-to-many to Python frames, but there's
> nothing tricky there.
>
> >
> > Is this change going to affect these tools or they will continue working
> as before? In case this change will affect this tools,
> > is there any workaround to produce the unified C/Python call stack given
> the Python stack and the C stack?
>
> It depends on what you mean by "similar tools".
> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it is at
> the moment :)
>
> Cheers,
> Mark.
>
> >
> > Kind regards,
> > Pablo Galindo Salgado
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZNCDQCYPATMZRU7SHQM4OTMVXUI6VX5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

Hi Pablo,

On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:

 >It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There 
are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)



For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it is at
the moment :)


I agree with that, but we need to have in mind that this will break
these tools and therefore we need to be aware on how difficult will
be to overcome the new situation. Is not the same that a small or
medium change is required than the case where is virtually impossible
to overcome. If we were designing the interpreter from scratch that would
not matter, but with an existing ecosystem of tools, it does. Breaking tools
in an established ecosystem has always been a major complaint of users 
when we do major releases.


Do you have any specific examples?

I feel the responsibility for this belongs with the tool authors.
If they aren't using the C-API, or requesting an alternative interface 
should the API be insufficient, then they can't really complain.




Notice as well that the gdb helpers *are* an out-of-process tool.


True, but they are in the standard library, so they are our 
responsibility to fix.


Cheers,
Mark.



Of course, I don't imply that this should be a show stopper or similar, 
but it should be taken into consideration along all the other pros and cons.


Regards from cloudy London,
Pablo Galindo Salgado

On Wed, 20 Jan 2021 at 15:12, Mark Shannon > wrote:


Hi Pablo,

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
 > Hi Mark,
 >
 > Thanks for gathering this proposal! Looks very interesting. I
have some preliminary questions: how is this going to affect
 > the "py-bt" command of the gdb helpers

(https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
 > and other similar tools that produce a unified Python-C
backtrace? There are several debuggers that
 > rely on the fact that they can merge the C stack and the Python
stack by substituting every call to "_PyEval_EvalFrameDefault"
 > and friends for the respective Python frame obtained separately
or by inspecting the frame object in the stack (like gdb does).

I don't see any problem fixing up the gdb helpers. The code will
need to
be aware that C frames will be one-to-many to Python frames, but
there's
nothing tricky there.

 >
 > Is this change going to affect these tools or they will continue
working as before? In case this change will affect this tools,
 > is there any workaround to produce the unified C/Python call
stack given the Python stack and the C stack?

It depends on what you mean by "similar tools".
For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it
is at
the moment :)

Cheers,
Mark.

 >
 > Kind regards,
 > Pablo Galindo Salgado
 > ___
 > Python-Dev mailing list -- python-dev@python.org

 > To unsubscribe send an email to python-dev-le...@python.org

 > https://mail.python.org/mailman3/lists/python-dev.python.org/
 > Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
 > Code of Conduct: http://python.org/psf/codeofconduct/
 >


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HHEKGU66A5LZRTQDZWO5627MK5CL37NC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Pablo Galindo Salgado]

> Do you have any specific examples?

>From my previous message: gdb helpers, lldb helpers, TotalView debugger,
py-spy and I know of at least 3 closed source tools.

>I feel the responsibility for this belongs with the tool authors.
> If they aren't using the C-API, or requesting an alternative interface
> should the API be insufficient, then they can't really complain.

The responsibility to fix it is on the authors, yes, but is not that back
and white. In the worst case, what users will see is that tools that worked
now don't and in the worst-worst case users will see that those tools will
not work again because is virtually impossible with the new system. And
that, as I said is not a show stopper, but it matters.

In any case, if we are able to fix the gdb helpers in the stdlib, then I am
not afraid because it means that everyone could fix their own thing.

Cheers,
Pablo Galindo Salgado


On Wed, 20 Jan 2021 at 16:06, Mark Shannon  wrote:

> Hi Pablo,
>
> On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:
> >  >It depends on what you mean by "similar tools".
> >
> > Any 3rd party tool or debugger that is printing merged stacks. (There
> > are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)
> >
> >> For in-process tools, then the API will continue to work.
> >> For out-of-process debuggers, then the author's are on their own.
> >> But that's always been the case.
> >> The source code is public, and it won't be any more opaque than it is at
> >> the moment :)
> >
> > I agree with that, but we need to have in mind that this will break
> > these tools and therefore we need to be aware on how difficult will
> > be to overcome the new situation. Is not the same that a small or
> > medium change is required than the case where is virtually impossible
> > to overcome. If we were designing the interpreter from scratch that would
> > not matter, but with an existing ecosystem of tools, it does. Breaking
> tools
> > in an established ecosystem has always been a major complaint of users
> > when we do major releases.
>
> Do you have any specific examples?
>
> I feel the responsibility for this belongs with the tool authors.
> If they aren't using the C-API, or requesting an alternative interface
> should the API be insufficient, then they can't really complain.
>
> >
> > Notice as well that the gdb helpers *are* an out-of-process tool.
>
> True, but they are in the standard library, so they are our
> responsibility to fix.
>
> Cheers,
> Mark.
>
> >
> > Of course, I don't imply that this should be a show stopper or similar,
> > but it should be taken into consideration along all the other pros and
> cons.
> >
> > Regards from cloudy London,
> > Pablo Galindo Salgado
> >
> > On Wed, 20 Jan 2021 at 15:12, Mark Shannon  > > wrote:
> >
> > Hi Pablo,
> >
> > On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> >  > Hi Mark,
> >  >
> >  > Thanks for gathering this proposal! Looks very interesting. I
> > have some preliminary questions: how is this going to affect
> >  > the "py-bt" command of the gdb helpers
> > (
> https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897
> )
> >  > and other similar tools that produce a unified Python-C
> > backtrace? There are several debuggers that
> >  > rely on the fact that they can merge the C stack and the Python
> > stack by substituting every call to "_PyEval_EvalFrameDefault"
> >  > and friends for the respective Python frame obtained separately
> > or by inspecting the frame object in the stack (like gdb does).
> >
> > I don't see any problem fixing up the gdb helpers. The code will
> > need to
> > be aware that C frames will be one-to-many to Python frames, but
> > there's
> > nothing tricky there.
> >
> >  >
> >  > Is this change going to affect these tools or they will continue
> > working as before? In case this change will affect this tools,
> >  > is there any workaround to produce the unified C/Python call
> > stack given the Python stack and the C stack?
> >
> > It depends on what you mean by "similar tools".
> > For in-process tools, then the API will continue to work.
> > For out-of-process debuggers, then the author's are on their own.
> > But that's always been the case.
> > The source code is public, and it won't be any more opaque than it
> > is at
> > the moment :)
> >
> > Cheers,
> > Mark.
> >
> >  >
> >  > Kind regards,
> >  > Pablo Galindo Salgado
> >  > ___
> >  > Python-Dev mailing list -- python-dev@python.org
> > 
> >  > To unsubscribe send an email to python-dev-le...@python.org
> > 
> >  > https://mail.python.org/mailman3/lists/python-dev.python.org/
> >  > Message archived at
> >
> https://mail.python.org/arch

[Python-Dev] PEP 651: Robust Overflow Handling, version 2

[Mark Shannon]

Hi,

I've updated the PEP in light of my experiments and feedback.
The new API is simpler and a bit more backwards compatible.

https://www.python.org/dev/peps/pep-0651/

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PZWV3ENNDOSJLSE2C7CJNX5ZZVSZFGNT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please explain how to migrate when a function is removed, thanks ;-)

[Kyle Stanley]

Thanks for bringing attention to this, Victor, and to Ken Jin (GH:
Fidget-Spinner) for the PR. I've just completed reviewing and merging the
PR, so hopefully anyone affected will now have a more clear idea of how to
migrate their asyncio code to 3.10. Having the porting method explicitly
documented certainly helps to smooth the version transition process and
reduce headaches. :-)

Going forward, I'll try to make more of an active effort to ensure any
potentially incompatible changes I'm involved with include a clear method
of porting documented in their respective whatsnew. It can be easy to
forget at times that a seemingly minor fix which is intuitively clear to
the authors of a change may not be as clear to those not involved with it,
regardless of how difficult the fix actually is.

On Tue, Jan 19, 2021 at 12:03 PM Victor Stinner  wrote:

> A PR was proposed to document the removal of the loop parameter:
> https://github.com/python/cpython/pull/24256
>
> Victor
>
> On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner 
> wrote:
> >
> > Hi,
> >
> > We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
> > are facing many Python 3.10 incompatible changes. Most changes were
> > planned with a deprecation period, but, as usual, projects are not
> > tested with DeprecationWarning treated as errors, and so tons of
> > packages now fail to build.
> >
> > I'm not here to talk about DeprecationWarning, but how we communicate
> > on incompatible changes made on purpose. For example, What's New in
> > Python 3.8 announces: "In asyncio, the explicit passing of a loop
> > argument has been deprecated and will be removed in version 3.10". As
> > expected, the parameter was removed in Python 3.10 (bpo-42392), but I
> > cannot see anything in What's New in Python 3.10. The problem is that
> > I don't know how to fix projects broken by this change. I'm not
> > complaining about this specific change, but more generally.
> >
> > I strongly suggest to well document incompatible changes. The bare
> > minimum is to mention them in "Porting to Python 3.10" section:
> > https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
> >
> > A link to the bpo sometimes helps to understand how to port code. But
> > I would really appreciate if authors of incompatible changes would
> > explain how to add Python 3.10 support to existing projects, without
> > losing support for older Python versions. Not just "this function is
> > now removed, good luck!" :-)
> >
> > I didn't touch asyncio for at least 1 year, so I don't know what
> > happens if I remove a loop argument. Does an application remain
> > compatible with Python 3.6 without passing loop?
> >
> > I know that I made multiple incompatible changes myself and I'm sure
> > that the documentation should probably be enhanced, but I also expect
> > others to help on that! ;-)
> >
> > Please think about people who have to port 4000 Python projects to
> Python 3.10!
> >
> > ---
> >
> > The best would be to ship a tool which adds Python 3.10 support to
> > existing projects without losing support with Python 3.6. Maybe
> > something like https://github.com/asottile/pyupgrade could be used for
> > that? pyupgrade looks more specific to the Python syntax, than the
> > usage of the stdlib.
> >
> > I wrote such tool to add Python 3.10 support to C extensions without
> > losing support with Python 2.7. It relies on a header file
> > (pythoncapi_compat.h) which provides new C API functions on old Python
> > versions.
> > https://github.com/pythoncapi/pythoncapi_compat
> >
> > For example, it replaces "obj->ob_refcnt = refcnt;" with
> > "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
> > 3.8 and older (function added to Python 3.9)
> >
> > Victor
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CVLDV7VIVENV6HMQ4PRAN3VRFU26CMJI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RGSEPSSPDIVKWYKACONKRUFBEXQ5YKGC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please explain how to migrate when a function is removed, thanks ;-)

[Chris Jerdonek]

I don't know if this is already covered in the discussion and in our
processes, but in addition to documenting the instructions in the release
in which things break, I think it would also be good to include such
instructions in any earlier release in which the thing is merely
deprecated. In those cases, the instructions would be the same, but they
would be on how to make the deprecation warning go away rather than to
unbreak things. That way, people have the information to make their code
forward-compatible earlier, as opposed to waiting for the breaking release.
Personally, I like to address DeprecationWarnings when I first see them, as
opposed to waiting until later.

Is there / would it make sense to have a section analogous to "Porting to
Python X" that covers "Make All DeprecationWarnings Go Away in X"? If we
had such a section, the "Porting to" section could be constructed by
copying the relevant bits from that section in the previous release.

--Chris


On Wed, Jan 20, 2021 at 4:47 PM Kyle Stanley  wrote:

> Thanks for bringing attention to this, Victor, and to Ken Jin (GH:
> Fidget-Spinner) for the PR. I've just completed reviewing and merging the
> PR, so hopefully anyone affected will now have a more clear idea of how to
> migrate their asyncio code to 3.10. Having the porting method explicitly
> documented certainly helps to smooth the version transition process and
> reduce headaches. :-)
>
> Going forward, I'll try to make more of an active effort to ensure any
> potentially incompatible changes I'm involved with include a clear method
> of porting documented in their respective whatsnew. It can be easy to
> forget at times that a seemingly minor fix which is intuitively clear to
> the authors of a change may not be as clear to those not involved with it,
> regardless of how difficult the fix actually is.
>
> On Tue, Jan 19, 2021 at 12:03 PM Victor Stinner 
> wrote:
>
>> A PR was proposed to document the removal of the loop parameter:
>> https://github.com/python/cpython/pull/24256
>>
>> Victor
>>
>> On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner 
>> wrote:
>> >
>> > Hi,
>> >
>> > We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
>> > are facing many Python 3.10 incompatible changes. Most changes were
>> > planned with a deprecation period, but, as usual, projects are not
>> > tested with DeprecationWarning treated as errors, and so tons of
>> > packages now fail to build.
>> >
>> > I'm not here to talk about DeprecationWarning, but how we communicate
>> > on incompatible changes made on purpose. For example, What's New in
>> > Python 3.8 announces: "In asyncio, the explicit passing of a loop
>> > argument has been deprecated and will be removed in version 3.10". As
>> > expected, the parameter was removed in Python 3.10 (bpo-42392), but I
>> > cannot see anything in What's New in Python 3.10. The problem is that
>> > I don't know how to fix projects broken by this change. I'm not
>> > complaining about this specific change, but more generally.
>> >
>> > I strongly suggest to well document incompatible changes. The bare
>> > minimum is to mention them in "Porting to Python 3.10" section:
>> > https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
>> >
>> > A link to the bpo sometimes helps to understand how to port code. But
>> > I would really appreciate if authors of incompatible changes would
>> > explain how to add Python 3.10 support to existing projects, without
>> > losing support for older Python versions. Not just "this function is
>> > now removed, good luck!" :-)
>> >
>> > I didn't touch asyncio for at least 1 year, so I don't know what
>> > happens if I remove a loop argument. Does an application remain
>> > compatible with Python 3.6 without passing loop?
>> >
>> > I know that I made multiple incompatible changes myself and I'm sure
>> > that the documentation should probably be enhanced, but I also expect
>> > others to help on that! ;-)
>> >
>> > Please think about people who have to port 4000 Python projects to
>> Python 3.10!
>> >
>> > ---
>> >
>> > The best would be to ship a tool which adds Python 3.10 support to
>> > existing projects without losing support with Python 3.6. Maybe
>> > something like https://github.com/asottile/pyupgrade could be used for
>> > that? pyupgrade looks more specific to the Python syntax, than the
>> > usage of the stdlib.
>> >
>> > I wrote such tool to add Python 3.10 support to C extensions without
>> > losing support with Python 2.7. It relies on a header file
>> > (pythoncapi_compat.h) which provides new C API functions on old Python
>> > versions.
>> > https://github.com/pythoncapi/pythoncapi_compat
>> >
>> > For example, it replaces "obj->ob_refcnt = refcnt;" with
>> > "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
>> > 3.8 and older (function added to Python 3.9)
>> >
>> > Victor
>> > --
>> > Night gathers, and now my watch begins. It shall not end until my death.
>>
>>
>>
>> --
>> Ni

[Python-Dev] Re: Upcoming 3.7.10 and 3.6.13 Security Releases

[Ned Deily]

On Jan 20, 2021, at 08:17, Miro Hrončok  wrote:
> On 20. 01. 21 13:43, Christian Heimes wrote:
>> On 20/01/2021 13.06, Miro Hrončok wrote:
>>> On 10. 01. 21 21:15, Ned Deily wrote:
 We are planning to produce the next security-fix rollup releases for
 Python 3.7.x and 3.6.x on 2021-01-15. The most recent releases for
 these versions were on 2020-08-17.  There has not been a lot of
 activity for either branch since then.
 
 Core developers: if you know of any additional security issues that
 should be addressed in these releases, please mark the relevant bpo
 issues as "release blocker" and, if possible, submit PRs for review
 prior to the end of 2021-01-14 AOE.
>>> 
>>> Hi Ned. I am not a core developer, but can
>>> https://bugs.python.org/issue42938 please be included? I know it is past
>>> the deadline, but I have not seen the releases.
>> Ned has postponed the upcoming security releases for the issue.
>> Benjamin's fix has landed two days ago. The fixes will be included in
>> 3.7.10 and 3.6.13.

Thanks, Christian, that's correct.

In the meantime, another potential security issue has arisen that might impact 
3.7 and 3.6 so I'm going to continue to hold off on the releases until we have 
a resolution of that.

https://bugs.python.org/issue42967

--
  Ned Deily
  n...@python.org -- []
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D47G5BO6B6SRPZ3QPTP6ZCUZRCQSRPBC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Upcoming 3.7.10 and 3.6.13 Security Releases

[Senthil Kumaran]

On Wed, Jan 20, 2021 at 7:22 PM Ned Deily  wrote:

>
>
> In the meantime, another potential security issue has arisen that might
> impact 3.7 and 3.6 so I'm going to continue to hold off on the releases
> until we have a resolution of that.
>
> https://bugs.python.org/issue42967
>
>
And another security issue was brought up to our attention here (today):

https://bugs.python.org/issue42987
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GICSB5JVN5EJ3CUOCUSZV5JPXY7P3N7Z/
Code of Conduct: http://python.org/psf/codeofconduct/