[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
OK, better late than never... here's a much-delayed review of the PEP. Thank you Irit and Guido for carrying this forward while I've been AWOL! It's fantastic to see my old design sketches turned into something like, actually real. == Overall feelings == Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any way around adding ExceptionGroups in some form, because it's just a fact of life that in a concurrent program, multiple things can go wrong at once, and we want Python to be usable for writing concurrent programs. Right now the state of the art is "exceptions in background threads/tasks get dropped on the floor", and almost anything is better than that. The current PEP is definitely better than that. But at the same time, there are a lot of compromises needed to retrofit this onto Python's existing system, and the current proposal feels like a bunch of awkward hacks with hacks on top. That's largely my fault for not thinking of something better, and maybe there is nothing better. But I still wish we could come up with something more elegant, and I do see why this proposal has made people uncomfortable. For example: - I'm uncomfortable with how in some contexts we treat EG's as placeholders for the contained exceptions, and other places we treat them like a single first-class exceptions. (Witness all the feedback about "why not just catch the ExceptionGroup and handle it by hand?", and imagine writing the docs explaining all the situations where that is or isn't a good idea and the pitfalls involved...) If we could somehow pick one and stick to it then I think that would make it easier for users to grasp. (My gut feeling is that making them pure containers is better, which to me is why it makes sense for them to be @final and why I keep hoping we can figure out some better way for plain 'except' and EGs to interact.) - If a function wants to start using concurrency internally, then now *all* its exceptions have to get wrapped in EGs and callers have to change *all* their exception handling code to use except* or similar. You would think this was an internal implementation detail that the caller shouldn't have to care about, but instead it forces a major change on the function's public API. And this is because regular 'except' can't do anything useful with EGs. - We have a special-case hack to keep 'except Exception' working, but it has tricky edge cases (Exceptions can still sneak past if they're paired up with a BaseException), and it really is specific to 'except Exception'; it doesn't work for any other 'except SomeError' code. This smells funny. Anyway, that's just abstract context to give an idea where I'm coming from. Maybe we just have to accept these trade-offs, but if anyone has any ideas, speak up... == Most important comment == Flat ExceptionGroups: there were two basic design approaches we discussed last year, which I'll call "flat" vs "nested". The current PEP uses the nested design, where ExceptionGroups form a tree, and traceback information is distributed in pieces over this tree. This is the source of a lot of the complexity in the current PEP: for example, it's why EG's don't have one obvious iteration semantics, and it's why once an exception is wrapped in an EG, it can never be unwrapped again (because it would lose traceback information). The idea of the "flat" design is to instead store all the traceback info directly on the leaf exceptions, so the EG itself can be just a pure container holding a list of exceptions, that's it, with no nesting. The downside is that it requires changes to the interpreter's code for updating __traceback__ attributes, which is currently hard-coded to only update one __traceback__ at a time. For a third-party library like Trio, changing the interpreter is obviously impossible, so we never considered it seriously. But in a PEP, changing the interpreter is possible. And now I'm worried that we ruled out a better solution early on for reasons that no longer apply. The more I think about it, the more I suspect that flat EGs would end up being substantially simpler all around? So I think we should at least think through what that would look like (and Irit, I'd love your thoughts here now that you're the expert on the CPython details!), and document an explicit decision one way or another. (Maybe we should do a call or something to go over the details? I'm trying to keep this email from ballooning out of control...) == Smaller points == - In my original proposal, EGs didn't just hold a list of exceptions, but also a list of "origins" for each exception. The idea being that if, say, you attempt to connect to a host with an IPv4 address and an IPv6 address, and they raised two different OSErrors that got bundled together into one EG, then it would be nice to know which OSError came from which attempt. Or in asyncio/trio it would be nice if tracebacks could show which task each exception came from. It seems like this got dropped at
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
On Mon, Apr 5, 2021 at 3:07 AM Nathaniel Smith wrote:
> - Recording pre-empted exceptions: This is another type of metadata that
> would be useful to print along with the traceback. It's non-obvious and a
> bit hard to explain, but multiple trio users have complained about this, so
> I assume it will bite asyncio users too as soon as TaskGroups are added.
> The situation is, you have a parent task P and two child tasks C1 and C2:
>
> P
>/ \
> C1 C2
>
> C1 terminates with an unhandled exception E1, so in order to continue
> unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the
> middle of unwinding another, different exception E2 (so e.g. the
> cancellation arrived during a `finally` block). E2 gets replaced with a
> `Cancelled` exception whose __context__=E2, and that exception unwinds out
> of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards
> it, then re-raises E1 so it can continue unwinding.
>
> The problem here is that E2 gets "lost" -- there's no record of it in
> the final output. Basically E1 replaced it. And that can be bad: for
> example, if the two children are interacting with each other, then E2 might
> be the actual error that broke the program, and E1 is some exception
> complaining that the connection to C2 was lost. If you have two exceptions
> that are triggered from the same underlying event, it's a race which one
> survives.
>
This point reminded me again of this issue in the tracker ("Problems with
recursive automatic exception chaining" from 2013):
https://bugs.python.org/issue18861
I'm not sure if it's exactly the same, but you can see that a couple of the
later comments there talk about "exception trees" and other types of
annotations.
If that issue were addressed after ExceptionGroups were introduced, does
that mean there would then be two types of exception-related trees layered
over each other (e.g. groups of trees, trees of groups, etc)? It makes me
wonder if there's a more general tree structure that could accommodate both
use cases...
--Chris
This is conceptually similar to the way an exception in an 'except' block
> used to cause exceptions to be lost, so we added __context__ to avoid that.
> And just like for __context__, it would be nice if we could attach some
> info to E1 recording that E2 had happened and then got preempted. But I
> don't see how we can reuse __context__ itself for this, because it's a
> somewhat different relationship: __context__ means that an exception
> happened in the handler for another exception, while in this case you might
> have multiple preempted exceptions, and they're associated with particular
> points in the stack trace where the preemption occurred.
>
> This is a complex issue and maybe we should call it out-of-scope for the
> first version of ExceptionGroups. But I mention it because it's a second
> place where adding some extra annotations to the traceback info would be
> useful, and maybe we can keep it simple by adding some minimal hooks in the
> core traceback machinery and let libraries like trio/asyncio handle the
> complicated parts?
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/NIBYXIXQCXJGFMIKNW5HMRTUCHDDCDIB/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Nathaniel, Thank you for your feedback. See a few comment below. On Mon, Apr 5, 2021 at 11:01 AM Nathaniel Smith wrote: > OK, better late than never... here's a much-delayed review of the PEP. > Thank you Irit and Guido for carrying this forward while I've been AWOL! > It's fantastic to see my old design sketches turned into something like, > actually real. > > == Overall feelings == > > Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any > way around adding ExceptionGroups in some form, because it's just a fact of > life that in a concurrent program, multiple things can go wrong at once, > and we want Python to be usable for writing concurrent programs. Right now > the state of the art is "exceptions in background threads/tasks get dropped > on the floor", and almost anything is better than that. The current PEP is > definitely better than that. But at the same time, there are a lot of > compromises needed to retrofit this onto Python's existing system, and the > current proposal feels like a bunch of awkward hacks with hacks on top. > That's largely my fault for not thinking of something better, and maybe > there is nothing better. But I still wish we could come up with something > more elegant, and I do see why this proposal has made people uncomfortable. > People were uncomfortable about three main points: 1. In the first draft "except Exception" would not catch exception groups, this was changed in the second draft. 2. In the first draft exception groups were not subclassable, this also changed in the second draft. 3. Do we really need except*? Why is ExceptionGroup not enough? I think we replied, but this question may still be on some people's minds and if so we should continue discussing it. Was there anything else? > For example: > > - I'm uncomfortable with how in some contexts we treat EG's as > placeholders for the contained exceptions, and other places we treat them > like a single first-class exceptions. (Witness all the feedback about "why > not just catch the ExceptionGroup and handle it by hand?", and imagine > writing the docs explaining all the situations where that is or isn't a > good idea and the pitfalls involved...) If we could somehow pick one and > stick to it then I think that would make it easier for users to grasp. (My > gut feeling is that making them pure containers is better, which to me is > why it makes sense for them to be @final and why I keep hoping we can > figure out some better way for plain 'except' and EGs to interact.) > I'm not sure what you mean by "a placeholder". An exception group is an exception, and except* is a new syntax that helps you manipulate exception groups. I don't think the confusion you mention was due to the fact that except can catch EGs, but rather due to the fact that the simplest examples don't show you why except is not good enough, and why we need except*. Suppose we did as you suggest and made exception group a container which is not exception. Now suppose that an exception is raised in an except* block. What is the context of this exception? Do we change the requirement that an exception's context is always an exception? How do you raise an exception group if it's not an Exception? Do we go back to allowing random objects being raised? > > - If a function wants to start using concurrency internally, then now > *all* its exceptions have to get wrapped in EGs and callers have to change > *all* their exception handling code to use except* or similar. You would > think this was an internal implementation detail that the caller shouldn't > have to care about, but instead it forces a major change on the function's > public API. And this is because regular 'except' can't do anything useful > with EGs. > I don't understand this point. If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others like in the old days when there weren't EGs (which is presumably what your caller is expecting). In other words, if you make drastic changes in a function's implementation, it is your responsibility to ensure that the old API is preserved. > > - We have a special-case hack to keep 'except Exception' working, but it > has tricky edge cases (Exceptions can still sneak past if they're paired up > with a BaseException), and it really is specific to 'except Exception'; it > doesn't work for any other 'except SomeError' code. This smells funny. > I agree with this point, my personal preference would be to have ExceptionGroup(BaseExceptionGroup) as in the first draft, with a runaway exception group being something you should fix. The current choice is a compromise for backwards compatibility, not so much with the language but with the practice of using "except Exception" to "catch almost everything, log it and move on". I don't think we can go against that at this point. > > Anyway, that's just
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Just wanted to elaborate a little bit on StopIteration to add to Irit's reply: On Mon, Apr 5, 2021 at 9:52 AM Irit Katriel via Python-Dev wrote: > On Mon, Apr 5, 2021 at 11:01 AM Nathaniel Smith wrote: >> - There are a number of places where the Python VM itself catches exceptions >> and has hard-coded handling for certain exception types. For example: >> >> - Unhandled exceptions that reach the top of the main thread generally >> cause a traceback to be printed, but if the exception is SystemExit then the >> interpreter instead exits silently with status exc.args[0]. >> >> - 'for' loops call iter.__next__, and catch StopIteration while allowing >> other exceptions to escape. >> >> - Generators catch StopIteration from their bodies and replace it with >> RuntimeError (PEP 479) >> >> With this PEP, it's now possible for the main thread to terminate with >> ExceptionGroup(SystemExit), __next__ to raise ExceptionGroup(StopIteration), >> a generator to raise ExceptionGroup(StopIteration), either alone or mixed >> with other exceptions. How should the VM handle these new cases? Should they >> be using except* or except? >> >> I don't think there's an obvious answer here, and possibly the answer is >> just "don't do that". But I feel like the PEP should say what the language >> semantics are in these cases, one way or another. There's no need to do anything about ExceptionGroups potentially wrapping StopIteration or StopAsyncIteration exceptions. The point of PEP 479 was to solve a problem of one of nested frames raising a StopIteration (often by mistake) and the outer generator being stopped. That lead to some really tricky situations to debug. In our case, a rogue StopIteration wrapped in an EG would not stop a generator silently, it would do that loud and clear. As for SystemExit, we'll change "asyncio.run()" to unpack SystemExit and propagate them unwrapped, potentially allowing to dump the exception tree into a log file for later debug, if configured. Trio should do the same. Problem solved. It's important to understand that the PEP doesn't propose a magical mechanism to turn all exceptions into EGs automatically, it's up to the framework/user code how to build them and what to propagate out. In Python 3.9 you can just as well write `except BaseException: pass` and silence a SystemExit (and people do that from time to time!) Yury ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E5RMN6CSKXSIFXBKRIHYLPK2FRXIM4AE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems
On 4/4/21 7:10 AM, Michał Górny wrote: This is precisely what I meant when I said I don't like the idea of combining security fixes with irrelevant changes. Good that I've chosen to backport the secfixes instead of pushing the new version to Gentoo stable. If I'm a user of Gentoo stable, how would I know from the Python installation itself that those vulnerabilities have been fixed? Would I have to go find the release/update notes to know? -- ~Ethan~ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BO3KGFYNQ2XZZQOFPB6AMILRGLFQDPYA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems
On Mon, 2021-04-05 at 11:17 -0700, Ethan Furman wrote: > On 4/4/21 7:10 AM, Michał Górny wrote: > > > This is precisely what I meant when I said I don't like the idea of > > combining security fixes with irrelevant changes. Good that I've chosen > > to backport the secfixes instead of pushing the new version to Gentoo > > stable. > > If I'm a user of Gentoo stable, how would I know from the Python installation > itself that those vulnerabilities have been fixed? Would I have to go find > the release/update notes to know? I suppose the best way is to look at the security bug: https://bugs.gentoo.org/779841 I'm working on a better tool to check your system for vulnerable packages but I can only dedicate a little time every few days to work on it, so it will take some time before it's ready. -- Best regards, Michał Górny ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WP3HX4IQWJXGIJNWP3UFFBJIARM2NPDG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 652 Accepted -- Maintaining the Stable ABI
Hi Petr, Thank you for submitting PEP 652 (Maintaining the Stable ABI). After evaluating the situation and discussing the PEP, the Steering Council is happy with the PEP and hereby accepts it. The Steering council thinks that this is a great step forward in order to have a clear definition of what goes into the Stable ABI and what guarantees the Python core team offers regarding the stable ABI while offering at the same time a plan to improve the maintenance and stability of the stable ABI. We would also like to see some improvements in the official documentation (not only on the devguide) regarding this topic and what guarantees do we offer (currently we only have a small section about this in https://docs.python.org/3/c-api/stable.html but there is a lot of information and clarifications in the PEP that we would like to be also in the documentation). Congratulations, Petr! With thanks from the whole Python Steering Council, Pablo Galindo Salgado ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IN4XMFLQJ6D6V67EXU27GV3QWSEHHNNH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Mailman 2 - not receiving moderators emails
I recently rebuilt my server (Ubuntu 20.04) and rebuilt mailman 2 - upgrading to the latest version 2.1.34. The mail server is postfix I run three mailing lists, including "TESTmail". The same issue below affects all three lists. On the General Page, I have filled in an administrator and a owner. The list options are set up that emails from non-members are held and the moderator should be notified. * The poster of the email receives a notification that the email is being held * The moderator does not receive any messages * But looking at the postfix logs, it appears to me that mailman does try to send a message to the moderator Apr 5 22:12:05 ip-xxx-xxx-xxx-xxx postfix/smtp[9473]: 6D0973EB91: to=, But, the email message is being sent to the generic owner, rather than the specific names filled into either the moderator or administrator boxes on the general tab. Also, I note that the footer of the mailing list has: TESTmail list run by testmail-owner at lists.XXX.org.uk rather than the specific owner. Any clues as to what is not quite correct in the mailman setup? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YJG4MBWTEUOYXLSPAZLQSZOSTWEIRU5G/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Mailman 2 - not receiving moderators emails
On 4/5/21 4:49 PM, da...@thenicols.net wrote: I recently rebuilt my server (Ubuntu 20.04) and rebuilt mailman 2 - upgrading to the latest version 2.1.34. The mail server is postfix Howdy! This list is for developing the next version of Python itself. For help with mailman you can try mailman-us...@python.org Somebody there should have some ideas for you. Good luck! -- ~Etahn~ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/W5IOGIPWGRDJW53JXBXDR73OCDMZT3LP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems
Matthias Klose writes: > No, you can't see that with CPython's CI alone. The Debian and > Ubuntu build machines trigger CI tests per architecture for around > 3000 packages depending on python3.9, using the just built > python3.9, and without rebuilding these packages. That's where I > see the 32bit failures. Thank you, that's what I wanted to know. > How would delaying the release schedule have helped with the issue > that we just saw? I mean to have a period between the announcement of the release date, and the actual release. So it works the same way an rc would, except not rolling the tarball etc. For almost zero maintainer effort, tag it "rc", give you a few days to run your tests on builds from git. If you (and others) don't report a problem, he tags "final" and then produces tarballs, installers, etc to the extent those things are done at this point in the version's lifecycle. Distros could either drop the "rc" in hopes that this commit will indeed be final (and reroll their release if bugs are found), or they could use the sha1 as identifier for the specific commit instead of stuff like "rc" in versioning the package. (I'm brainstorming, this is *not* a thought-out recommendation!) See also Terry Reedy's post for a different explanation of what I believe is the same idea. Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RHBTR6ICP5LKRXAAX4I4I6UBSMZXB3O3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Paul Moore writes: > It *is* merged and publicly released - it's in the latest 3.10 > alpha. Merged, yes, but in my terminology alphas, betas, and rcs aren't "public releases", they're merely "accessible to the public". (I'm happy to adopt your terminology when you're in the conversation, I'm just explaining what I meant in my previous post.) > The fact that the implementation kept getting referred to as the > "reference implementation" confused me into thinking it hadn't been > released yet, and that simply isn't true. Calling it "the > implementation" avoids that confusion, IMO. The only thing I understand in that paragraph is "that [it hadn't been released yet] simply isn't true", which is true enough on your definition of "released". But why does "reference implementation" connote "unreleased"? That seems to be quite different from Mark's usage. I don't have an objection to your usage, I'd just like us all to converge on a set of terms so that Brandt has a compact way of saying "as far as I know, for the specification under discussion this implementation is completely accurate and folks are welcome to refer to the PEP, to the code, or to divergences as seems appropriate to them". I'm not sure if that's exactly what Brandt meant by "reference implementation", but that's how I understood it. Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JLDWPW52OXGR27W4GFGU4GVDOXA6PXGD/ Code of Conduct: http://python.org/psf/codeofconduct/
