[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

In earlier versions of the PEP ExceptionGroup was not immutable and split
actually removed matching exceptions from it.
It was also iterable so you could get a flat list of all the contained leaf
exceptions. Then we changed it.

ExceptionGroup is a tree of exceptions, and the internal nodes of the tree
(which are ExceptionGroups) have metadata on them -  context, cause,
traceback.
If you want the full traceback of a leaf exception you need to concatenate
the tracebacks of all the exceptions on the path from the root to the leaf.
If you flatten an ExceptionGroup and create a new list
ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has
depth 1).
Similarly, if you add an exception it needs to have matching metadata to
the ExceptionGroup you are adding it to. It's too easy to get that wrong.

split() and subgroup() take care to preserve the correct metadata on all
the internal nodes, and if you just use them you only make safe operations.
This is why I am hesitating to add iteration utilities to the API. Like we
did, people will naturally try that first, and it's not the safest API.

We actually have the  OSErrors example in the PEP, just above
https://www.python.org/dev/peps/pep-0654/#caught-exception-objects:

try:
low_level_os_operation()
except *OSerror as errors:
raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None



On Sun, Feb 28, 2021 at 6:30 AM Guido van Rossum  wrote:

> We really don’t want users pushing non-exceptions into the list, nor do we
> want e.g. KeyboardInterrupt to be added to a (non-Base-) ExceptionGroup.
>
> There’s a pattern for what you propose using the split() method and a
> lambda, or you can keep the exceptions in a list and re-wrap them at the
> end.
>
> On Sat, Feb 27, 2021 at 20:41 Cameron Simpson  wrote:
>
>> On 26Feb2021 02:44, Irit Katriel  wrote:
>> >On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum 
>> wrote:
>> >> OT: Is ExceptionGroup *really* immutable in the current
>> >> implementation? As
>> >> long as the 'errors' field is a list, I think one could mutate the list
>> >> directly.
>> >
>> >It's not, but we were going to make it an immutable tuple.
>>
>> Could you say why? Other than wanting to discourage mutation happy code
>> getting out there?
>>
>> The reason I ask is that the scenario which comes to my mind is
>> something like:
>>
>> except *OSError as e:
>>
>> AIUI "e" is an ExceptionGroup containing only OSErrors. So, one common
>> thing in my own code is this:
>>
>> try:
>> do something with a file
>> except OSError as e:
>> if e.errno == ENOENT:
>> # file is missing, but that is ok
>> # because we treat it like an empty file
>> elif ... some other ok situation ...
>> else:
>> raise
>>
>> My natural inclination with an ExceptionGroup would be to winnow the
>> OSErrors I'm handed, and push the _unhandled_ errors back into the
>> original ExceptionGroup. That way, after the various except* clauses, a
>> nonempty ExceptionGroup would remain with the unhandled errors, and it
>> might perhaps be reraised then.
>>
>> Cheers,
>> Cameron Simpson 
>>
> --
> --Guido (mobile)
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JAVBJENJCBIMC6EVUVZFJJYYPGKRC3R2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Jim J. Jewett]

> And unlike a venv, "python -m" doesn't let you ensure
> that the code executed is the version installed in user site-packages

I have had enough problems with this that when I do modify/replace something, I 
put in a marker that I can check for explicitly.  Expecting this marker to run 
automatically at import (and the results not to be swallowed) turned out to be 
insufficient.

> it could be coming from a directory earlier in sys.path.

The obvious solution is to put the current directory in front of sys.path.
Alas, security folks didn't like that idea.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M3U72J6OR53LF5REA7LOZBIQUUXXHV65/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Sun, Feb 28, 2021 at 2:40 AM Irit Katriel 
wrote:

>
> In earlier versions of the PEP ExceptionGroup was not immutable and split
> actually removed matching exceptions from it.
> It was also iterable so you could get a flat list of all the contained
> leaf exceptions. Then we changed it.
>
> ExceptionGroup is a tree of exceptions, and the internal nodes of the tree
> (which are ExceptionGroups) have metadata on them -  context, cause,
> traceback.
> If you want the full traceback of a leaf exception you need to concatenate
> the tracebacks of all the exceptions on the path from the root to the leaf.
> If you flatten an ExceptionGroup and create a new list
> ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has
> depth 1).
>

Is this a typo? Did you mean "If you flatten [it] and create a new list...
you *do* lose metadata (unless ... depth 1)"?


> Similarly, if you add an exception it needs to have matching metadata to
> the ExceptionGroup you are adding it to. It's too easy to get that wrong.
>
> split() and subgroup() take care to preserve the correct metadata on all
> the internal nodes, and if you just use them you only make safe operations.
> This is why I am hesitating to add iteration utilities to the API. Like we
> did, people will naturally try that first, and it's not the safest API.
>
> We actually have the  OSErrors example in the PEP, just above
> https://www.python.org/dev/peps/pep-0654/#caught-exception-objects:
>
> try:
> low_level_os_operation()
> except *OSerror as errors:
> raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None
>
>
> --
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JCRKPDWQMYPSQJUXBZDGSVDI2T67ZSC3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Sun, Feb 28, 2021 at 6:17 PM Guido van Rossum  wrote:

> On Sun, Feb 28, 2021 at 2:40 AM Irit Katriel 
> wrote:
>
>>
>> In earlier versions of the PEP ExceptionGroup was not immutable and split
>> actually removed matching exceptions from it.
>> It was also iterable so you could get a flat list of all the contained
>> leaf exceptions. Then we changed it.
>>
>> ExceptionGroup is a tree of exceptions, and the internal nodes of the
>> tree (which are ExceptionGroups) have metadata on them -  context, cause,
>> traceback.
>> If you want the full traceback of a leaf exception you need to
>> concatenate the tracebacks of all the exceptions on the path from the root
>> to the leaf.
>> If you flatten an ExceptionGroup and create a new list
>> ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has
>> depth 1).
>>
>
> Is this a typo? Did you mean "If you flatten [it] and create a new list...
> you *do* lose metadata (unless ... depth 1)"?
>

It is a typo, thanks.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M3DF5VIJWVE73SU4576ICKOGLEEWQBNO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

Looking at the following PEP example, I'm still not sure what he should do to 
handle some but not all OSError instances:

...raise ExceptionGroup(
... "eg",
... [
... ValueError(1),
... TypeError(2),
... OSError(3),
... ExceptionGroup(
... "nested",
... [OSError(4), TypeError(5), ValueError(6)])
... ]

except *OSError seems to get him an ExceptionGroup that he still has to 
manually tree-walk to handle the OSErrors that he can actually handle.

Once he has manually tree-walked, he has to manually re-raise the other 
OSErrors that he hasn't filtered out, and they become a sibling of the original 
ExceptionGroup, because they're (now) being raised at a different location.  
(Or, not *really* the original ExceptionGroup, but one that looks like it 
without any OSErrors.)  This new sibling relationship is there to show that a 
failed or incomplete attempt was made at resolving the OSErrors, but no such 
attempt was made at the ValueErrors or TypeErrors.

Alternatively, he *might* be able to just grab Exception, then use a 
complicated callback to subgroup only the OSErrors that he will be able to 
handle, and raise the complement of that.  It will still add another layer of 
ExceptionGroup, but won't split the remainders.

This seems like an awful lot of work to preserve and elaborate structural 
relations between different exceptions that were originally accidental.  (I am 
assuming that the original relation was "these all happened, probably in 
different execution threads" and the new relations are just accidents of what 
got partially handled along the way, or maybe the order in which different 
execution threads got merged.)

I can't imagine ever needing to care about the full tree structure unless I'm 
re-implementing asycio/trio/curio.  Even if that structural information is 
important to my own framework, that just means I should create more 
intermediate tasks to handle it more quickly, before the next level of 
aggregation.

I would much prefer an API that let me pattern-match (even just by class of the 
exception instances, though preferably also by attributes), process each 
exception (as though it were the only one) within the code block that match 
selects, and indicate whether that one exception should be raised further or 
not.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WTZUGBFWFXMGAFHNTTBMUYFGMQCPV5N6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

You can pass an arbitrary function to eg.split() and eg.subgroup(), and
that function can have a side effect. Suppose you want to log and ignore
OSError with errno==ENOENT but re-raise the rest, you can do this:
```
def log_and_ignore(err):
if err.errno == ENOENT:
log(err)
return False
else:
return True

try:
. . .
except *OSError as eg:
eg = eg.subgroup(log_and_ignore)
if eg is not None:
raise eg
```


On Sun, Feb 28, 2021 at 10:35 AM Jim J. Jewett  wrote:

> Looking at the following PEP example, I'm still not sure what he should do
> to handle some but not all OSError instances:
>
> ...raise ExceptionGroup(
> ... "eg",
> ... [
> ... ValueError(1),
> ... TypeError(2),
> ... OSError(3),
> ... ExceptionGroup(
> ... "nested",
> ... [OSError(4), TypeError(5), ValueError(6)])
> ... ]
>
> except *OSError seems to get him an ExceptionGroup that he still has to
> manually tree-walk to handle the OSErrors that he can actually handle.
>
> Once he has manually tree-walked, he has to manually re-raise the other
> OSErrors that he hasn't filtered out, and they become a sibling of the
> original ExceptionGroup, because they're (now) being raised at a different
> location.  (Or, not *really* the original ExceptionGroup, but one that
> looks like it without any OSErrors.)  This new sibling relationship is
> there to show that a failed or incomplete attempt was made at resolving the
> OSErrors, but no such attempt was made at the ValueErrors or TypeErrors.
>
> Alternatively, he *might* be able to just grab Exception, then use a
> complicated callback to subgroup only the OSErrors that he will be able to
> handle, and raise the complement of that.  It will still add another layer
> of ExceptionGroup, but won't split the remainders.
>
> This seems like an awful lot of work to preserve and elaborate structural
> relations between different exceptions that were originally accidental.  (I
> am assuming that the original relation was "these all happened, probably in
> different execution threads" and the new relations are just accidents of
> what got partially handled along the way, or maybe the order in which
> different execution threads got merged.)
>
> I can't imagine ever needing to care about the full tree structure unless
> I'm re-implementing asycio/trio/curio.  Even if that structural information
> is important to my own framework, that just means I should create more
> intermediate tasks to handle it more quickly, before the next level of
> aggregation.
>
> I would much prefer an API that let me pattern-match (even just by class
> of the exception instances, though preferably also by attributes), process
> each exception (as though it were the only one) within the code block that
> match selects, and indicate whether that one exception should be raised
> further or not.
>
> -jJ
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WTZUGBFWFXMGAFHNTTBMUYFGMQCPV5N6/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BJYWU6VZNYTTV43T5ZL5CFXLXEOCBDEQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Oscar Benjamin]

On Sun, 28 Feb 2021 at 07:04, Stephen J. Turnbull
 wrote:
>
> Jim J. Jewett writes:
>
>  > > which file am I actually running?
>  > > which interpreter am I actually running?
>  > > how do I tell the computer to use a different interpreter?
>  >
>  > If you need to care about any of these, then the environment is
>  > fighting you -- and the application probably stinks.  Programmers
>  > have to deal with it because of bootstrapping, but there is no
>  > reason that we should assume all python users need that flexibility
>  > or want that responsibility.
>

>
> So I agree with the basic principle that programs should Just Work by
> default, but it's not always possible.  Eg, there's good reason why
> Downloads and Documents both exist.  So it helps to have users know
> something about implementations and sysadminning stuff, especially
> when remote troubleshooting.
>
> The question for Python is given the history and the plethora of ways
> to invoke Python and find packages and programs, can we do better with
> a minimum of backward incompatibility?  Humans are generally better at
> learning this stuff than bags of bits on spinning platters are!

I think that the py launcher is actually a good solution to many of
these problems. It can also be improved but the main reason it doesn't
help right now is just the fact that it's only for Windows and also
isn't always installed on Windows. When the launcher is installed
though then:

- It is on PATH
- It can list the Python installations with "py -0p"
- It can be used to select a particular Python version/installation to
use from the command line e.g. "py -3.8 myscript.py"
- It is possible to configure a default version (although I think you
have to do it with an environment variable)

There are some ways that the launcher situation could be improved though:

- I think that the launcher is only installed in an all users install.
There would need to be a way to do it in user-space as well (even if
it means the user has to configure PATH).
- Listing installations with "py -0p" is somewhat cryptic
- It would be better if you could use the launcher itself to set the
default Python e.g. "py --set-default-python=3.8"
- The launcher isn't installed by Anaconda

On the last point I think that although Anaconda doesn't install the
launcher you can use the launcher to run the python from the Anaconda
installation. Ideally Anaconda would install the launcher but it could
also be possible to install it separately.

If py was provided on OSX, Linux etc along with the very clear
guidance that it is supposed to be for "users" rather than the
"system" then maybe that could be a better situation. It would be
backwards compatible since the launcher already exists on Windows and
"py" hasn't been used for anything on non-Windows platforms (AFAIK).


--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LZGV4DDFZKA4NYQVR6JVNG4HVBXDRZFZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

Is there a tool that (1) detects import name collisions; and (2) attempts
to read package metadata and package file checksums (maybe from the ZIP
'manifest')?

In order to:

- troubleshoot module shadowing issues
  - $PATH
  - sys.path
- `python -m site`
  - incomplete and overlapping uninstallations:

pip install a
pip install a_modified # pip uninstall a?
pip install pdbpp
pip uninstall a_modified
ls -altr "${site-packages[*]}"
strace -e trace=file python -c 'import pdb'



When shouldn't site customizations be added to the site module?
https://docs.python.org/3/library/site.html

When should customizations be built into the build instead of a runtime
conditional?


On Sat, Feb 27, 2021, 23:12 Nick Coghlan  wrote:

> On Wed, 24 Feb 2021 at 10:49, Random832  wrote:
> >
> > I was reading a discussion thread <
> https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about
> various issues with the Debian packaged version of Python, and the
> following statement stood out for me as shocking:
> >
> > Christian Heimes wrote:
> > > Core dev and PyPA has spent a lot of effort in promoting venv because
> we don't want users to break their operating system with sudo pip install.
> >
> > I don't think sudo pip install should break the operating system. And I
> think if it does, that problem should be solved rather than merely advising
> users against using it. And why is it, anyway, that distributions whose
> package managers can't coexist with pip-installed packages don't ever seem
> to get the same amount of flak for "damaging python's brand" as Debian is
> getting from some of the people in the discussion thread? Why is it that
> this community is resigned to recommending a workaround when distributions
> decide the site-packages directory belongs to their package manager rather
> than pip, instead of bringing the same amount of fiery condemnation of that
> practice as we apparently have for *checks notes* splitting parts of the
> stdlib into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
>
> The reason venv is promoted as heavily as it is is because it's the
> only advice that can be given that is consistently correct regardless
> of the operating system the user is running locally, whereas safely
> using a system-wide Python installation varies a lot depending on
> whether you're on Windows, Mac OS X, or Linux (let alone some other
> platform outside the big 3 desktop clients).
>
> conda is also popular for the same reason: while the instructions for
> installing conda in the first place are OS-dependent, once it is up
> and running you can use consistent platform independent conda commands
> rather than having to caveat all your documentation with
> platform-specific instructions.
>
> Apple moved all of their dynamic language interpreter implementations
> to inaccessible-by-default locations so Mac OS X users would stop
> using them to run their own code.
>
> Alongside that, we *have* worked with the Linux distro vendors to help
> make "sudo pip install" safe (e.g [1]), but that only helps if a user
> is running a new enough version of a distro that has participated in
> that work.
>
> However, while the option of running "platform native" environments
> will never go away, and work will continue to make it less error
> prone, the level of knowledge of your specific OS's idiosyncrasies
> that it requires is almost certainly going to remain too high for it
> to ever again become the default recommendation that it used to be.
>
> Cheers,
> Nick.
>
> [1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note:
> this change mitigated some aspects of the problem in a way similar to
> what Debian does, but still doesn't solve it completely, as custom
> Python builds may still make arbitrary changes)
>
> P.S. "But what about user site-packages?" you ask. Until relatively
> recently, Debian didn't put the user's local bin directory on the
> system path by default, so commands provided by user level package
> installs didn't work without the user adjusting their PATH. The
> CPython Windows installer also doesn't adjust PATH by default (for
> good reasons). And unlike a venv, "python -m" doesn't let you ensure
> that the code executed is the version installed in user site-packages
> - it could be coming from a directory earlier in sys.path.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RDLEH6DUF57UB6U4HNL2QRVAJY4KDSSJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an e

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Cameron Simpson]

This message is longer than I had anticipated.

To aid comprehension, I'm:

- accepting that .split and .subgroup help my "handle some excpetions 
  but not others" situation, barely

- arguing for ExceptionGroups acting like other containers: truthy if 
  nonempty, falsey if empty; iterable; .subgroup and .split _not_ 
  returning None for an empty subgroup, so that the container-like 
  aspects can be used directly

On 28Feb2021 10:40, Irit Katriel  wrote:
>split() and subgroup() take care to preserve the correct metadata on 
>all
>the internal nodes, and if you just use them you only make safe operations.
>This is why I am hesitating to add iteration utilities to the API. Like we
>did, people will naturally try that first, and it's not the safest API.

Wouldn't it be safe if the ExceptionGroup were immutable, as you plan?  
Or have you another hazard in mind?

>We actually have the  OSErrors example in the PEP, just above
>https://www.python.org/dev/peps/pep-0654/#caught-exception-objects:
>
>try:
>low_level_os_operation()
>except *OSerror as errors:
>raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None

Indeed. That basicly addresses my pattern. Along with:

>On Sun, Feb 28, 2021 at 6:30 AM Guido van Rossum  
>wrote:
>> There’s a pattern for what you propose using the split() method and a
>> lambda, or you can keep the exceptions in a list and re-wrap them at the
>> end.

The keep-a-list approach was my fallback, absent a way to push an 
unhandled exception back in some sense.

Guido:
>> We really don’t want users pushing non-exceptions into the list, nor 
>> do we
>> want e.g. KeyboardInterrupt to be added to a (non-Base-) ExceptionGroup.

I was only imagining pushing exceptions from the original group back in.  
Enforcing that would be tedious and confusing though, so I was imagining 
some way of marking specific subexeceptions as handled or not handled.

But I had not understood the subgroup method.

I think my handled/unhandled concerns are (barely) sufficient addressed 
above.  If I wanted to sweep the group for handled exceptions and then 
reraise the unhandled ones in their own ExceptionGroup at the end, that 
seems tractable.

But all that said, being able to iterable the subexceptions seems a 
natural way to manage that:

unhandled = []
try:
.
except *OSError as eg:
for e in eg:
if an ok exception:
handle it
else:
unhandled.append(e)
if unhandled:
raise ExceptionGroup("unhandled", unhandled)

There are some immediate shortcomings above. In particular, I have no 
way of referencing the original ExceptionGroup without surprisingly 
cumbersome:

try:
.
except ExceptionGroup as eg0:
unhandled = []
eg, os_eg = eg0.split(OSError)
if os_eg:
for e in os_eg:
if an ok exception:
handle it
else:
unhandled.append(e)
if eg:
eg, value_eg = eg.split(ValueError)
if value_eg:
for e in value_eg:
if some_ValueError_we_understand:
handle it
else:
unhandled.append(e)
if eg:
unhandled.append(eg)
if unhandled:
raise ExceptionGroup("unhandled", unhandled) from eg0

I have the following concerns with the pattern above:

There's no way to make a _new_ ExceptionGroup with the same __cause__ 
and __context__ and message as the original group: not that I can't 
assign to these, but I'd need to enuerate them; what if the exceptions 
grew new attributes I wanted to replicate?

This cries out for another factory method like .subgroup but which makes 
a new ExceptionGroup from an original group, containing a new sequence 
of exceptions but the same message and coontext. Example:

unhandled_eg = eg0.with_exceptions(unhandled)

I don't see a documented way to access the group's message.

I'm quite unhappy about .subgroup (and presumably .split) returning None 
when the group is empty. The requires me to have the gratuitous "if eg:" 
and "if value_eg:" if-statements in the example above.

If, instead, ExceptionGroups were like any other container I could just 
test if they were empty:

if eg:

_and_ even if they were empty, iterate over them. Who cares if the loop 
iterates zero times? The example code would become:

try:
.
except ExceptionGroup as eg0:
unhandled = []
eg, os_eg = eg0.split(OSError)
for e in os_eg:
if an ok exception:
handle it
else:
unhandled.append(e)
eg, value_eg = eg.split(ValueError)
for e in value_eg:
if some_ValueError_we_understand:
handle it
else:
unhandled.append(e)
if eg:
unhan

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Cameron Simpson]

On 27Feb2021 00:54, Irit Katriel  wrote:
>On Sat, Feb 27, 2021 at 12:47 AM Jim J. Jewett  wrote:
>> > Is this not allowed?
>>
>> >try:
>> >try:
>> >obj.func()# function that raises ExceptionGroups
>> >except AttributeError:
>> >logger.info("obj doesn't have a func")
>> >except *(AttributeError, SyntaxError):
>> >logger.info("func had some problems")
>>
>> Allowed, but probably in error ... no AttributeError will get through to
>> the except * unless it happened inside the except AttributeError handler.
>> Did you mean:
>
>If obj.func() raises an ExceptionGroup that contains AttributeError then
>"except AttributeError" doesn't catch it. So it will get through.

And I, for one, would expect that. And _want_ that: I want the code to 
do what I said, not have some magic which silently/invisibly intercepts 
ExceptionGroups which contain something buried deep in their subgroup 
tree.

We already allow "deep" exceptions out, to be caught at an arbitrary 
outer call stack level. I don't see why ExceptionGroups should be any 
different.

I certainly do not want ExceptionGroup([AttributeError]) conflated with 
AttributeError. That fills me with horror.

Cheers,
Cameron Simpson 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CSJJSEEW2EY56JKMBCSD6R5SEJOF2ZVU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Marco Sulla]

On Sat, 27 Feb 2021 at 00:35, Guido van Rossum  wrote:
> On Fri, Feb 26, 2021 at 3:18 PM Marco Sulla  
> wrote:
>>
>> Excuse me if I post here. Maybe is a stupid question: why, instead of
>> introducing except*, Python can't extend the functionality of except,
>> so it can do what except* would do?
>
>
> Good question. Here's an example:
> ```
> try:
> . . .
> except OSError as err:
> if err.errno != ENOENT:
> raise
> . . .
> ```
> If this would catch ExceptionGroup(OSError), the `err` variable would be an 
> ExceptionGroup instance, which does not have an `errno` attribute.

Thank you for the clarification :)

I must admit I read the PEP quickly, so I thought that the
subexception will be raised by except*, not the exception group. But
obviously this can't work.

The fact is I am really used to think that

except OsError as e

means "except for any OsError, which we name `e` from this moment on"

that I thought it applied also for except*. But, if I understood well

except* OsError as eg

means "except for any OsError contained in an ExceptionGroup. If the
OsError is not contained in an ExceptionGroup, Python wraps it in a
new ExceptionGroup. The ExceptionGroup is named `eg` from this moment
on"

I have to say that it's not obvious for me reading except* this way.
Anyway, I can't find a more readable semantic that does not use a new
keyword and it's short ^^'
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XZRM57YQU3NRU5JLAHDULVR7TKUF5FT3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi Cameron,

If you go long, I go longer :)


On Sun, Feb 28, 2021 at 10:51 PM Cameron Simpson  wrote:

> On 28Feb2021 10:40, Irit Katriel  wrote:
> >split() and subgroup() take care to preserve the correct metadata on
> >all
> >the internal nodes, and if you just use them you only make safe
> operations.
> >This is why I am hesitating to add iteration utilities to the API. Like we
> >did, people will naturally try that first, and it's not the safest API.
>
> Wouldn't it be safe if the ExceptionGroup were immutable, as you plan?
> Or have you another hazard in mind?
>

Making them immutable won't help the metadata issue. split() and subgroup()
copy the (context, cause traceback) from the original ExceptionGroups (root
and internal nodes of the tree) to the result trees.   If you DIY creating
new
ExceptionGroups you need to take care of that.



> But all that said, being able to iterable the subexceptions seems a
> natural way to manage that:
>
> unhandled = []
> try:
> .
> except *OSError as eg:
> for e in eg:
> if an ok exception:
> handle it
> else:
> unhandled.append(e)
> if unhandled:
> raise ExceptionGroup("unhandled", unhandled)
>


You just lost the metadata of eg. It has no context, cause and its
traceback begins here. And the exceptions contained in it, if they came
from a deeper tree that you flattened into the list, now look like their
traceback jumps straight to here from the place they were actually first
inserted into an ExceptionGroup. This may well be an impossible code path.
Here's an example:

>>> import traceback
>>>
>>> def flatten(exc):
... if isinstance(exc, ExceptionGroup):
... for e in exc.errors:
...yield from flatten(e)
... else:
... yield exc
...
>>> def f():
... try:
... raise ValueError(42)
... except ValueError as e:
... return e
...
>>> def g():
... try:
... raise ExceptionGroup("g", [f()])
... except ExceptionGroup as e:
... return e
...
>>> def h():
... try:
... raise ExceptionGroup("h", [g()])
... except ExceptionGroup as e:
... return e
...
>>> def flat_h():
... try:
... raise ExceptionGroup("flat_h", list(flatten(h(
... except ExceptionGroup as e:
... return e
...
>>>
>>> traceback.print_exception(h())
Traceback (most recent call last):
  File "", line 3, in h
ExceptionGroup: h
   
   Traceback (most recent call last):
 File "", line 3, in g
   ExceptionGroup: g
  
  Traceback (most recent call last):
File "", line 3, in f
  ValueError: 42

>>> traceback.print_exception(flat_h())
Traceback (most recent call last):
  File "", line 3, in flat_h
ExceptionGroup: flat_h
   
   Traceback (most recent call last):
 File "", line 3, in f
   ValueError: 42
>>>


traceback.print_exception(h()) prints a reasonable traceback - h() called
g() called f().

But according to  traceback.print_exception(flat_h()),   flat_h() called
f().


You can preserve the metadata (and the nested structure with all its
metadata) if you replace the last line with:
raise eg.subgroup(lambda e: e in  unhandled)
And for the part before that, iteration, Guido's pattern showed that you
can roll it into the subgroup callback.



>
> There are some immediate shortcomings above. In particular, I have no
> way of referencing the original ExceptionGroup without surprisingly
> cumbersome:


> try:
> .
> except ExceptionGroup as eg0:
> unhandled = []
> eg, os_eg = eg0.split(OSError)
> if os_eg:
> for e in os_eg:
> if an ok exception:
> handle it
> else:
> unhandled.append(e)
> if eg:
> eg, value_eg = eg.split(ValueError)
> if value_eg:
> for e in value_eg:
> if some_ValueError_we_understand:
> handle it
> else:
> unhandled.append(e)
> if eg:
> unhandled.append(eg)
> if unhandled:
> raise ExceptionGroup("unhandled", unhandled) from eg0
>


This is where except* can help:

  try:
.
except except *OSError as eg:
unhandled = []
handled, unhandled = eg.split(lambda e: e is an ok exception)  #
with side effect to handle e
if unhandled:
raise unhandled
except *ValueError as eg:
handled, unhandled = eg.split(lambda e: e is a value error we
understand)  # with side effect to handle e
if  unhandled:


I have the following concerns with the pattern above:
>
> There's no way to make a _new_ ExceptionGroup with 

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Eryk Sun]

On 2/28/21, Oscar Benjamin  wrote:
>
> - It is possible to configure a default version (although I think you
> have to do it with an environment variable)

The py launcher in Windows supports a "py.ini" file beside the
executable and in %LocalAppData%. The equivalent of the PY_PYTHON,
PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the
"[defaults]" section as "python", "python2", and "python3" settings.

The ini file also supports a "[commands]" section to define additional
virtual commands for shebangs. Regular filepaths are also supported in
shebangs -- e.g. #!"path\to\any\program.exe".

> - I think that the launcher is only installed in an all users install.

It defaults to an all-users install, but it can also be installed for
just the current user in "%LocalAppData%\Programs\Python\Launcher". In
this case, the installation directory always gets added to PATH.

> - Listing installations with "py -0p" is somewhat cryptic

There's also the long-form options `--list` and `--list-paths`.

> - It would be better if you could use the launcher itself to set the
> default Python e.g. "py --set-default-python=3.8"

It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value
in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe
naively to set PATH.)

> On the last point I think that although Anaconda doesn't install the
> launcher you can use the launcher to run the python from the Anaconda
> installation.

I don't use Anaconda, but I don't think that's supposed to be the case
according to PEP 514. The launcher only looks for PSF development
distributions in the "PythonCore" registry key.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JPQLBEYVWV2XGXO44JJ4YPN3KFJJYN2Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Oscar Benjamin]

On Mon, 1 Mar 2021 at 00:13, Eryk Sun  wrote:
>
> On 2/28/21, Oscar Benjamin  wrote:
> >
> > - It is possible to configure a default version (although I think you
> > have to do it with an environment variable)
>
> The py launcher in Windows supports a "py.ini" file beside the
> executable and in %LocalAppData%. The equivalent of the PY_PYTHON,
> PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the
> "[defaults]" section as "python", "python2", and "python3" settings.
>
> The ini file also supports a "[commands]" section to define additional
> virtual commands for shebangs. Regular filepaths are also supported in
> shebangs -- e.g. #!"path\to\any\program.exe".

Yes, I forgot about that. I still think it would be better if you
could do it with the py command...

> > - I think that the launcher is only installed in an all users install.
>
> It defaults to an all-users install, but it can also be installed for
> just the current user in "%LocalAppData%\Programs\Python\Launcher". In
> this case, the installation directory always gets added to PATH.

Oh, okay. So does that mean that it's always on PATH unless the user
*explicitly unticks* the "install the launcher" box for both single
user and all user installs?

> > - It would be better if you could use the launcher itself to set the
> > default Python e.g. "py --set-default-python=3.8"
>
> It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value
> in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe
> naively to set PATH.)

Those commands aren't portable to any other OS though. The advantage
of having it done through py is that it can (hopefully) be made
consistent across platforms.

> > On the last point I think that although Anaconda doesn't install the
> > launcher you can use the launcher to run the python from the Anaconda
> > installation.
>
> I don't use Anaconda, but I don't think that's supposed to be the case
> according to PEP 514. The launcher only looks for PSF development
> distributions in the "PythonCore" registry key.

Maybe I misunderstood the situation. If py can only be used for PSF
binaries then that wouldn't work. I guess that can be changed
though...


--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VDB4RXS3RUO6ALU4S22N6PKWTHIHWI2K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Eryk Sun]

On 2/28/21, Oscar Benjamin  wrote:
>
> Oh, okay. So does that mean that it's always on PATH unless the user
> *explicitly unticks* the "install the launcher" box for both single
> user and all user installs?

If the launcher gets installed, it will be available in PATH. IIRC,
the installer only allows installing the launcher for the current user
if it is not installed already for all users. Thus only one version
should ever exist in PATH, which is set in either the user or system
"PATH" value in the registry, but not both.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3QWZRQSI5BTM3GQRWNH2UQN4L7RONAPJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Cameron Simpson]

On 28Feb2021 23:56, Irit Katriel  wrote:
>If you go long, I go longer :)

:-)

>On Sun, Feb 28, 2021 at 10:51 PM Cameron Simpson  wrote:
>> On 28Feb2021 10:40, Irit Katriel  wrote:
>> >split() and subgroup() take care to preserve the correct metadata on
>> >all
>> >the internal nodes, and if you just use them you only make safe
>> operations.
>> >This is why I am hesitating to add iteration utilities to the API. Like we
>> >did, people will naturally try that first, and it's not the safest API.
>>
>> Wouldn't it be safe if the ExceptionGroup were immutable, as you plan?
>> Or have you another hazard in mind?
>
>Making them immutable won't help the metadata issue. split() and subgroup()
>copy the (context, cause traceback) from the original ExceptionGroups (root
>and internal nodes of the tree) to the result trees.   If you DIY creating
>new ExceptionGroups you need to take care of that.

Ah, right. Yes.

The overflows into my request for a factory method to construct a "like" 
ExceptionGroup with a new exception tree lower down.

>> But all that said, being able to iterable the subexceptions seems a
>> natural way to manage that:
>>
>> unhandled = []
>> try:
>> .
>> except *OSError as eg:
>> for e in eg:
>> if an ok exception:
>> handle it
>> else:
>> unhandled.append(e)
>> if unhandled:
>> raise ExceptionGroup("unhandled", unhandled)
>
>
>You just lost the metadata of eg. It has no context, cause and its
>traceback begins here.

Aye. Hence a wish, again lower down, for some reference to the source 
ExceptionGroup and therefore a handy factory for making a new group with 
the right metadata.

>And the exceptions contained in it, if they came
>from a deeper tree that you flattened into the list, now look like their
>traceback jumps straight to here from the place they were actually first
>inserted into an ExceptionGroup. This may well be an impossible code path.

Perhaps so. But it doesn't detract from how useful it is to iterate over 
the inner exceptions. I see this as an argument for making it possible 
to obtain the correct metadata, not against iteration itself. Even if 
the iteration yielded some proxy or wrapper for the inner exception 
instead of the naked exception itself.

>Here's an example:
[... flattened ExceptionGroup with uninformative tracebacks ...]
 import traceback
 def flatten(exc):
>... if isinstance(exc, ExceptionGroup):
>... for e in exc.errors:
>...yield from flatten(e)
>... else:
>... yield exc
[...]
 traceback.print_exception(flat_h())
>Traceback (most recent call last):
>  File "", line 3, in flat_h
>ExceptionGroup: flat_h
>   
>   Traceback (most recent call last):
> File "", line 3, in f
>   ValueError: 42
>
>traceback.print_exception(h()) prints a reasonable traceback - h() called
>g() called f().
>
>But according to  traceback.print_exception(flat_h()),   flat_h() called
>f().
>
>You can preserve the metadata (and the nested structure with all its
>metadata) if you replace the last line with:
>raise eg.subgroup(lambda e: e in  unhandled)

Ok. That works for me as my desired factory. Verbose maybe, but 
workable. Um. It presumes exception equality will do - that feels 
slightly error prone (including some similar but not unhandled 
exceptions).  But I can write something more pointed based on id().

>And for the part before that, iteration, Guido's pattern showed that you
>can roll it into the subgroup callback.

Aye.

>> There are some immediate shortcomings above. In particular, I have no
>> way of referencing the original ExceptionGroup without surprisingly
>> cumbersome:
>
>> try:
>> .
>> except ExceptionGroup as eg0:
>> unhandled = []
>> eg, os_eg = eg0.split(OSError)
>> if os_eg:
>> for e in os_eg:
>> if an ok exception:
>> handle it
>> else:
>> unhandled.append(e)
>> if eg:
>> eg, value_eg = eg.split(ValueError)
>> if value_eg:
>> for e in value_eg:
>> if some_ValueError_we_understand:
>> handle it
>> else:
>> unhandled.append(e)
>> if eg:
>> unhandled.append(eg)
>> if unhandled:
>> raise ExceptionGroup("unhandled", unhandled) from eg0
>
>This is where except* can help:
>
>  try:
>.
>except except *OSError as eg:
>unhandled = []
>handled, unhandled = eg.split(lambda e: e is an ok exception)  #
>with side effect to handle e
>if unhandled:
>raise unhandled
>except *ValueError as eg:
>handled, unhandled = eg.split(lambda e: e is a value error we
>understand)  # with side effect to handle e

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

I'm trying to shorten this again...

On Sun, Feb 28, 2021 at 5:54 PM Cameron Simpson  wrote:

>
> Let's turn this on its head:
>
> - what specific harm comes from giving EGs container truthiness for size
>   testing?
>

For something that's not a pure container, this is an anti-pattern. No
other subclass of BaseException (that I know of) can ever be falsey, so
existing code would be forgiven to write "if e:" where "if e is not None:"
was meant.


> - what specific harm comes from returning an empty EG from split on no
>   match instead of None?
>

It would be an exception. What is raising an empty EG supposed to do?


> - what specific harm comes from supporting iteration with a caveat about
>   metadata in the docstring, and maybe a recommendation of subgroup?
>

Iteration is such an easy-to-use API that it would become an attractive
nuisance -- people would use it regardless of whether it's the right tool
for the job.

>
> - if I wanted to override subgroup and split to not return None, is that
>   even possible with the current ? i.e. can I make a clean metadata
>   preserved EG from an empty list? For example:
>
> eg2 = eg.subgroup(lambda e: False)
>
> Does that get me None, or an empty group? If the latter, I can roll my
> own subclass with my desired features. If not, I can't AFAICT.
>

It returns None. Writing a subclass that behaves differently than the base
class will just confuse users who aren't aware of your overrides.


> EGs _have_ a .errors attribute which has all these aspects, why not
> expand it to the class as a whole?
>

They are different. The `errors` attribute may contain other EGs. But
presumably iteration would recurse into those so as to flatten the tree and
yield only leaf nodes. The attribute is necessary so it is possible to
write traversal functions (including that iterator you so desperately want
-- as a helper function).


> You seem very happen to implement 80% of what I want using callbacks
> (lambda e: ...), but I find explicit iteration much easier to read. I
> rarely use filter() for example, and often prefer a generator expression
> of list comprehension.
>

Our (the PEP authors') belief is that in most cases users are better off
not handling the leaf exceptions. If you want to handle e.g. KeyError, the
place to do that is in the try/except clause immediately surrounding the
offending `__getitem__` operation (which will never raise an EG), not
several stack frames below, where you're potentially dealing with an
aggregate of many KeyErrors, or perhaps a KeyError and a TypeError. At that
point the most common option is to ignore or log some or all exception
types.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GBUJJLRV5OH3ZE4OGWVVOBY2YLHFOWRR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Cameron Simpson]

On 28Feb2021 20:05, Guido van Rossum  wrote:
>I'm trying to shorten this again...
>
>On Sun, Feb 28, 2021 at 5:54 PM Cameron Simpson  wrote:
>> Let's turn this on its head:
>> - what specific harm comes from giving EGs container truthiness for 
>> size
>>   testing?
>
>For something that's not a pure container, this is an anti-pattern. No
>other subclass of BaseException (that I know of) can ever be falsey, so
>existing code would be forgiven to write "if e:" where "if e is not None:"
>was meant.

I that's how you feel, I'll withdraw my advocacy. I agree it's a 
difference.

I'll outline my view below the objections below, but agree to disagree 
here - I'm not seeking to convince you any more.

>> - what specific harm comes from returning an empty EG from split on 
>> no
>>   match instead of None?
>
>It would be an exception. What is raising an empty EG supposed to do?

It'd be rather pointless. It assumes a little care on the person making 
it (well, using it where I'd asked subgroup to make it). I guess if 
there's a fall through in the except* which reraises, its a burden on 
the clause author to check such a contingency.  But wouldn't such a 
check normally be written:

if eg:
raise

on the presumption that None (falsey) means nothing to raise? And 
wouldn't the same code work if eg looks like a container in truthiness?

>> - what specific harm comes from supporting iteration with a caveat 
>> about
>>   metadata in the docstring, and maybe a recommendation of subgroup?
>
>Iteration is such an easy-to-use API that it would become an attractive
>nuisance -- people would use it regardless of whether it's the right tool
>for the job.

Perhaps. But an easy to use API is normally what one wants. For you, 
this is a downside because you hold that it is also usually a poor 
choice for this facility. Maybe my use case and habits are unusual.

>> - if I wanted to override subgroup and split to not return None, is 
>> that
>>   even possible with the current ? i.e. can I make a clean metadata
>>   preserved EG from an empty list? For example:
>>
>> eg2 = eg.subgroup(lambda e: False)
>>
>> Does that get me None, or an empty group? If the latter, I can roll my
>> own subclass with my desired features. If not, I can't AFAICT.
>
>It returns None. Writing a subclass that behaves differently than the base
>class will just confuse users who aren't aware of your overrides.

Yeah, a subclass might be misleading. Maybe all I need is a help 
function or class.

Cheers,
Cameron Simpson 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z4BM5JR7XQOIXCLW6GCE5N6V5Q72XE7W/
Code of Conduct: http://python.org/psf/codeofconduct/