Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
On 26 February 2018 at 01:08, Chris Angelico wrote: > Speaking as a C programmer who's quite happy to write code like "while > ((var = func()) != sentinel)", I wouldn't object to this coming up in > Python; the "as name" syntax has the huge advantage over C's syntax in > that you can't accidentally leave off one equals sign and get the > wrong behaviour. But I know that a lot of people dislike this at a > more fundamental level. > > If someone wants to push for this, I think it probably needs a PEP - > it's a point that comes up periodically. I don't think it's ever had a > PEP written about it, but it's a bit hard to search for; maybe someone > else knows off hand? > PEP 3150 is the most closely related PEP we have at the moment (and that only works for simple statements, since it relies on using a trailing block to name the subexpressions). The "(EXPR as NAME)" syntax comes up periodically, especially in the context of while loops (where it would allow a direct translation of C-style embedded assignment idioms). In addition to the potential confusion for "with (EXPR as NAME):" vs "with EXPR as NAME:" (and the similar ambiguity for "except" clauses), some other major questions to be resolved are: * are statement locals in a class namespace turned into attributes on the resulting class? (it would be more useful if they weren't) * are statement locals in a module namespace turned into attributes on the resulting module? (it would be more useful if they weren't) * are statement locals in a function/generator/coroutine namespace kept alive until the entire call terminates? (it would be more useful if they weren't) * do currently defined statement locals appear in calls to locals() or in frame.f_locals? * can lexically nested scopes see names bound this way? (a lot of complex name resolution problems disappear if they can't, plus you get a clearer distinction between these and regular function locals) To be interesting enough to potentially be worthy of syntax, I think name bindings written this way would need to be truly statement local: * reference is released at the end of the statement (whether simple or compound) * no ability to close over them (this goes hand in hand with eagerly dropping the reference) * we play name mangling games and/or use different opcodes to avoid overwriting regular function locals and to avoid appearing in locals() That's still only enough to get the concept into python-ideas territory though (as per the discussion of "(EXPR as .NAME)" in https://mail.python.org/pipermail/python-ideas/2018-February/049002.html) - it's still a *long* way from being fully baked enough to make into a concrete change proposal for 3.8+. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
On Mon, Feb 26, 2018 at 8:00 PM, Nick Coghlan wrote: > On 26 February 2018 at 01:08, Chris Angelico wrote: >> >> Speaking as a C programmer who's quite happy to write code like "while >> ((var = func()) != sentinel)", I wouldn't object to this coming up in >> Python; the "as name" syntax has the huge advantage over C's syntax in >> that you can't accidentally leave off one equals sign and get the >> wrong behaviour. But I know that a lot of people dislike this at a >> more fundamental level. >> >> If someone wants to push for this, I think it probably needs a PEP - >> it's a point that comes up periodically. I don't think it's ever had a >> PEP written about it, but it's a bit hard to search for; maybe someone >> else knows off hand? > > > PEP 3150 is the most closely related PEP we have at the moment (and that > only works for simple statements, since it relies on using a trailing block > to name the subexpressions). > > The "(EXPR as NAME)" syntax comes up periodically, especially in the context > of while loops (where it would allow a direct translation of C-style > embedded assignment idioms). > > In addition to the potential confusion for "with (EXPR as NAME):" vs "with > EXPR as NAME:" (and the similar ambiguity for "except" clauses), some other > major questions to be resolved are: > > * are statement locals in a class namespace turned into attributes on the > resulting class? (it would be more useful if they weren't) > * are statement locals in a module namespace turned into attributes on the > resulting module? (it would be more useful if they weren't) > * are statement locals in a function/generator/coroutine namespace kept > alive until the entire call terminates? (it would be more useful if they > weren't) > * do currently defined statement locals appear in calls to locals() or in > frame.f_locals? > * can lexically nested scopes see names bound this way? (a lot of complex > name resolution problems disappear if they can't, plus you get a clearer > distinction between these and regular function locals) > > To be interesting enough to potentially be worthy of syntax, I think name > bindings written this way would need to be truly statement local: > > * reference is released at the end of the statement (whether simple or > compound) > * no ability to close over them (this goes hand in hand with eagerly > dropping the reference) > * we play name mangling games and/or use different opcodes to avoid > overwriting regular function locals and to avoid appearing in locals() Definitely possible. I wish I still had the POC patch that I put together a while ago that created block-local variables from 'as' bindings in except and with statements, but it got lost in a hard drive crash (because, at the time, I didn't think it was useful for anything more than "haha, isn't that cute"). Will see if I can recreate it. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
Michel Desmoulin writes: > Le 25/02/2018 à 14:11, Nikolaus Rath a écrit : >>> result = [ (f(x) as y) + g(y) for x in range(10)] > Honestly I find this version the most readable while the double for > loop is completely weird to me, despite doing python for a living > for years. I find this one less readable because I don't expect name binding syntax to return a value. My brain is nonplussed by the "+". :-) > I really hope the later doesn't become a common idiom. It already is common, for values of "common" = "some people have been using it where it's useful, but it's not useful all that often". I suppose it's rare because it's a less-than-readable optimization, which is frowned on in Python programming. Somebody counted four or five ways to perform this optimization, including this "double for" that allows the common subexpression optimization to be made explicit in the comprehension. We don't need another, not at the cost of new syntax. If we find that we really want a C-like assignment expression that returns the assigned value[1], I don't have an objection to that. But I don't personally feel a need for it. Footnotes: [1] Yes, I know that technically it's a local name binding, not an assignment. But until the scope of "local" is resolved, it looks, smells, and tastes like an assignment. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Code stability and a big thank you
As part of the discussions on local name binding, I dug up a patch that I sent to python-ideas back in June 2015. That's two and a half years ago, and was based on CPython 3.5. Here's what happened when I attempted to apply it to the current master branch (3.8): $ patch <~/Downloads/scope_hack.patch -p1 patching file Python/compile.c Hunk #1 succeeded at 100 (offset 3 lines). Hunk #2 succeeded at 134 (offset 3 lines). Hunk #3 succeeded at 793 (offset 2 lines). Hunk #4 succeeded at 2824 (offset 432 lines). Hunk #5 succeeded at 2871 with fuzz 2 (offset 430 lines). Hunk #6 succeeded at 3312 (offset 402 lines). Hunk #7 succeeded at 3330 (offset 402 lines). Hunk #8 succeeded at 5442 with fuzz 1 (offset 831 lines). Hunk #9 succeeded at 5457 with fuzz 2 (offset 832 lines). Hunk #10 succeeded at 5520 with fuzz 2 (offset 823 lines). $ And the build ran fine, the test suite didn't gain any new failures (there's no tests for the new functionality, but it didn't break anything else), and my original demo program worked just fine. Thank you to all the CPython core devs for maintaining code that is THIS stable. Stuff happens but there's no unnecessary code churn. Sure, GNU Patch had to do a bit of work to merge in the changes, but it did it completely automatically. You folks are awesome! ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should the dataclass frozen property apply to subclasses?
I've opened https://bugs.python.org/issue32953 to track this.
On 2/22/18 5:55 AM, Eric V. Smith wrote:
On 2/22/2018 1:56 AM, Raymond Hettinger wrote:
When working on the docs for dataclasses, something unexpected came
up. If a dataclass is specified to be frozen, that characteristic is
inherited by subclasses which prevents them from assigning additional
attributes:
>>> @dataclass(frozen=True)
class D:
x: int = 10
>>> class S(D):
pass
>>> s = S()
>>> s.cached = True
Traceback (most recent call last):
File "", line 1, in
s.cached = True
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/dataclasses.py",
line 448, in _frozen_setattr
raise FrozenInstanceError(f'cannot assign to field {name!r}')
dataclasses.FrozenInstanceError: cannot assign to field 'cached'
This is because "frozen-ness" is implemented by adding __setattr__ and
__delattr__ methods in D, which get inherited by S.
...
A related issue is that dataclasses derived from frozen dataclasses are
automatically "promoted" to being frozen.
@dataclass(frozen=True)
... class A:
... i: int
...
@dataclass
... class B(A):
... j: int
...
b = B(1, 2)
b.j = 3
Traceback (most recent call last):
File "", line 1, in
File "C:\home\eric\local\python\cpython\lib\dataclasses.py", line 452,
in _frozen_setattr
raise FrozenInstanceError(f'cannot assign to field {name!r}')
dataclasses.FrozenInstanceError: cannot assign to field 'j'
This is tricky to fix.
Here's the problem with a inheriting a non-frozen dataclass from a
frozen one. Consider class Y in this example:
@dataclass
class X:
x: int
@dataclass
class Y(X):
y: int
Y's __init__ looks like:
def __init__(self, x, y):
self.x = x
self.y = y
That is, all of the initializing for Y and its base classes which are
dataclasses is done in Y's __init__. There are a number of reasons for
this, including performance and not knowing how to call the base
classes' __init__ methods.
If Y is frozen, then the __init__ currently looks like:
def __init__(self, x, y):
object.__setattr__(self, 'x', x)
object.__setattr__(self, 'y', y)
If X is frozen but Y isn't, then it should look like:
def __init__(self, x, y):
object.__setattr__(self, 'x', x)
self.y = y
But I currently can't generate the code that way, because I don't know
if a base dataclass is frozen. That information is not saved on a dataclass.
I think the right thing to do is to record with each dataclass if it is
frozen or not, so that derived classes can generate the correct code.
Another option would be to always use object.__setattr__, but that will
hurt performance in the common case.
As long as I'm saving if a dataclass is frozen, I should save all of the
dataclass parameters on the class. Since it's per-class, it's not a lot
of overhead.
I should probably separate the two issues here: 1) deriving a
non-dataclass from a frozen dataclass and 2) deriving a non-frozen
dataclass from a frozen dataclass, but since they're somewhat related
I'm going to keep them together for the time being. #1 was Raymond's
initial report.
Eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should the dataclass frozen property apply to subclasses?
On 2/22/18 9:43 PM, Nick Coghlan wrote:
On 22 February 2018 at 20:55, Eric V. Smith wrote:
On 2/22/2018 1:56 AM, Raymond Hettinger wrote:
When working on the docs for dataclasses, something unexpected came up.
If a dataclass is specified to be frozen, that characteristic is inherited
by subclasses which prevents them from assigning additional attributes:
>>> @dataclass(frozen=True)
class D:
x: int = 10
>>> class S(D):
pass
>>> s = S()
>>> s.cached = True
Traceback (most recent call last):
File "", line 1, in
s.cached = True
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/dataclasses.py",
line 448, in _frozen_setattr
raise FrozenInstanceError(f'cannot assign to field {name!r}')
dataclasses.FrozenInstanceError: cannot assign to field 'cached'
This is because "frozen-ness" is implemented by adding __setattr__ and
__delattr__ methods in D, which get inherited by S.
Other immutable classes in Python don't behave the same way:
>>> class T(tuple):
pass
>>> t = T([10, 20, 30])
>>> t.cached = True
>>> class F(frozenset):
pass
>>> f = F([10, 20, 30])
>>> f.cached = True
>>> class B(bytes):
pass
>>> b = B()
>>> b.cached = True
The only way I can think of emulating this is checking in __setattr__ to see
if the field name is a field of the frozen class, and only raising an error
in that case.
If you were going to do that then it would likely make more sense to
convert the frozen fields to data descriptors, so __setattr__ only
gets called for attempts to add new attributes.
Then for the `frozen=False` case, the decorator could force
__setattr__ and __delattr__ back to the default implementations from
object, rather than relying on the behaviour inherited from base
classes.
I guess it depends on what we're trying to do. By using data
descriptors, we'd be saying "the fields can't be modified, but other
parts of the class can". For example, what should happen here:
@dataclass(frozen=True)
class F:
i: int
f = F(10)
f.i = 0 # should be an error
f.j = 0 # should this be an error? It's not a field.
I was hoping to get all of this finished today, by a2, but it looks like
that's not going to happen.
A related issue is that dataclasses derived from frozen dataclasses are
automatically "promoted" to being frozen.
@dataclass(frozen=True)
... class A:
... i: int
...
@dataclass
... class B(A):
... j: int
...
b = B(1, 2)
b.j = 3
Traceback (most recent call last):
File "", line 1, in
File "C:\home\eric\local\python\cpython\lib\dataclasses.py", line 452, in
_frozen_setattr
raise FrozenInstanceError(f'cannot assign to field {name!r}')
dataclasses.FrozenInstanceError: cannot assign to field 'j'
Maybe it should be an error to declare B as non-frozen?
It would be nice to avoid that, as a mutable subclass of a frozen base
class could be a nice way to model hashable-but-mutable types:
>>> @dataclass(frozen=True) # This is the immutable/hashable bit
... class A:
... i: int
...
>>> @dataclass # This adds the mutable-but-comparable parts
... class B(A):
... j: int
... __hash__ = A.__hash__
However, disallowing this case for now *would* be a reasonable way to
postpone making a decision until 3.8 - in the meantime, folks would
still be able to experiment by overriding __setattr__ and __delattr__
after the dataclass decorator sets them.
For today, I'm going to make it an error to derive a frozen dataclass
from a non-frozen dataclass, and also an error to derive a non-frozen
dataclass from a frozen dataclass. With any luck (and Guido and Ned
willing), we can address this by a3.
Eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
On 22/02/2018 19:04, Serhiy Storchaka wrote: Yet one discussion about reusing common subexpressions in comprehensions took place last week on the Python-ideas maillist (see topic "Temporary variables in comprehensions" [1]). The problem is that in comprehension like `[f(x) + g(f(x)) for x in range(10)]` the subexpression `f(x)` is evaluated twice. In normal loop you can introduce a temporary variable for `f(x)`. The OP wanted to add a special syntax for introducing temporary variables in comprehensions. This idea already was discussed multiple times in the past. There are several ways of resolving this problem with existing syntax. [snip] Stephan Houben proposed an idiom which looks similar to new hypothetic syntax: result = [y + g(y) for x in range(10) for y in [f(x)]] `for y in [expr]` in a comprehension means just assigning expr to y. I never seen this idiom before, but it can be a good replacement for a hypothetic syntax for assignment in comprehensions. It changes the original comprehension less than other approaches, just adds yet one element in a sequence of for-s and if-s. I think that after using it more widely it will become pretty idiomatic. I have created a patch that optimizes this idiom, making it as fast as a normal assignment. [2] Yury suggested to ask Guido on the mailing list if he agrees that this language patten is worth optimizing/promoting. Here's a thought: allow the syntax for VAR = EXPR to define a for-loop that is executed exactly once (both inside and outside comprehensions), i.e. pretty much a synonym for for VAR in [ EXPR ] for VAR in ( EXPR , ) especially if Serhiy's optimisation means that the list/tuple is not actually constructed in the latter. Pros: (1) Stephan Houben's example could be written as result = [y + g(y) for x in range(10) for y = f(x)] which I find more readable. (2) Code such as for i in xrange(10): could be changed on the fly to: for i = 1: I see this as especially useful in debugging, where you want to limit the program execution to a known problematic bit. But it some contexts it could be good style. (3) Preserves the compatibility between a list comprehension and its "expansion" into for-loops. (4) Backward compatible, since it is currently illegal syntax (5) No extra keyword needed (6) It goes some way towards providing the functionality of with VAR as EXPR that has been discussed multiple times. Best wishes Rob Cliffe ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should the dataclass frozen property apply to subclasses?
On 02/26/2018 05:40 AM, Eric V. Smith wrote: As long as I'm saving if a dataclass is frozen, I should save all of the dataclass parameters on the class. Since it's per-class, it's not a lot of overhead. That's what Enum does. A bit of pain, but makes so many other things easier. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
I would like to remind all wannabe language designers that grammar design is not just solving puzzles. It's also about keeping the overall feel of the language readable. I'm getting the idea that none of the proposals discussed so far (whether new syntax or clever use of existing syntax) satisfy that constraint. Sometimes a for-loop is just better. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
On 27 February 2018 at 05:08, Guido van Rossum wrote: > I would like to remind all wannabe language designers that grammar design > is not just solving puzzles. It's also about keeping the overall feel of > the language readable. I'm getting the idea that none of the proposals > discussed so far (whether new syntax or clever use of existing syntax) > satisfy that constraint. Sometimes a for-loop is just better. > +1 This is the main reason PEP 3150 (which adds a more limited form of statement local named subexpressions) has spent more time Deferred than it has ever being discussed as an active draft proposal: while naming subexpressions is an occasionally attractive prospect, it's also an addition that has significant potential to change the way various kinds of code is typically written (even more so than something like type hints or f-strings). When even a PEP's author is thinking "I'm not sure this will actually be a net improvement to the language", it's really not a good sign :) Cheers, Nick. P.S. The comprehension-centric variants at least have the virtue of precedent in Haskell's "let" clauses: https://stackoverflow.com/questions/6067839/haskell-let-where-equivalent-within-list-comprehension/6067878#6067878 -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
On 26/02/2018 19:08, Guido van Rossum wrote: I would like to remind all wannabe language designers that grammar design is not just solving puzzles. It's also about keeping the overall feel of the language readable. I'm getting the idea that none of the proposals discussed so far (whether new syntax or clever use of existing syntax) satisfy that constraint. Sometimes a for-loop is just better. I don't know if you intended these remarks to include my proposal (to allow "for VAR = EXPR"), as your message was posted only 27 minutes after mine. With respect, I honestly feel that this is a relatively small change that makes the language *more* readable. Feel free, one and all, to tell me why I'm wrong. Best wishes, Rob Cliffe ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The `for y in [x]` idiom in comprehensions
On Mon, Feb 26, 2018 at 4:30 PM, Rob Cliffe via Python-Dev < python-dev@python.org> wrote: > On 26/02/2018 19:08, Guido van Rossum wrote: > > I would like to remind all wannabe language designers that grammar design > is not just solving puzzles. It's also about keeping the overall feel of > the language readable. I'm getting the idea that none of the proposals > discussed so far (whether new syntax or clever use of existing syntax) > satisfy that constraint. Sometimes a for-loop is just better. > > I don't know if you intended these remarks to include my proposal (to > allow "for VAR = EXPR"), as your message was posted only 27 minutes after > mine. > With respect, I honestly feel that this is a relatively small change that > makes the language *more* readable. > > Feel free, one and all, to tell me why I'm wrong. > Best wishes, > Rob Cliffe > I didn't want to single you out, but yes, I did include your proposal. The reason is that for people who are not Python experts there's no obvious reason why `for VAR = EXPR` should mean one thing and `for VAR in EXPR` should mean another. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
