Re: [Python-Dev] Second post: PEP 557, Data Classes
On 11/26/2017 12:23 AM, Carl Meyer wrote: Hi Eric, Really excited about this PEP, thanks for working on it. Thanks, Carl. A couple minor questions: If compare is True, then eq is ignored, and __eq__ and __ne__ will be automatically generated. IMO it's generally preferable to make nonsensical parameter combinations an immediate error, rather than silently ignore one of them. Is there a strong reason for letting nonsense pass silently here? (I reviewed the previous thread; there was a lot of discussion about enums/flags vs two boolean params, but I didn't see explicit discussion of this issue; the only passing references I noticed said the invalid combo should be "disallowed", e.g. Guido in [1], which to me implies "an error.") I think you're right here. I'll change it to a ValueError. isdataclass(instance): Returns True if instance is an instance of a Data Class, otherwise returns False. Something smells wrong with the naming here. If I have @dataclass class Person: name: str I think it would be considered obvious and undeniable (in English prose, anyway) that Person is a dataclass. So it seems wrong to have `isdataclass(Person)` return `False`. Is there a reason not to let it handle either a class or an instance (looks like it would actually simplify the implementation)? I think of this as really "isdataclassinstance". Let's see what others think. There are places in dataclasses.py where I need to know both ways: for example fields() works with either a class or instance, but asdict() just an instance. For what it's worth, the equivalent attrs API, attr.has(), returns True for both an instance and a class. And the recommended solution for a namedtuple (check for existence of _fields) would also work for an instance and a class. And I suppose it's easy enough for the caller to further disallow a class, if that's what they want. Eric. Carl [1] https://mail.python.org/pipermail/python-dev/2017-September/149505.html On 11/25/2017 01:06 PM, Eric V. Smith wrote: The updated version should show up at https://www.python.org/dev/peps/pep-0557/ shortly. The major changes from the previous version are: - Add InitVar to specify initialize-only fields. - Renamed __dataclass_post_init__() to __post_init(). - Rename cmp to compare. - Added eq, separate from compare, so you can test unorderable items for equality. - Flushed out asdict() and astuple(). - Changed replace() to just call __init__(), and dropped the complex post-create logic. The only open issues I know of are: - Should object comparison require an exact match on the type? https://github.com/ericvsmith/dataclasses/issues/51 - Should the replace() function be renamed to something else? https://github.com/ericvsmith/dataclasses/issues/77 Most of the items that were previously discussed on python-dev were discussed in detail at https://github.com/ericvsmith/dataclasses. Before rehashing an old discussion, please check there first. Also at https://github.com/ericvsmith/dataclasses is an implementation, with tests, that should work with 3.6 and 3.7. The only action item for the code is to clean up the implementation of InitVar, but that's waiting for PEP 560. Oh, and if PEP 563 is accepted I'll also need to do some work. Feedback is welcomed! Eric. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/carl%40oddbird.net ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
On 11/26/2017 3:35 AM, Eric V. Smith wrote: On 11/26/2017 12:23 AM, Carl Meyer wrote: A couple minor questions: If compare is True, then eq is ignored, and __eq__ and __ne__ will be automatically generated. IMO it's generally preferable to make nonsensical parameter combinations an immediate error, rather than silently ignore one of them. Is there a strong reason for letting nonsense pass silently here? (I reviewed the previous thread; there was a lot of discussion about enums/flags vs two boolean params, but I didn't see explicit discussion of this issue; the only passing references I noticed said the invalid combo should be "disallowed", e.g. Guido in [1], which to me implies "an error.") I think you're right here. I'll change it to a ValueError. While creating an issue for this (https://github.com/ericvsmith/dataclasses/issues/88), it occurs to me that the class-level parameter really should be "order" or "orderable", not "compare". It made more sense when it was called "cmp", but "compare" now seems wrong. Because "eq" says "can I compare two instances", and what's currently called "compare" is "can I order two instances". Nick had a similar suggestion before the PEP was written (https://mail.python.org/pipermail/python-ideas/2017-May/045740.html). The field-level parameter should stay "compare", because it's used for both __gt__ and friends, as well as __eq__ and __ne__. It's saying "is this field used in all of the comparison methods". Eric. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
On 11/26/2017 3:48 AM, Eric V. Smith wrote: While creating an issue for this (https://github.com/ericvsmith/dataclasses/issues/88), it occurs to me that the class-level parameter really should be "order" or "orderable", not "compare". It made more sense when it was called "cmp", but "compare" now seems wrong. Because "eq" says "can I compare two instances", and what's currently called "compare" is "can I order two instances". Nick had a similar suggestion before the PEP was written (https://mail.python.org/pipermail/python-ideas/2017-May/045740.html). The field-level parameter should stay "compare", because it's used for both __gt__ and friends, as well as __eq__ and __ne__. It's saying "is this field used in all of the comparison methods". I created https://github.com/ericvsmith/dataclasses/issues/90 for this. I think I'll leave 'eq' alone, and change 'compare' to 'order', for the class-level parameter name. Eric. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 4:57 PM, Nick Coghlan wrote: > On 26 November 2017 at 02:59, Guido van Rossum wrote: > > > > I'd be happy to stop with the conclusion that we're going to rip out some > > confusing syntax rather than trying to generate code for it -- IMO we've > > proved to ourselves that this stuff is too complicated to be useful. > > I'll also note that even if we go through a period of deprecating and > then prohibiting the syntax entirely, we'll still have the option of > bringing support for "yield" in comprehensions back later with > deliberately designed semantics (as happened for "await" in > https://www.python.org/dev/peps/pep-0530/#await-in-comprehensions), as > opposed to the accident-of-implementation semantics they have now. > > It may also turn out that as more asynchronous code is able to switch > to being 3.6+ only, allowing "await" and prohibiting "yield" will > prove to be sufficient for all practical purposes (as even the "yield > from" based spelling is Python-3-only, so it's only code that still > has to support 3.3, 3.4, 3.5, without needing to support 2.7, that > could use "yield from + yield" comprehensions, but wouldn't have the > option of just switching to async+await instead). > And I'll also note that some style guides recommend using comprehensions only for simple cases. Example: https://google.github.io/styleguide/pyguide.html#List_Comprehensions -- expand by clicking on the arrow; the "Con" comment is "Complicated list comprehensions or generator expressions can be hard to read." -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
On Sat, Nov 25, 2017, 14:00 Eric V. Smith, wrote: > The updated version should show up at > https://www.python.org/dev/peps/pep-0557/ shortly. > > The major changes from the previous version are: > > - Add InitVar to specify initialize-only fields. > - Renamed __dataclass_post_init__() to __post_init(). > - Rename cmp to compare. > - Added eq, separate from compare, so you can test >unorderable items for equality. > - Flushed out asdict() and astuple(). > - Changed replace() to just call __init__(), and dropped >the complex post-create logic. > It looks great and I'm excited to get to start using this PEP! > The only open issues I know of are: > - Should object comparison require an exact match on the type? >https://github.com/ericvsmith/dataclasses/issues/51 I say don't require the type comparison for duck typing purposes. -Brett > - Should the replace() function be renamed to something else? >https://github.com/ericvsmith/dataclasses/issues/77 > > Most of the items that were previously discussed on python-dev were > discussed in detail at https://github.com/ericvsmith/dataclasses. Before > rehashing an old discussion, please check there first. > > Also at https://github.com/ericvsmith/dataclasses is an implementation, > with tests, that should work with 3.6 and 3.7. The only action item for > the code is to clean up the implementation of InitVar, but that's > waiting for PEP 560. Oh, and if PEP 563 is accepted I'll also need to do > some work. > > Feedback is welcomed! > > Eric. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
On 11/26/2017 1:04 PM, Brett Cannon wrote: The only open issues I know of are: - Should object comparison require an exact match on the type? https://github.com/ericvsmith/dataclasses/issues/51 I say don't require the type comparison for duck typing purposes. The problem with that is that you end up with cases like this, which I don't think we want: @dataclass class Point: x: int y: int @dataclass class Point3d: x: int y: int z: int assert Point(1, 2) == Point3d(1, 2, 3) Eric. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum wrote: > On Sat, Nov 25, 2017 at 1:05 PM, David Mertz wrote: >> >> FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in real >> code. Probably 80% of those would be fine with yield statements, but a >> significant fraction use `gen.send()`. >> >> On the other hand, I have yet once to use 'await', or 'async' outside of >> pedagogical contexts. There are a whole lot of generators, including ones >> utilizing state injection, that are useful without the scaffolding of an >> event loop, in synchronous code. > > > Maybe you didn't realize async/await don't need an event loop? Driving an > async/await-based coroutine is just as simple as driving a yield-from-based > one (`await` does exactly the same thing as `yield from`). Technically anything you can write with yield/yield from could also be written using async/await and vice-versa, but I think it's actually nice to have both in the language. The distinction I'd make is that yield/yield from is what you should use for ad hoc coroutines where the person writing the code that has 'yield from's in it is expected to understand the details of the coroutine runner, while async/await is what you should use when the coroutine running is handled by a library like asyncio, and the person writing code with 'await's in it is expected to treat coroutine stuff as an opaque implementation detail. (NB I'm using "coroutine" in the CS sense here, where generators and async functions are both "coroutines".) I think of this as being sort of half-way between a style guideline and a technical guideline. It's like the guideline that lists should be homogenously-typed and variable length, while tuples are heterogenously-typed and fixed length: there's nothing in the language that outright *enforces* this, but it's a helpful convention *and* things tend to work better if you go along with it. Here are some technical issues you'll run into if you try to use async/await for ad hoc coroutines: - If you don't iterate an async function, you get a "coroutine never awaited" warning. This may or may not be what you want. - async/await has associated thread-global state like sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async libraries assume that they own these, and arbitrarily weird things may happen if you have multiple async/await coroutine runners in same thread with no coordination between them. - In async/await, it's not obvious how to write leaf functions: 'await' is equivalent to 'yield from', but there's no equivalent to 'yield'. You have to jump through some hoops by writing a class with a custom __await__ method or using @types.coroutine. Of course it's doable, and it's no big deal if you're writing a proper async library, but it's awkward for quick ad hoc usage. For a concrete example of 'ad hoc coroutines' where I think 'yield from' is appropriate, here's wsproto's old 'yield from'-based incremental websocket protocol parser: https://github.com/python-hyper/wsproto/blob/4b7db502cc0568ab2354798552148dadd563a4e3/wsproto/frame_protocol.py#L142 The flow here is: received_frames is the public API: it gives you an iterator over all completed frames. When it stops you're expected to add more data to the buffer and then call it again. Internally, received_frames acts as a coroutine runner for parse_more_gen, which is the main parser that calls various helper methods to parse different parts of the websocket frame. These calls eventually bottom out in _consume_exactly or _consume_at_most, which use 'yield' to "block" until enough data is available in the internal buffer. Basically this is the classic trick of using coroutines to write an incremental state machine parser as ordinary-looking code where the state is encoded in local variables on the stack. Using coroutines here isn't just a cute trick; I'm pretty confident that there is absolutely no other way to write a readable incremental websocket parser in Python. This is the 3rd rewrite of wsproto's parser, and I think I've read the code for all the other Python libraries that do this too. The websocket framing format is branchy enough that trying to write out the state machine explicitly will absolutely tie you in knots. (Of course we then rewrote wsproto's parser a 4th time for py2 compatibility; the current version's not *terrible* but the 'yield from' version was simpler and more maintainable.) For wsproto's use case, I think using 'await' would be noticeably worse than 'yield from'. It'd make the code more opaque to readers (people know generators but no-one shows up already knowing what @types.coroutine does), the "coroutine never awaited" warnings would be obnoxious (it's totally fine to instantiate a parser and then throw it away without using it!), and the global state issues would make us very nervous (wsproto is absolutely designed to be used alongside a library like asyncio or trio). But that's fine; 'yield from' exists a
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith wrote: > On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum > wrote: > > On Sat, Nov 25, 2017 at 1:05 PM, David Mertz wrote: > >> > >> FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in > real > >> code. Probably 80% of those would be fine with yield statements, but a > >> significant fraction use `gen.send()`. > >> > >> On the other hand, I have yet once to use 'await', or 'async' outside of > >> pedagogical contexts. There are a whole lot of generators, including > ones > >> utilizing state injection, that are useful without the scaffolding of an > >> event loop, in synchronous code. > > > > > > Maybe you didn't realize async/await don't need an event loop? Driving an > > async/await-based coroutine is just as simple as driving a > yield-from-based > > one (`await` does exactly the same thing as `yield from`). > > Technically anything you can write with yield/yield from could also be > written using async/await and vice-versa, but I think it's actually > nice to have both in the language. > Perhaps. You seem somewhat biased towards the devil you know, but you also bring up some good points. > The distinction I'd make is that yield/yield from is what you should > use for ad hoc coroutines where the person writing the code that has > 'yield from's in it is expected to understand the details of the > coroutine runner, while async/await is what you should use when the > coroutine running is handled by a library like asyncio, and the person > writing code with 'await's in it is expected to treat coroutine stuff > as an opaque implementation detail. (NB I'm using "coroutine" in the > CS sense here, where generators and async functions are both > "coroutines".) > > I think of this as being sort of half-way between a style guideline > and a technical guideline. It's like the guideline that lists should > be homogenously-typed and variable length, while tuples are > heterogenously-typed and fixed length: there's nothing in the language > that outright *enforces* this, but it's a helpful convention *and* > things tend to work better if you go along with it. > Hm. That would disappoint me. We carefully tried to design async/await to *not* require an event loop. (I'll get to the global state below.) > Here are some technical issues you'll run into if you try to use > async/await for ad hoc coroutines: > > - If you don't iterate an async function, you get a "coroutine never > awaited" warning. This may or may not be what you want. > It should indicate a bug, and the equivalent bug is silent when you're using yield-from, so I see this as a positive. If you find yourself designing an API where abandoning an async function is a valid action you should probably think twice. > - async/await has associated thread-global state like > sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async > libraries assume that they own these, and arbitrarily weird things may > happen if you have multiple async/await coroutine runners in same > thread with no coordination between them. > The existence of these is indeed a bit unfortunate for this use case. I'm CC'ing Yury to ask him if he can think of a different way to deal with the problems that these are supposed to solve. For each, the reason they exist is itself an edge case -- debugging for the former, finalization for the latter. A better solution for these problem may also be important for situations where multiple event loops exist (in the same thread, e.g. running alternately). Maybe a context manager could be used to manage this state better? > - In async/await, it's not obvious how to write leaf functions: > 'await' is equivalent to 'yield from', but there's no equivalent to > 'yield'. You have to jump through some hoops by writing a class with a > custom __await__ method or using @types.coroutine. Of course it's > doable, and it's no big deal if you're writing a proper async library, > but it's awkward for quick ad hoc usage. > Ah, yes, you need the equivalent of a Future. Maybe we should have a simple one in the stdlib that's not tied to asyncio. > For a concrete example of 'ad hoc coroutines' where I think 'yield > from' is appropriate, here's wsproto's old 'yield from'-based > incremental websocket protocol parser: > > https://github.com/python-hyper/wsproto/blob/ > 4b7db502cc0568ab2354798552148dadd563a4e3/wsproto/frame_protocol.py#L142 > Ah yes, these kinds of parsers are interesting use cases for coroutines. There are many more potential use cases than protocol parsers -- e.g. the Python REPL could really use one if you want to replicate it completely in pure Python. > The flow here is: received_frames is the public API: it gives you an > iterator over all completed frames. When it stops you're expected to > add more data to the buffer and then call it again. Internally, > received_frames acts as a coroutine runner for parse_more_gen, which > is the main parser that
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sun, Nov 26, 2017 at 6:51 PM, Guido van Rossum wrote: > On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith wrote: [..] >> - async/await has associated thread-global state like >> sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async >> libraries assume that they own these, and arbitrarily weird things may >> happen if you have multiple async/await coroutine runners in same >> thread with no coordination between them. > > > The existence of these is indeed a bit unfortunate for this use case. I'm > CC'ing Yury to ask him if he can think of a different way to deal with the > problems that these are supposed to solve. For each, the reason they exist > is itself an edge case -- debugging for the former, finalization for the > latter. A better solution for these problem may also be important for > situations where multiple event loops exist (in the same thread, e.g. > running alternately). Maybe a context manager could be used to manage this > state better? Yeah, both of them are for solving edge cases: - sys.set_coroutine_wrapper() is a debug API: asyncio uses it to slightly enhance the warning about non-awaited coroutines by showing where they were *created*. Capturing traceback when we create every coroutine is expensive, hence we only do that in asyncio debug mode. - sys.set_asyncgen_hooks() is more important, we use it to let event loops finalize partially iterated and then abandoned asynchronous generators. The rule of thumb is to get the previous coro-wrapper/asyncgen-hook, set your own (both are thread specific), and after you're done -- restore the saved old versions back. This should work fine in all use cases (even a trio event loop nested in an asyncio event loop). But nested coroutine loops is a sign of bad design, the nested loop will completely block the execution of the outer event loop, so situations like that should be avoided. Not to mention that debugging becomes much harder when you have complex nested coroutine runners. If someone wants to run some async/await code without an event loop for educational purposes or to implement some pattern, they likely don't even need to use any of the above APIs. For me the status quo is OK: asyncio/twisted/curio/trio use these APIs internally, and for them it shouldn't be a problem to use a couple low-level functions from sys. That said I have a couple ideas: - We can simplify these low-level APIs by combining set_coroutine_wrapper() and set_asyncgen_hooks() into one function -- set_async_hooks(coroutine_wrapper=, firstiter=, etc). It could return a context-manager, so the following would be possible: with sys.set_async_hooks(...): # run async code or an event loop - Another option is to design an API that would let you to stack your coro-wrapper/asyncgen-hooks, so that many coroutine runners can control coroutines/generators simultaneously. This sounds very complex to me though, and I haven't seen any compelling real-world use case that would require this kind of design. sys.set_coroutine_wrapper() is marked as an experimental debug API; sys.set_asyncgen_hooks() is still provisional. So we can in theory change/improve both of them. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 27 November 2017 at 06:29, Nathaniel Smith wrote: > - In async/await, it's not obvious how to write leaf functions: > 'await' is equivalent to 'yield from', but there's no equivalent to > 'yield'. You have to jump through some hoops by writing a class with a > custom __await__ method or using @types.coroutine. Of course it's > doable, and it's no big deal if you're writing a proper async library, > but it's awkward for quick ad hoc usage. > > For a concrete example of 'ad hoc coroutines' where I think 'yield > from' is appropriate, here's wsproto's old 'yield from'-based > incremental websocket protocol parser: > > > https://github.com/python-hyper/wsproto/blob/4b7db502cc0568ab2354798552148dadd563a4e3/wsproto/frame_protocol.py#L142 sys.set_coroutine_wrapper itself is another case where you genuinely *can't* rely on async/await in the wrapper implementation. The current example shown at https://docs.python.org/3/library/sys.html#sys.set_coroutine_wrapper is of a case that will *fail*, since it would otherwise result in infinite recursion when the coroutine wrapper attempts to call the coroutine wrapper. Cheers, Nick. P.S. Making that point reminded me that I still haven't got around to updating those docs to also include examples of how to do it *right*: https://bugs.python.org/issue30578 -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Using async/await in place of yield expression
Changing subject line because this is way off to the side. Guido and Nathaniel point out that you can do everything yield expressions do with async/await *without* an explicit event loop. While I know that is true, it feels like the best case is adding fairly considerable ugliness to the code in the process. > On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum > wrote: > > Maybe you didn't realize async/await don't need an event loop? Driving an > > async/await-based coroutine is just as simple as driving a > yield-from-based > > one (`await` does exactly the same thing as `yield from`). > > On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith wrote: > Technically anything you can write with yield/yield from could also be > written using async/await and vice-versa, but I think it's actually > nice to have both in the language. > Here is some code which is definitely "toy", but follows a pattern pretty similar to things I really code using yield expressions: In [1]: from itertools import takewhile In [2]: def injectable_fib(a=1, b=2): ...: while True: ...: new = yield a ...: if new is not None: ...: a, b = new ...: a, b = b, a+b ...: In [3]: f = injectable_fib() In [4]: list(takewhile(lambda x: x<200, f)) Out[4]: [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144] In [5]: f.send((100,200)) Out[5]: 200 In [6]: list(takewhile(lambda x: x<1000, f)) Out[6]: [300, 500, 800] Imagining that 'yield' vanished from the language tomorrow, and I wanted to write the same thing with async/await, I think the best I can come up with is... actually, I just don't know who to do it without any `yield`. I can get as far as a slightly flawed: In [9]: async def atakewhile(pred, coro): ...: l = [] ...: async for x in coro: ...: if pred(x): ...: return l ...: l.append(x) But I just have no idea what would go in the body of async def afib_injectable(): (that is, if I'm prohibited a `yield` in there) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using async/await in place of yield expression
On Mon, Nov 27, 2017 at 2:20 PM, David Mertz wrote: > Changing subject line because this is way off to the side. Guido and > Nathaniel point out that you can do everything yield expressions do with > async/await *without* an explicit event loop. While I know that is true, it > feels like the best case is adding fairly considerable ugliness to the code > in the process. > >> >> On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum >> wrote: >> > Maybe you didn't realize async/await don't need an event loop? Driving >> > an >> > async/await-based coroutine is just as simple as driving a >> > yield-from-based >> > one (`await` does exactly the same thing as `yield from`). > > >> >> On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith wrote: >> Technically anything you can write with yield/yield from could also be >> written using async/await and vice-versa, but I think it's actually >> nice to have both in the language. > > > Here is some code which is definitely "toy", but follows a pattern pretty > similar to things I really code using yield expressions: > > In [1]: from itertools import takewhile > In [2]: def injectable_fib(a=1, b=2): >...: while True: >...: new = yield a >...: if new is not None: >...: a, b = new >...: a, b = b, a+b >...: > In [3]: f = injectable_fib() > In [4]: list(takewhile(lambda x: x<200, f)) > Out[4]: [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144] > In [5]: f.send((100,200)) > Out[5]: 200 > In [6]: list(takewhile(lambda x: x<1000, f)) > Out[6]: [300, 500, 800] > > > Imagining that 'yield' vanished from the language tomorrow, and I wanted to > write the same thing with async/await, I think the best I can come up with > is... actually, I just don't know who to do it without any `yield`. > > I can get as far as a slightly flawed: > > In [9]: async def atakewhile(pred, coro): >...: l = [] >...: async for x in coro: >...: if pred(x): >...: return l >...: l.append(x) > > > But I just have no idea what would go in the body of > > async def afib_injectable(): > > > (that is, if I'm prohibited a `yield` in there) > Honestly, this is one of Python's biggest problems when it comes to async functions. I don't know the answer to that question, and I don't know where in the docs I'd go looking for it. In JavaScript, async functions are built on top of promises, so you can just say "well, you return a promise, tada". But in Python, this isn't well documented. Snooping the source code for asyncio.sleep() shows that it uses @coroutine and yield, and I have no idea what magic @coroutine does, nor how you'd use it without yield. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using async/await in place of yield expression
On 27 November 2017 at 13:20, David Mertz wrote: > > Imagining that 'yield' vanished from the language tomorrow, and I wanted > to write the same thing with async/await, I think the best I can come up > with is... actually, I just don't know who to do it without any `yield`. > I recently had to look into these things with quite some detail. When using `yield` for *iteration* specifically, you cannot use async/await to replace it. I find it easiest to think about all this in the context of the ABC table: https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes The `Generator` ABC also implement the `Iterator` protocol so this allows "normal" iteration to work, i.e. for-loop, while-loop, and comprehensions and so on. In contrast, the `Coroutine` ABC implements only send(), throw() and close(). This means that if you want to iterate a coroutine, *something* must drive send(), and Python's iteration syntax features don't do that. async/await is only useful when an event loop drives coroutines using the Coroutine ABC protocol methods. Note that AsyncIterable and AsyncIterator doesn't help because objects implementing these protocols may only legally appear inside a coroutine, i.e. an `async def` coroutine function, which you still cannot drive via the Iterator protocol (e.g., from a for-loop). The Coroutine ABC simply doesn't implement the Iterator protocol, so it seems it cannot be a replacement for generators. It is however true that `async/await` completely replaces `yield from` for *coroutines*, but both of those required a loop of some kind. I'd be very grateful if anyone can point out if my understanding of the above is incorrect. Private email is fine if you prefer not to post to the list. rgds Caleb ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
On 27 November 2017 at 06:22, Eric V. Smith wrote: > On 11/26/2017 1:04 PM, Brett Cannon wrote: >> >> The only open issues I know of are: >> - Should object comparison require an exact match on the type? >> https://github.com/ericvsmith/dataclasses/issues/51 >> >> >> I say don't require the type comparison for duck typing purposes. > > > The problem with that is that you end up with cases like this, which I don't > think we want: > > @dataclass > class Point: > x: int > y: int > > @dataclass > class Point3d: > x: int > y: int > z: int > > assert Point(1, 2) == Point3d(1, 2, 3) Perhaps the check could be: (type(lhs) == type(rhs) or fields(lhs) == fields(rhs)) and all (individual fields match) That way the exact type check would be an optimisation to speed up the common case, while the formal semantic constraint would be that the field definitions have to match (including their names and order). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using async/await in place of yield expression
On Sun, Nov 26, 2017 at 11:23 PM, Caleb Hattingh wrote: [..] > I'd be very grateful if anyone can point out if my understanding of the > above is incorrect. Private email is fine if you prefer not to post to the > list. It is correct. While 'yield from coro()', where 'coro()' is an 'async def' coroutine would make sense in some contexts, it would require coroutines to implement the iteration protocol. That would mean that you could write 'for x in coro()', which is meaningless for coroutines in all contexts. Therefore, coroutines do not implement the iterator protocol. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
Nick Coghlan wrote: Perhaps the check could be: (type(lhs) == type(rhs) or fields(lhs) == fields(rhs)) and all (individual fields match) I think the types should *always* have to match, or at least one should be a subclass of the other. Consider: @dataclass class Point3d: x: float y: float z: float @dataclass class Vector3d: x: float y: float z: float Points and vectors are different things, and they should never compare equal, even if they have the same field names and values. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using async/await in place of yield expression
On 27 November 2017 at 14:53, Yury Selivanov wrote: > It is correct. While 'yield from coro()', where 'coro()' is an 'async > def' coroutine would make sense in some contexts, it would require > coroutines to implement the iteration protocol. That would mean that > you could write 'for x in coro()', which is meaningless for coroutines > in all contexts. Therefore, coroutines do not implement the iterator > protocol. The two worlds (iterating vs awaiting) collide in an interesting way when one plays with custom Awaitables. >From your PEP, an awaitable is either a coroutine, or an object implementing __await__, *and* that __await__ returns an iterator. The PEP only says that __await__ must return an iterator, but it turns out that it's also required that that iterator should not return any intermediate values. This requirement is only enforced in the event loop, not in the `await` call itself. I was surprised by that: >>> class A: ... def __await__(self): ... for i in range(3): ... yield i # <--- breaking the rules, returning a value ... return 123 >>> async def cf(): ... x = await A() ... return x >>> c = cf() >>> c.send(None) 0 >>> c.send(None) 1 >>> c.send(None) 2 >>> c.send(None) Traceback (most recent call last): File "", line 1, in StopIteration: 123 123 So we drive the coroutine manually using send(), and we see that intermediate calls return the illegally-yielded values. I broke the rules because my __await__ iterator is returning values (via `yield i`) on each iteration, and that isn't allowed because the event loop wouldn't know what to do with these intermediate values; it only knows that "awaiting" is finished when a value is returned via StopIteration. However, you only find out that it isn't allowed if you use the loop to run the coroutine function: >>> import asyncio >>> loop = asyncio.get_event_loop() >>> loop.run_until_complete(f()) Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.6/asyncio/base_events.py", line 467, in run_until_complete return future.result() File "", line 2, in f File "", line 4, in __await__ RuntimeError: Task got bad yield: 0 Task got bad yield: 0 I found this quite confusing when I first came across it, before I understood how asyncio/async/await was put together. The __await__ method implementation must return an iterator that specifically doesn't return any intermediate values. This should probably be explained in the docs. I'm happy to help with any documentation improvements if help is desired. rgds Caleb ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
On 27 November 2017 at 15:04, Greg Ewing wrote: > Nick Coghlan wrote: >> >> Perhaps the check could be: >> >> (type(lhs) == type(rhs) or fields(lhs) == fields(rhs)) and all >> (individual fields match) > > > I think the types should *always* have to match, or at least > one should be a subclass of the other. Consider: > > @dataclass > class Point3d: > x: float > y: float > z: float > > @dataclass > class Vector3d: > x: float > y: float > z: float > > Points and vectors are different things, and they should never > compare equal, even if they have the same field names and values. And I guess if folks actually want more permissive structure-based matching, that's one of the features that collections.namedtuple offers that data classes don't. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
