Re: [Python-Dev] Using async/await in place of yield expression

[Caleb Hattingh]

On 27 November 2017 at 13:20, David Mertz  wrote:
>
> Imagining that 'yield' vanished from the language tomorrow, and I wanted
> to write the same thing with async/await, I think the best I can come up
> with is... actually, I just don't know who to do it without any `yield`.
>

I recently had to look into these things with quite some detail. When using
`yield` for *iteration* specifically, you cannot use async/await to replace
it. I find it easiest to think about all this in the context of the ABC
table:

https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes

The `Generator` ABC also implement the `Iterator` protocol so this allows
"normal" iteration to work, i.e. for-loop, while-loop, and comprehensions
and so on. In contrast, the `Coroutine` ABC implements only send(), throw()
and close().  This means that if you want to iterate a coroutine,
*something* must drive send(), and Python's iteration syntax features don't
do that.  async/await is only useful when an event loop drives coroutines
using the Coroutine ABC protocol methods. Note that AsyncIterable and
AsyncIterator doesn't help because objects implementing these protocols may
only legally appear inside a coroutine, i.e. an `async def` coroutine
function, which you still cannot drive via the Iterator protocol (e.g.,
from a for-loop).

The Coroutine ABC simply doesn't implement the Iterator protocol, so it
seems it cannot be a replacement for generators. It is however true that
`async/await` completely replaces `yield from` for *coroutines*, but both
of those required a loop of some kind.

I'd be very grateful if anyone can point out if my understanding of the
above is incorrect.  Private email is fine if you prefer not to post to the
list.

rgds
Caleb
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using async/await in place of yield expression

[Caleb Hattingh]

On 27 November 2017 at 14:53, Yury Selivanov 
wrote:

> It is correct.  While 'yield from coro()', where 'coro()' is an 'async
> def' coroutine would make sense in some contexts, it would require
> coroutines to implement the iteration protocol.  That would mean that
> you could write 'for x in coro()', which is meaningless for coroutines
> in all contexts.  Therefore, coroutines do not implement the iterator
> protocol.


The two worlds (iterating vs awaiting) collide in an interesting way when
one plays with custom Awaitables.
>From your PEP, an awaitable is either a coroutine, or an object
implementing __await__,
*and* that __await__  returns an iterator.

The PEP only says that __await__ must return an iterator, but it turns out
that it's also required that that iterator
should not return any intermediate values.  This requirement is only
enforced in the event loop, not
in the `await` call itself.  I was surprised by that:

>>> class A:
... def __await__(self):
... for i in range(3):
... yield i   # <--- breaking the rules, returning
a value
... return 123

>>> async def cf():
... x = await A()
... return x

>>> c = cf()
>>> c.send(None)
0
>>> c.send(None)
1
>>> c.send(None)
2
>>> c.send(None)
Traceback (most recent call last):
  File "", line 1, in 
StopIteration: 123
123


So we drive the coroutine manually using send(), and we see that
intermediate calls return the illegally-yielded values.  I broke the rules
because my __await__ iterator is returning values (via `yield i`) on each
iteration, and that isn't allowed because the event loop wouldn't know what
to do with these intermediate values; it only knows that "awaiting" is
finished when a value is returned via StopIteration.  However, you only
find out that it isn't allowed if you use the loop to run the coroutine
function:

>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(f())
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 467, in
run_until_complete
return future.result()
  File "", line 2, in f
  File "", line 4, in __await__
RuntimeError: Task got bad yield: 0
Task got bad yield: 0


I found this quite confusing when I first came across it, before I
understood how asyncio/async/await was put together. The __await__ method
implementation must return an iterator that specifically doesn't return any
intermediate values.  This should probably be explained in the docs. I'm
happy to help with any documentation improvements if help is desired.

rgds
Caleb
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using async/await in place of yield expression

[Caleb Hattingh]

On 27 November 2017 at 18:41, Nathaniel Smith  wrote:
>
> In asyncio, the convention is that the values you send back must be
> Future objects,


Thanks this is useful. I didn't pick this up from the various PEPs or
documentation.  I guess I need to go through the src :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com