[Python-Dev] async/await behavior on multiple calls
Howdy,
I'm experimenting with async/await in Python 3, and one very surprising
behavior has been what happens when calling `await` twice on an Awaitable.
In C#, Hack/HHVM, and the new async/await spec in Ecmascript 7. In Python,
calling `await` multiple times results in all future results getting back
`None`. Here's a small example program:
async def echo_hi():
result = ''
echo_proc = await asyncio.create_subprocess_exec(
'echo', 'hello', 'world',
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.DEVNULL)
result = await echo_proc.stdout.read()
await echo_proc.wait()
return result
async def await_twice(awaitable):
print('first time is {}'.format(await awaitable))
print('second time is {}'.format(await awaitable))
loop = asyncio.get_event_loop()
loop.run_until_complete(await_twice(echo_hi()))
This makes writing composable APIs using async/await in Python very
difficult since anything that takes an `awaitable` has to know that it
wasn't already awaited. Also, since the behavior is radically different
than in the other programming languages implementing async/await it makes
adopting Python's flavor of async/await difficult for folks coming from a
language where it's already implemented.
In C#/Hack/JS calls to `await` return a Task/AwaitableHandle/Promise that
can be awaited multiple times and either returns the result or throws any
thrown exceptions. It doesn't appear that the Awaitable class in Python
has a `result` or `exception` field but `asyncio.Future` does.
Would it make sense to shift from having `await` functions return a `
*Future-like`* return object to returning a Future?
Thanks,
Roy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await behavior on multiple calls
Thanks for the insight Guido.
I've mostly used async/await inside of HHVM/Hack, and used Guava/Java
Futures extensively in the past so I found this behavior to be quite
surprising. I'd like to use Awaitables to represent a DAG of work that
needs to get done. For example, I used to be one of the maintainers of
Buck (a build tool similar to Bazel) and we used a collection of futures
for building all of our dependencies. For each rule, we'd effectively:
dependency_results = await asyncio.gather(*dependencies)
# Proceed with building.
Rules were free to depend on the same dependency and since the Future would
just return the same result when resolved more than one time things just
worked.
Similarly when building up the results for say a web request, I effectively
want to construct a DAG of work that needs to get done and then just await
on that DAG in a similar manner without having to enforce that the DAG is
actually a tree. I can of course write a function to wrap everything in
Futures, but this seems to be against the spirit of async/await.
Thanks,
Roy
On Tue, Dec 15, 2015 at 12:08 PM, Guido van Rossum wrote:
> I think this goes back all the way to a debate we had when we were
> discussing PEP 380 (which introduced 'yield from', on which 'await' is
> built). In fact I believe that the reason PEP 380 didn't make it into
> Python 2.7 was that this issue was unresolved at the time (the PEP author
> and I preferred the current approach, but there was one vocal opponent who
> disagreed -- although my memory is only about 60% reliable on this :-).
>
> In any case, problem is that in order to implement the behavior you're
> asking for, the generator object would have to somehow hold on to its
> return value so that each time __next__ is called after it has already
> terminated it can raise StopIteration with the saved return value. This
> would extend the lifetime of the returned object indefinitely (until the
> generator object itself is GC'ed) in order to handle a pretty obscure
> corner case.
>
> I don't know how long you have been using async/await, but I wonder if
> it's possible that you just haven't gotten used to the typical usage
> patterns? In particular, your claim "anything that takes an `awaitable` has
> to know that it wasn't already awaited" makes me sound that you're just
> using it in an atypical way (perhaps because your model is based on other
> languages). In typical asyncio code, one does not usually take an
> awaitable, wait for it, and then return it -- one either awaits it and then
> extracts the result, or one returns it without awaiting it.
>
> On Tue, Dec 15, 2015 at 11:56 AM, Roy Williams wrote:
>
>> Howdy,
>>
>> I'm experimenting with async/await in Python 3, and one very surprising
>> behavior has been what happens when calling `await` twice on an Awaitable.
>> In C#, Hack/HHVM, and the new async/await spec in Ecmascript 7. In Python,
>> calling `await` multiple times results in all future results getting back
>> `None`. Here's a small example program:
>>
>>
>> async def echo_hi():
>> result = ''
>> echo_proc = await asyncio.create_subprocess_exec(
>> 'echo', 'hello', 'world',
>> stdout=asyncio.subprocess.PIPE,
>> stderr=asyncio.subprocess.DEVNULL)
>> result = await echo_proc.stdout.read()
>> await echo_proc.wait()
>> return result
>>
>> async def await_twice(awaitable):
>> print('first time is {}'.format(await awaitable))
>> print('second time is {}'.format(await awaitable))
>>
>> loop = asyncio.get_event_loop()
>> loop.run_until_complete(await_twice(echo_hi()))
>>
>> This makes writing composable APIs using async/await in Python very
>> difficult since anything that takes an `awaitable` has to know that it
>> wasn't already awaited. Also, since the behavior is radically different
>> than in the other programming languages implementing async/await it makes
>> adopting Python's flavor of async/await difficult for folks coming from a
>> language where it's already implemented.
>>
>> In C#/Hack/JS calls to `await` return a Task/AwaitableHandle/Promise that
>> can be awaited multiple times and either returns the result or throws any
>> thrown exceptions. It doesn't appear that the Awaitable class in Python
>> has a `result` or `exception` field but `asyncio.Future` does.
>>
>> Would it make sense to shift from having `await` functions return a `
>> *Future-like`* return object to returning a Future?
>>
Re: [Python-Dev] async/await behavior on multiple calls
@Kevin correct, that's the point I'd like to discuss. Most other
mainstream languages that implements async/await expose the programming
model with Tasks/Futures/Promises as opposed to coroutines PEP 492 states
'Objects with __await__ method are called Future-like objects in the rest
of this PEP.' but their behavior differs from that of Futures in this core
way. Given that most other languages have standardized around async
returning a Future as opposed to a coroutine I think it's worth exploring
why Python differs.
There's a lot of benefits to making the programming model coroutines
without a doubt. It's absolutely brilliant that I can just call code
annotated with @asyncio.coroutine and have it just work. Code using the
old @asyncio.coroutine/yield from syntax should absolutely stay the same.
Similarly, since ES7 async/await is backed by Promises it'll just work for
any existing code out there using Promises.
My proposal would be to automatically wrap the return value from an `async`
function or any object implementing `__await__` in a future with
`asyncio.ensure_future()`. This would allow async/await code to behave in
a similar manner to other languages implementing async/await and would
remain compatible with existing code using asyncio.
What's your thoughts?
Thanks,
Roy
On Tue, Dec 15, 2015 at 3:35 PM, Kevin Conway
wrote:
> I think there may be somewhat of a language barrier here. OP appears to be
> mixing the terms of coroutines and futures. The behavior OP describes is
> that of promised or async tasks in other languages.
>
> Consider a JS promise that has been resolved:
>
> promise.then(function (value) {...});
>
> promise.then(function (value) {...});
>
> Both of the above will execute the callback function with the resolved
> value regardless of how much earlier the promise was resolved. This is not
> entirely different from how Futures work in Python when using
> 'add_done_callback'.
>
> The code example from OP, however, is showing the behaviour of awaiting a
> coroutine twice rather than awaiting a Future twice. Both objects are
> awaitable but both exhibit different behaviour when awaited multiple times.
>
> A scenario I believe deserves a test is what happens in the asyncio
> coroutine scheduler when a promise is awaited multiple times. The current
> __await__ behaviour is to return self only when not done and then to return
> the value after resolution for each subsequent await. The Task, however,
> requires that it must be a Future emitted from the coroutine and not a
> primitive value. Awaiting a resolved future should result
>
> On Tue, Dec 15, 2015, 14:44 Guido van Rossum wrote:
>
>> Agreed. (But let's hear from the OP first.)
>>
>> On Tue, Dec 15, 2015 at 12:27 PM, Andrew Svetlov <
>> andrew.svet...@gmail.com> wrote:
>>
>>> Both Yury's suggestions sounds reasonable.
>>>
>>> On Tue, Dec 15, 2015 at 10:24 PM, Yury Selivanov
>>> wrote:
>>> > Hi Roy and Guido,
>>> >
>>> > On 2015-12-15 3:08 PM, Guido van Rossum wrote:
>>> > [..]
>>> >>
>>> >>
>>> >> I don't know how long you have been using async/await, but I wonder if
>>> >> it's possible that you just haven't gotten used to the typical usage
>>> >> patterns? In particular, your claim "anything that takes an
>>> `awaitable` has
>>> >> to know that it wasn't already awaited" makes me sound that you're
>>> just
>>> >> using it in an atypical way (perhaps because your model is based on
>>> other
>>> >> languages). In typical asyncio code, one does not usually take an
>>> awaitable,
>>> >> wait for it, and then return it -- one either awaits it and then
>>> extracts
>>> >> the result, or one returns it without awaiting it.
>>> >
>>> >
>>> > I agree. Holding a return value just so that coroutine can return it
>>> again
>>> > seems wrong to me.
>>> >
>>> > However, since coroutines are now a separate type (although they share
>>> a lot
>>> > of code with generators internally), maybe we can change them to throw
>>> an
>>> > error when they are awaited on more than one time?
>>> >
>>> > That should be better than letting them return `None`:
>>> >
>>> > coro = coroutine()
>>> > await coro
>>> > await coro # <- will raise RuntimeError
>>> >
>>> >
>>> > I'd also add a check that the coroutine isn't being awaited by more
>>> than one
>>> > coroutine simultaneously (another, completely different issue, more on
>>> which
>>> > here: https://github.com/python/asyncio/issues/288). This was fixed
>>> in
>>> > asyncio in debug mode, but ideally, we should fix this in the
>>> interpreter
>>> > core.
>>> >
>>> > Yury
>>> > ___
>>> > Python-Dev mailing list
>>> > Python-Dev@python.org
>>> > https://mail.python.org/mailman/listinfo/python-dev
>>> > Unsubscribe:
>>> >
>>> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andrew Svetlov
>>>
>> ___
>>> Python-Dev mailing list
>>> Pyt
Re: [Python-Dev] async/await behavior on multiple calls
I totally agree that async/await should not be tied to any underlying message pump/event loop. Ensuring that async/await works with existing systems like Tornado is great. As for the two options, option 1 is the expected behavior from developers coming from other languages implementing async/await which is why I found the existing behavior to be so unintuitive. To Barry and Kevin's point, this problem is exacerbated by a lack of documentation and examples that one can follow to learn about the Pythonic approach to async/await. Thanks, Roy On Tue, Dec 15, 2015 at 7:33 PM, Yury Selivanov wrote: > Roy, > > On 2015-12-15 8:29 PM, Roy Williams wrote: > [..] > >> >> My proposal would be to automatically wrap the return value from an >> `async` function or any object implementing `__await__` in a future with >> `asyncio.ensure_future()`. This would allow async/await code to behave in >> a similar manner to other languages implementing async/await and would >> remain compatible with existing code using asyncio. >> >> What's your thoughts? >> > > Other languages, such as JavaScript, have a notion of event loop > integrated on a very deep level. In Python, there is no centralized event > loop, and asyncio is just one way of implementing one. > > In asyncio, Future objects are designed to inter-operate with an event > loop (that's also true for JS Promises), which means that in order to > automatically wrap Python coroutines in Futures, we'd have to define the > event loop deep in Python core. Otherwise it's impossible to implement > 'Future.add_done_callback', since there would be nothing that calls the > callbacks on completion. > > To avoid adding a built-in event loop, PEP 492 introduced coroutines as an > abstract language concept. David Beazley, for instance, doesn't like > Futures, and his new framework 'curio' does not have them at all. > > I highly doubt that we want to add a generalized event loop in Python > core, define a generalized Future interface, and make coroutines return > it. It's simply too much work with no clear wins. > > Now, your initial email highlights another problem: > >coro = coroutine() >print(await coro) # will print the result of coroutine >await coro # prints None > > This is a bug that needs to be fixed. We have two options: > > 1. Cache the result when the coroutine object is awaited first time. > Return the cached result when the coroutine object is awaited again. > > 2. Raise an error if the coroutine object is awaited more than once. > > The (1) option would solve your problem. But it also introduces new > complexity: the GC of result will be delayed; more importantly, some users > will wonder if we cache the result or run the coroutine again. It's just > not obvious. > > The (2) option is Pythonic and simple to understand/debug, IMHO. In this > case, the best way for you to solve your initial problem, would be to have > a decorator around your tasks. The decorator should wrap coroutines with > Futures (with asyncio.ensure_future) and everything will work as you expect. > > Thanks, > Yury > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rwilliams%40lyft.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Expose environment variable for Py_Py3kWarningFlag (Issue 28288)
Howdy all, I'd love to be able to get http://bugs.python.org/issue28288 into 2.7.12 . I've found the -3 flag to be immensely useful when porting Python code to Python 3, but unfortunately it can be difficult to use with services that run python in subprocesses (like Gunicorn or Xdist with py.test). With this patch I'd be able to set the `PYTHON3WARNINGS` environment variable to ensure I get warnings everywhere. Thanks, let me know what you think, Roy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
