[Python-Dev] async/await PEP
Hello, I've just posted a new PEP about adding async/await to python-ideas. Maybe I should have posted it here instead.. Anyways, please take a look. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await PEP
Hi Martin, On 2015-04-21 4:23 AM, Martin Teichmann wrote: Hi Yury, Hi List, I do certainly like the idea of PEP 492, just some small comments: Thank you! why do we need two keywords? To me it is not necessarily intuitive when to use async and when to use await (why is it async for and not await for?), so wouldn't it be much simpler, and more symmetric, to just have one keyword? I personally prefer await for that, then it is "await def", and "await for" (and "await with", etc.). "await" is a verb - call for action, while "async" is an adjective (although non-existent in plain English). Hence "async" tells us that the statement after it is asynchronous. At least that's my reasoning about it ;) Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] async/await in Python; v2
Hi python-dev, I'm moving the discussion from python-ideas to here. The updated version of the PEP should be available shortly at https://www.python.org/dev/peps/pep-0492 and is also pasted in this email. Updates: 1. CO_ASYNC flag was renamed to CO_COROUTINE; 2. sys.set_async_wrapper() was renamed to sys.set_coroutine_wrapper(); 3. New function: sys.get_coroutine_wrapper(); 4. types.async_def() renamed to types.coroutine(); 5. New section highlighting differences from PEP 3152. 6. New AST node - AsyncFunctionDef; the proposal now is 100% backwards compatible; 7. A new section clarifying that coroutine-generators are not part of the current proposal; 8. Various small edits/typos fixes. There's is a bug tracker issue to track code review of the reference implementation (Victor Stinner is doing the review): http://bugs.python.org/issue24017 While the PEP isn't accepted, we want to make sure that the reference implementation is ready when such a decision will be made. Let's discuss some open questions: 1. Victor raised a question if we should locate coroutine() function from 'types' module to 'functools'. My opinion is that 'types' module is a better place for 'corotine()', since it adjusts the type of the passed generator. 'functools' is about 'partials', 'lru_cache' and 'wraps' kind of things. 2. I propose to disallow using of 'for..in' loops, and builtins like 'list()', 'iter()', 'next()', 'tuple()' etc on coroutines. It's possible by modifying PyObject_GetIter to raise an exception if it receives a coroutine-object. 'yield from' can also be modified to only accept coroutine objects if it is called from a generator with CO_COROUTINE flag. This will further separate coroutines from generators, making it harder to screw something up by an accident. I have a branch of reference implementation https://github.com/1st1/cpython/tree/await_noiter where this is implemented. I did not observe any performance drop. There is just one possible backwards compatibility issue here: there will be an exception if some user of asyncio actually used to iterate over generators decorated with @coroutine. But I can't imagine why would someone do that, and even if they did -- it's probably a bug or wrong usage of asyncio. That's it! I'd be happy to hear some feedback! Thanks, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015 Abstract This PEP introduces new syntax for coroutines, asynchronous ``with`` statements and ``for`` loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); * it is not possible to natively define a coroutine which has no ``yield`` or ``yield from`` statements, again requiring the use of decorators to fix potential refactoring issues; * support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification = This proposal introduces new syntax and semantics to enhance coroutine support in Python, it does not change the internal implementation of coroutines, which are still based on generators. It is strongly suggested that the reader understands how coroutin
Re: [Python-Dev] async/await in Python; v2
Hi Damien, Thanks for noticing! I pushed a fix to the peps repo. Thanks, Yury On 2015-04-21 5:20 PM, Damien George wrote: Hi Yury, In your PEP 492 draft, in the Grammar section, I think you're missing the modifications to the "flow_stmt" line. Cheers, Damien. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Greg, On 2015-04-22 2:05 AM, Greg Ewing wrote: Yury Selivanov wrote: 1. CO_ASYNC flag was renamed to CO_COROUTINE; 2. sys.set_async_wrapper() was renamed to sys.set_coroutine_wrapper(); 3. New function: sys.get_coroutine_wrapper(); 4. types.async_def() renamed to types.coroutine(); I still don't like the idea of hijacking the generic term "coroutine" and using it to mean this particular type of object. In my opinion it's OK. We propose a new dedicated syntax to define coroutines. Old approach to use generators to implement coroutines will, eventually, be obsolete. That's why, in PEP 492, we call 'async def' functions "coroutines", and the ones that are defined with generators "generator-based coroutines". You can also have "greenlets-based coroutines" and "stackless coroutines", but those aren't native Python concepts. I'm not sure if you can apply term "cofunctions" to coroutines in PEP 492. I guess we can call them "async functions". 2. I propose to disallow using of 'for..in' loops, and builtins like 'list()', 'iter()', 'next()', 'tuple()' etc on coroutines. PEP 3152 takes care of this automatically from the fact that you can't make an ordinary call to a cofunction, and cocall combines a call and a yield-from. You have to go out of your way to get hold of the underlying iterator to use in a for-loop, etc. On the one hand I like your idea to disallow calling coroutines without a special keyword (await in case of PEP 492). It has downsides, but there is some elegance in it. On the other hand, I hate the idea of grammatically requiring parentheses for 'await' expressions. That feels non-pytonic to me. I'd be happy to hear others opinion on this topic. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Guido, On 2015-04-22 11:50 AM, Guido van Rossum wrote: On Wed, Apr 22, 2015 at 8:40 AM, Yury Selivanov wrote: On the one hand I like your idea to disallow calling coroutines without a special keyword (await in case of PEP 492). It has downsides, but there is some elegance in it. On the other hand, I hate the idea of grammatically requiring parentheses for 'await' expressions. That feels non-pytonic to me. I'd be happy to hear others opinion on this topic. I'm slowly warming up to Greg's notion that you can't call a coroutine (or whatever it's called) without a special keyword. This makes a whole class of bugs obvious the moment the code is executed. OTOH I'm still struggling with what you have to do to wrap a coroutine in a Task, the way its done in asyncio by the Task() constructor, the loop.create_task() method, and the async() function (perhaps to be renamed to ensure_task() to make way for the async keyword). If we apply Greg's ideas to PEP 492 we will have the following (Greg, correct me if I'm wrong): 1. '_typeobject' struct will get a new field 'tp_await'. We can reuse 'tp_reserved' for that probably. 2. We'll hack Gen(/ceval.c?) objects to raise an error if they are called directly and have a 'CO_COROUTINE' flag. 3. Task(), create_task() and async() will be modified to call 'coro.__await__(..)' if 'coro' has a 'CO_COROUTINE' flag. 4. 'await' will require parentheses grammatically. That will make it different from 'yield' expression. For instance, I still don't know what would 'await coro(123)()' mean. 5. 'await foo(*a, **k)' will be an equivalent to 'yield from type(coro).__await__(coro, *a, **k)' 6. If we ever decide to implement coroutine-generators -- async def functions with 'await' *and* some form of 'yield' -- we'll need to reverse the rule -- allow __call__ and disallow __await__ on such objects (so that you'll be able to write 'async for item in coro_gen()' instead of 'async for item in await coro_gen()'. To be honest, that's a lot of steps and hacks to make this concept work. I think that 'set_coroutine_wrapper()' solves all these problems while keeping the grammar and implementation simpler. Moreover, it allows to add some additional features to the wrapped coroutines, such as nicer repr() in debug mode (CoroWrapper in asyncio already does that) and other runtime checks. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Rajiv, On 2015-04-22 12:53 PM, Rajiv Kumar wrote: I'd like to suggest another way around some of the issues here, with apologies if this has already been discussed sometime in the past. From the viewpoint of a Python programmer, there are two distinct reasons for wanting to suspend execution in a block of code: 1. To yield a value from an iterator, as Python generators do today. 2. To cede control to the event loop while waiting for an asynchronous task to make progress in a coroutine. As of today both of these reasons to suspend are supported by the same underlying mechanism, i.e. a "yield" at the end of the chain of "yield from"s. PEPs 492 and 3152 introduce "await" and "cocall", but at the bottom of it all there's effectively still a yield as I understand it. I think that the fact that these two concepts use the same mechanism is what leads to the issues with coroutine-generators that Greg and Yury have raised. With that in mind, would it be possible to introduce a second form of suspension to Python to specifically handle the case of ceding to the event loop? I don't know what the implementation complexity of this would be, or if it's even feasible. But roughly speaking, the syntax for this could use "await", and code would look just like it does in the PEP. The semantics of "await " would be analogous to "yield from " today, with the difference that the Task would go up the chain of "await"s to the outermost caller, which would typically be asyncio, with some modifications from its form today. Progress would be made via __anext__ instead of __next__. I think that what you propose is a great idea. However, its implementation will be far more invasive than what PEP 492 proposes. I doubt that we'll be able to make it in 3.5 if we choose this route. BUT: With my latest proposal to disallow for..in loops and iter()/list()-like builtins, the fact that coroutines are based internally on generators is just an implementation detail. There is no way users can exploit the underlying generator object. Coroutine-objects only provide 'send()' and 'throw()' methods, which they would also have with your implementation idea. This gives us freedom to consider your approach in 3.6 if we decide to add coroutine-generators. To make this work we might want to patch inspect.py to make isgenerator() family of functions to return False for coroutines/coroutine-objects. Thanks a lot for the feedback! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Andrew, On 2015-04-22 2:32 PM, Andrew Svetlov wrote: For now I can use mix asyncio.coroutines and `async def` functions, I mean I can write `await f()` inside async def to call asyncio.coroutine `f` and vise versa: I can use `yield from g()` inside asyncio.coroutine to call `async def g(): ...`. That's another good point that I forgot to add to the list. Thanks for bringing this up. If we forbid to call `async def` from regualr code how asyncio should work? I'd like to push `async def` everywhere in asyncio API where asyncio.coroutine required. You'll have to use a wrapper that will do the following: async def foo(): return 'spam' @asyncio.coroutine def bar(): what = yield from foo.__await__(foo, *args, **kwargs) # OR: what = yield from await_call(foo, *args, **kwargs) Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
On 2015-04-22 2:53 PM, Andrew Svetlov wrote: On Wed, Apr 22, 2015 at 9:45 PM, Yury Selivanov wrote: [...] If we forbid to call `async def` from regualr code how asyncio should work? I'd like to push `async def` everywhere in asyncio API where asyncio.coroutine required. You'll have to use a wrapper that will do the following: async def foo(): return 'spam' @asyncio.coroutine def bar(): what = yield from foo.__await__(foo, *args, **kwargs) # OR: what = yield from await_call(foo, *args, **kwargs) If I cannot directly use `yield from f()` with `async def f():` then almost every `yield from` inside asyncio library should be wrapped in `await_call()`. Every third-party asyncio-based library should do the same. Also I expect a performance degradation on `await_call()` calls. I think there is another way... instead of pushing GET_ITER ... YIELD_FROM opcodes, we'll need to replace GET_ITER with another one: GET_ITER_SPECIAL ... YIELD_FROM Where "GET_ITER_SPECIAL (obj)" (just a working name) would check that if the current code object has CO_COROUTINE and the object that you will yield-from has it as well, it would push to the stack the result of (obj.__await__()) Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi PJ, On 2015-04-22 3:44 PM, PJ Eby wrote: On Tue, Apr 21, 2015 at 1:26 PM, Yury Selivanov wrote: It is an error to pass a regular context manager without ``__aenter__`` and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use ``async with`` outside of a coroutine. I find this a little weird. Why not just have `with` and `for` inside a coroutine dynamically check the iterator or context manager, and either behave sync or async accordingly? Why must there be a *syntactic* difference? One of the things that we try to avoid is to have implicit places where code execution might be suspended. For that we use 'yield from' right now, and want to use 'await' with PEP 492. To have implicit context switches there is Stackless Python and greenlets, however, it's harder to reason about the code written in such a way. Having explicit 'yield from/await' is the selling point of asyncio and other frameworks that use generator-based coroutines. Hence, we want to stress that 'async with' and 'async for' do suspend the execution in their protocols. I don't want to loose control over what kind of iteration or context manager I'm using. I don't want to iterate through a cursor that doesn't do prefetching, I want to make sure that it does. This problem is solved by the PEP. Not only would this simplify the syntax, it would also allow dropping the need for `async` to be a true keyword, since functions could be defined via "def async foo():" rather than "async def foo():" ...which, incidentally, highlights one of the things that's been bothering me about all this "async foo" stuff: "async def" looks like it *defines the function* asynchronously (as with "async with" and "async for"), rather than defining an asynchronous function. ISTM it should be "def async bar():" or even "def bar() async:". If we keep 'async with', then we'll have to keep 'async def' to make it symmetric and easier to remember. But, in theory, I'd be OK with 'def async'. 'def name() async' is something that will be extremely hard to notice in the code. Also, even that seems suspect to me: if `await` looks for an __await__ method and simply returns the same object (synchronously) if the object doesn't have an await method, then your code sample that supposedly will fail if a function ceases to be a coroutine *will not actually fail*. It doesn't just do that. In the reference implementation, a single 'await o' compiles to: (o) # await arg on top of the stack GET_AWAITABLE LOAD_CONST None YIELD_FROM Where GET_AWAITABLE does the following: - If it's a coroutine-object -- return it - If it's an object with __await__, return iter(object.__await__()) - Raise a TypeError of two above steps don't return If you had a code like that: await coro() where coro is async def coro(): pass you then can certainly refactor core to: def coro(): return future # or some awaitable, please refer to PEP492 And it won't break anything. So I'm not sure I understand your remark about "*will not actually fail*". In my experience working with coroutine systems, making a system polymorphic (do something appropriate with what's given) and idempotent (don't do anything if what's wanted is already done) makes it more robust. In particular, it eliminates the issue of mixing coroutines and non-coroutines. Unfortunately, to completely eliminate the issue of reusing existing "non-coroutine" code, or of writing "coroutine" code that can be used with "non-coroutine" code, you have to use gevent-kind of libraries. To sum up: I can see the use case for a new `await` distinguished from `yield`, but I don't see the need to create new syntax for everything; ISTM that adding the new asynchronous protocols and using them on demand is sufficient. Marking a function asynchronous so it can use asynchronous iteration and context management seems reasonably useful, but I don't think it's terribly important for the type of function result. Indeed, ISTM that the built-in `object` class could just implement `__await__` as a no-op returning self, and then *all* results are trivially asynchronous results and can be awaited idempotently, so that awaiting something that has already been waited for is a no-op. I see all objects implementing __await__ returning "self" as a very error prone approach. It's totally OK to write code like that: async def coro(): return fut future = await coro() In the above example, if coro ceases to be a coroutine, 'future' will be a result of 'fut', not 'fut' itself. (Prior art: the Javascript Promise.resolve() method, which takes either
Re: [Python-Dev] async/await in Python; v2
Ludovic, On 2015-04-22 5:00 PM, Ludovic Gasc wrote: Not related, but one of my coworkers asked me if with the new syntax it will be possible to write an async decorator for coroutines. If I understand correctly new grammar in PEP, it seems to be yes, but could you confirm ? There shouldn't be any problems with writing a decorator. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
On 2015-04-22 8:35 PM, Guido van Rossum wrote: On Wed, Apr 22, 2015 at 5:12 PM, Greg Ewing wrote: Guido van Rossum wrote: On Wed, Apr 22, > OTOH I'm still struggling with what you have to do to wrap a coroutine in a Task, the way its done in asyncio by the Task() constructor, the loop.create_task() method, and the async() function That's easy. You can always use costart() to adapt a cofunction for use with something expecting a generator-based coroutine, e.g. codef my_task_func(arg): ... my_task = Task(costart(my_task_func, arg)) If you're willing to make changes, Task() et al could be made to recognise cofunctions and apply costart() where needed. Hm, that feels backwards incompatible (since currently I can write Task(my_task_func(arg)) and also a step backwards in elegance (having to pass the args separately). OTOH the benefit is that it's much harder to accidentally forget to wait for a coroutine. And maybe the backward compatibility issue is not really a problem because you have to opt in by using codef or async def. So I'm still torn. :-) Somebody would need to take a mature asyncio app and see how often this is used (i.e. how many place would require adding costart() as in the above example). Somewhere in this thread Victor Stinner wrote: """A huge part of the asyncio module is based on "yield from fut" where fut is a Future object.""" So how would we do "await fut" if await requires parentheses? I think that the problem of forgetting 'yield from' is a bit exaggerated. Yes, I myself forgot 'yield from' once or twice. But that's it, it has never happened since. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
On 2015-04-22 9:04 PM, Guido van Rossum wrote: On Wed, Apr 22, 2015 at 5:55 PM, Yury Selivanov wrote: On 2015-04-22 8:35 PM, Guido van Rossum wrote: On Wed, Apr 22, 2015 at 5:12 PM, Greg Ewing wrote: Guido van Rossum wrote: On Wed, Apr 22, > OTOH I'm still struggling with what you have to do to wrap a coroutine in a Task, the way its done in asyncio by the Task() constructor, the loop.create_task() method, and the async() function That's easy. You can always use costart() to adapt a cofunction for use with something expecting a generator-based coroutine, e.g. codef my_task_func(arg): ... my_task = Task(costart(my_task_func, arg)) If you're willing to make changes, Task() et al could be made to recognise cofunctions and apply costart() where needed. Hm, that feels backwards incompatible (since currently I can write Task(my_task_func(arg)) and also a step backwards in elegance (having to pass the args separately). OTOH the benefit is that it's much harder to accidentally forget to wait for a coroutine. And maybe the backward compatibility issue is not really a problem because you have to opt in by using codef or async def. So I'm still torn. :-) Somebody would need to take a mature asyncio app and see how often this is used (i.e. how many place would require adding costart() as in the above example). Somewhere in this thread Victor Stinner wrote: """A huge part of the asyncio module is based on "yield from fut" where fut is a Future object.""" So how would we do "await fut" if await requires parentheses? We could make Future a valid co-callable object. So you would have to write 'await fut()'? This is non-intuitive. To make Greg's proposal work it'd be a *requirement* for 'await' (enforced by the grammar!) to have '()' after it. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Greg, On 2015-04-22 7:47 PM, Greg Ewing wrote: Yury Selivanov wrote: On the other hand, I hate the idea of grammatically requiring parentheses for 'await' expressions. That feels non-pytonic to me. How is it any different from grammatically requiring parens in an ordinary function call? Nobody ever complained about that. It is different. 1. Because 'await' keyword might be at a great distance from the object you're really calling: await foo.bar.baz['spam']() +---+ Can I chain the calls: await foo()() ? or await foo().bar()? 2. Because there is no other keyword in python with similar behaviour. 3. Moreover: unless I can write 'await future' - your proposal *won't* work with a lot of existing code and patterns. It's going to be radically different from all other languages that implement 'await' too. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
On 2015-04-23 8:10 AM, Greg Ewing wrote: Andrew Svetlov wrote: From my understanding to use cofunctions I must wrap it with costart call: yield from gather(costart(coro1, a1, a2), costart(coro2), fut3) There are other places in asyncio API those accept coroutines or futures as parameters, not only Task() and async(). In a PEP 3152 aware version of asyncio, they would all know about cofunctions and what to do with them. What do you mean by that? In a PEP 3152 aware version of asyncio, it's just *not possible to write* cocall gather(coro1(1,2), coro(2,3)) you just have to use your 'costart' built-in: cocall gather(costart(coro1, 1, 2), costart(coro, 2,3)). That's all. That's PEP 3152-aware world. Somehow you think that it's OK to write cocall fut() # instead of just cocall fut() *But it's not*. A huge amount of existing code won't work. You won't be able to migrate it to new syntax easily. If you have a future object 'fut', it's not intuitive or pythonic to write 'cocall fut()'. PEP 3152 was created in pre-asyncio era, and it shows. It's just not gonna work. I know because I designed PEP 492 with a reference implementation at hand, tuning the proposal to make it backwards compatible and on the other hand to actually improve things. Your idea of syntaticaly forcing to use 'cocall' with parens is cute, but it breaks so many things and habits that it just doesn't worth it. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
On 2015-04-23 9:01 AM, Yury Selivanov wrote: cocall fut() # instead of just cocall fut() Should be: cocall fut() # instead of just cocall fut Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Wolfgang, On 2015-04-23 8:27 AM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 12:35 PM, Paul Sokolovsky wrote: Hello, On Thu, 23 Apr 2015 12:18:51 +0300 Andrew Svetlov wrote: [] 3. async with and async for Bead idea, we clutter the language even more and it is one more thing every newbie could do wrong. for x in y: result = await f() is enough, every 'async' framework lived without it over years. async for i in iterable: pass is not equal for for fut in iterable: i = yield from fut But people who used Twisted all their life don't know that! They just know that "async for" is not needed and bad. I don't think it is bad nor not needed, but the syntax is not beautiful and for the 90% not doing async stuff irritating and one more thing to learn and do right/wrong. There is no way to do things wrong in PEP 492. An object either has __aiter__ or it will be rejected by async for. An object either has __aenter__ or it will be rejected by async with. transaction = yield from connection.transaction() try: ... except: yield from transaction.rollback() else: yield from transaction.commit() is certainly more irritating than async with connection.transcation(): ... I had also a need for async loop. But there are other solutions like channels, not needing a new syntax. Also possible a function returning futures and yield in the loop with a sentinel. All this goes the road down to a producer consumer pattern. Nothing more. I know I'm a bad guy to make such comments, too bad there's a bit of truth in them, or everyone would just call me an a%$&ole right away. Generally, I already provided feedback (on asyncio list) that asyncio is based not on native Python concepts like a coroutine, but on foreign concurrency concepts like callback or Future, and a coroutine is fitted as a second-class citizen on top of that. I understand why that was done - to not leave out all those twisteds from a shiny new world of asyncio, but sometimes one may wonder if having a clear cut would've helped (compat could then have been added as a clearly separate subpackage, implemented in terms of coroutines). Now people coming from non-coroutine frameworks who were promised compatibility see "bad" things in asyncio (and related areas), and people lured by a promise of native Python framework see bad things too. This has nothing to do with people using twisted or other async frameworks like tornado. I think a coroutine should be first class. But all this should be done in a way a beginner can handle and not design this stuff for experts only. I think that most of async frameworks out there are for experts only. Precisely because of 'yield from', 'yield', inlineCallbacks, '@coroutine', channels and other stuff. PEP 492 will make it all easier. And Twisted can use its features too. If we do this we scare away new people. It doesn't scare away anyone. async/await were the most awaited features in dart and javascript. One of the most popular features in c#. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi, On 2015-04-23 3:30 AM, Wolfgang Langner wrote: Hi, most of the time I am a silent reader but in this discussion I must step in. I use twisted and async stuff a lot over years followed development of asyncio closely. First it is good to do differentiate async coroutines from generators. So everyone can see it and have this in mind and don't mix up booth. It is also easier to explain for new users. Sometimes generators blows their mind and it takes a while to get used to them. Async stuff is even harder. 1. I am fine with using something special instead of "yield" or "yield from" for this. C# "await" is ok. Everything else suggested complicates the language and makes it harder to read. 2. async def f(): is harder to read and something special also it breaks the symmetry in front (def indent). Also every existing tooling must be changed to support it. Same for def async, def f() async: I thing a decorator is enough here @coroutine def f(): is the best solution to mark something as a coroutine. You can't combine a keyword (await) with runtime decorator. Also it's harder for tools to support @coroutine / @inlineCallbacks than "async". 3. async with and async for Bead idea, we clutter the language even more and it is one more thing every newbie could do wrong. for x in y: result = await f() is enough, every 'async' framework lived without it over years. I only lived without it because I used greenlets for async for's & with's. There must be a native language concept to do these things. Same for with statement. The main use case suggested was for database stuff and this is also where most are best with defer something to a thread and keep it none async. All together it is very late in the development cycle for 3.5 to incorporate such a big change. The PEP isn't a result of some quick brainstorming. It's a result of long experience using asyncio and working around many painpoints of async programming. Best is to give all this some more time and defer it to 3.6 and some alpha releases to experiment with. There is reference implementation. asyncio is fully ported, every package for asyncio should work. You can experiment right now and find a real issue why the PEP doesn't work. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3152 and yield from Future()
Hi Victor, On 2015-04-23 4:43 AM, Victor Stinner wrote: [...] From my understanding, the PEP 3151 simply does not support asyncio.Future. Am I right? Greg wants to implement __cocall__ on futures. This way you'll be able to write cocall fut() # instead of "await fut" So you *will have to* use "()" Another problem is functions that return future: def do_something(): ... return fut With Greg's idea to call it you would do: cocall (do_something())() That means that you can't refactor your "do_something" function and make it a coroutine. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Wolfgang, On 2015-04-23 11:57 AM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 5:32 PM, Yury Selivanov wrote: Hi Wolfgang, On 2015-04-23 8:27 AM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 12:35 PM, Paul Sokolovsky wrote: Hello, On Thu, 23 Apr 2015 12:18:51 +0300 Andrew Svetlov wrote: [] 3. async with and async for Bead idea, we clutter the language even more and it is one more thing every newbie could do wrong. for x in y: result = await f() is enough, every 'async' framework lived without it over years. async for i in iterable: pass is not equal for for fut in iterable: i = yield from fut But people who used Twisted all their life don't know that! They just know that "async for" is not needed and bad. I don't think it is bad nor not needed, but the syntax is not beautiful and for the 90% not doing async stuff irritating and one more thing to learn and do right/wrong. There is no way to do things wrong in PEP 492. An object either has __aiter__ or it will be rejected by async for. An object either has __aenter__ or it will be rejected by async with. transaction = yield from connection.transaction() try: ... except: yield from transaction.rollback() else: yield from transaction.commit() is certainly more irritating than async with connection.transcation(): ... I had also a need for async loop. But there are other solutions like channels, not needing a new syntax. Also possible a function returning futures and yield in the loop with a sentinel. All this goes the road down to a producer consumer pattern. Nothing more. I know I'm a bad guy to make such comments, too bad there's a bit of truth in them, or everyone would just call me an a%$&ole right away. Generally, I already provided feedback (on asyncio list) that asyncio is based not on native Python concepts like a coroutine, but on foreign concurrency concepts like callback or Future, and a coroutine is fitted as a second-class citizen on top of that. I understand why that was done - to not leave out all those twisteds from a shiny new world of asyncio, but sometimes one may wonder if having a clear cut would've helped (compat could then have been added as a clearly separate subpackage, implemented in terms of coroutines). Now people coming from non-coroutine frameworks who were promised compatibility see "bad" things in asyncio (and related areas), and people lured by a promise of native Python framework see bad things too. This has nothing to do with people using twisted or other async frameworks like tornado. I think a coroutine should be first class. But all this should be done in a way a beginner can handle and not design this stuff for experts only. I think that most of async frameworks out there are for experts only. Precisely because of 'yield from', 'yield', inlineCallbacks, '@coroutine', channels and other stuff. PEP 492 will make it all easier. And Twisted can use its features too. Yes and it is good to make it easier. But not complicate it for others. Beginners will be confronted with all this new syntax an my feel lost. Oh I have to different for loops, one with async. Same for with statement. Absolute beginners don't write async code and http servers. And when they start doing things like that, they're not someone who can't understand 'async' and 'await'. It's not harder than '@coroutine' and 'yield from'. What is hard for even experienced users, is to understand what's the difference between 'yield from' and 'yield', and how asyncio works in the core. Why you should always use the former etc. Unfortunately I just can't agree with you here. Too much time was spent trying to explain people how to write asyncio code, and it's hard. PEP 492 will make it easier. If we do this we scare away new people. It doesn't scare away anyone. async/await were the most awaited features in dart and javascript. One of the most popular features in c#. I like it in C#. I like await for Python but I don't like async there and how to specify it. I still think a decorator is enough and no special for and with syntax. Please read the "Debugging Features" section in the PEP. It explains why decorator is not enough. Also the "Rationale" section also stresses some points. Decorator solution is "enough", but it's far from being ideal. For IDEs and linters it's harder to support. What if you write "from asyncio import coroutine as coro"? You have to analyze the code statically to reason about what "@coroutine" is. Moreover, you need to enable good IDE support for other frameworks too. async in JavaScript is for execution a whole script asynchronously used in the s
Re: [Python-Dev] PEP 3152 and yield from Future()
On 2015-04-23 11:26 AM, Victor Stinner wrote: 2015-04-23 17:22 GMT+02:00 : I can live with `cocall fut()` but the difference between `data = yield from loop.sock_recv(sock, 1024)` and `data = cocall (loop.sock_recv(sock, 1024))()` frustrates me very much. You didn't answer to my question. My question is: is it possible to implement Future.__cocall__() since yield is defined in cofunctions. If it's possible, can you please show how? (Show me the code!) I can do that. class Future: def __iter__(self): __cocall__ = __iter__ I've outlined the problem with this approach in parallel email: https://mail.python.org/pipermail/python-dev/2015-April/139456.html Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Wolfgang, On 2015-04-23 12:12 PM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 5:32 PM, Yury Selivanov wrote: Hi Wolfgang, On 2015-04-23 8:27 AM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 12:35 PM, Paul Sokolovsky wrote: Hello, On Thu, 23 Apr 2015 12:18:51 +0300 Andrew Svetlov wrote: [] 3. async with and async for Bead idea, we clutter the language even more and it is one more thing every newbie could do wrong. for x in y: result = await f() is enough, every 'async' framework lived without it over years. async for i in iterable: pass is not equal for for fut in iterable: i = yield from fut But people who used Twisted all their life don't know that! They just know that "async for" is not needed and bad. I don't think it is bad nor not needed, but the syntax is not beautiful and for the 90% not doing async stuff irritating and one more thing to learn and do right/wrong. There is no way to do things wrong in PEP 492. An object either has __aiter__ or it will be rejected by async for. An object either has __aenter__ or it will be rejected by async with. Don't mean it can be done wrong on execution or syntax level. I mean for a beginner it is not as easy an more and some will try async in some places, yes they will get an error. But there is a new possibility to get such errors if async is there for with and for statements. And the next beginner will then implement __aiter__ instead of __iter__ because he/she don't get it. Sorry, Wolfgang, but I don't get your argument. Beginners shouldn't just randomly try to use statements. There is a documentation for that. Plus we can make exception messages better. On the other side I like "await" and __await__ implementation. Symmetric good easy to explain, same with "int" and "__int__" and all others. Glad to hear that! ;) transaction = yield from connection.transaction() try: ... except: yield from transaction.rollback() else: yield from transaction.commit() is certainly more irritating than async with connection.transcation(): ... Sorry till now I use async stuff and database access and do it in an extra thread in sync mode. No performance problems and can use all good maintained database libraries. Also twisteds RDBMS (adbapi) is enough here. First I thought it is not enough or to slow but this was not the case. https://twistedmatrix.com/documents/current/core/howto/rdbms.html Here I am on line with Mike Bayer: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ It's good article, I read it. The PEP will help those who don't want to use threads and want to use coroutines. There are no fundamental reasons why coroutines are slower than threads. It will only be improved. There is a fundamental reason to avoid doing threads in python though, because it can harm performance drastically. There are some fundamental reasons why Mike will always love threads though -- we won't ever have asynchronous __getattr__, so SQLAlchemy won't be as elegant as it is in sync mode. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3152 and yield from Future()
I've updated the PEP 492: https://hg.python.org/peps/rev/352d4f907266 Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Wolfgang, On 2015-04-23 12:58 PM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 6:22 PM, Yury Selivanov wrote: Wolfgang, On 2015-04-23 12:12 PM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 5:32 PM, Yury Selivanov wrote: Hi Wolfgang, On 2015-04-23 8:27 AM, Wolfgang Langner wrote: On Thu, Apr 23, 2015 at 12:35 PM, Paul Sokolovsky wrote: Hello, On Thu, 23 Apr 2015 12:18:51 +0300 Andrew Svetlov wrote: [] 3. async with and async for Bead idea, we clutter the language even more and it is one more thing every newbie could do wrong. for x in y: result = await f() is enough, every 'async' framework lived without it over years. async for i in iterable: pass is not equal for for fut in iterable: i = yield from fut But people who used Twisted all their life don't know that! They just know that "async for" is not needed and bad. I don't think it is bad nor not needed, but the syntax is not beautiful and for the 90% not doing async stuff irritating and one more thing to learn and do right/wrong. There is no way to do things wrong in PEP 492. An object either has __aiter__ or it will be rejected by async for. An object either has __aenter__ or it will be rejected by async with. Don't mean it can be done wrong on execution or syntax level. I mean for a beginner it is not as easy an more and some will try async in some places, yes they will get an error. But there is a new possibility to get such errors if async is there for with and for statements. And the next beginner will then implement __aiter__ instead of __iter__ because he/she don't get it. Sorry, Wolfgang, but I don't get your argument. Beginners shouldn't just randomly try to use statements. There is a documentation for that. Plus we can make exception messages better. Had to coach a lot of new users to Python and some to async stuff in twisted. And what beginners not should do don't care them. ;-) They will do strange stuff and go other way's than you expected. I only like to make it as easy as possible for them. Nothing more. Less keywords, less ways to do something, best only one way to do it. Don't get me wrong, I like the PEP it is well written and covers a lot of areas about async programming. I know Python must improve in this area and has a lot of potential. But don't hesitate, give the people time to try it and mature it. If all this should be in 3.5 it is to early. The thing about this PEP is that it's build on existing concepts that were validated with asyncio. As for is it enough time to review it or not -- it's up to BDFL to decide. I'm doing my best trying to get the reference implementation reviewed and to address all questions in the PEP, hoping that it will help. I can only say that if it doesn't land in 3.5, we'll have to wait another *1.5 years*. And it's not that people will download CPython 3.6.alpha0 and start rewriting their code and playing with it. In the meanwhile, people want more good reasons to migrate to Python 3, and I strongly believe that this PEP is a great reason. Moreover, it's several months before 3.5 is released. We still will be able to slightly alter the behaviour and gather feedback during beta periods. Also we can avoid the async keyword completely and do the same as for generators. If there is an await, it is a coroutine. Unfortunately there is a problem with this approach: refactoring of code becomes much harder. That's one of the corner cases that @coroutine decorator solves; see https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Barry, On 2015-04-23 1:51 PM, Barry Warsaw wrote: On Apr 21, 2015, at 01:26 PM, Yury Selivanov wrote: The updated version of the PEP should be available shortly at https://www.python.org/dev/peps/pep-0492 and is also pasted in this email. There's a lot to like about PEP 492. I only want to mildly bikeshed a bit on the proposed new syntax. Thanks! Why "async def" and not "def async"? To my eye 'async def name()', 'async with', 'async for' look better than 'def async name()', 'with async' and 'for async'. But that's highly subjective. I also read "for async item in iter:" as "I'm iterating iter with async item". My concern is about existing tools that already know that "def" as the first non-whitespace on the line starts a function/method definition. Think of a regexp in an IDE that searches backwards from the current line to find the function its defined on. Sure, tools can be updated but it is it *necessary* to choose a syntax that breaks tools? def async useful(): seems okay to me. Probably the biggest impact on the PEP would be symmetry with asynchronous with and for. What about: with async lock: and for async data in cursor: That would also preserve at least some behavior of existing tools. Anyway, since the PEP doesn't explicitly describe this as an alternative, I want to bring it up. (I have mild concerns about __a*__ magic methods, since I think they'll be harder to visually separate, but here the PEP does describe the __async_*__ alternatives.) Anyways, I'm open to change the order of keywords if most people like it that way. Same for __async_*__ naming. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Barry, On 2015-04-23 2:12 PM, Barry Warsaw wrote: On Apr 23, 2015, at 02:02 PM, Yury Selivanov wrote: To my eye 'async def name()', 'async with', 'async for' look better than 'def async name()', 'with async' and 'for async'. But that's highly subjective. Would you be willing to add this as an alternative to the PEP, under the "Why async def" section probably? As with all such bikesheds, Guido will pick the color and we'll ooh and ahh. :) Sure! I think it's a great idea: https://hg.python.org/peps/rev/8cb4c0ab0931 https://hg.python.org/peps/rev/ec319bf4c86e Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
On 2015-04-23 3:03 AM, Greg Ewing wrote:
Yury Selivanov wrote:
- If it's an object with __await__, return iter(object.__await__())
Is the iter() really needed? Couldn't the contract of
__await__ be that it always returns an iterator?
I wrote it the wrong way. iter() isn't needed, you're right.
This is a quote from the ref implementation:
if (!PyIter_Check(await_obj)) {
PyErr_Format(PyExc_TypeError,
"__await__ must return an iterator, %.100s",
Py_TYPE(await_obj)->tp_name);
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3152 and yield from Future()
On 2015-04-23 9:05 PM, Greg Ewing wrote: Combining those last two lines into one would require some extra parenthesisation, but I don't think that's something you're going to be doing much in practice. It's a common pattern in asyncio when functions return futures. It's OK later to refactor those functions to coroutines *and* vice-versa. This is a fundamental problem for PEP 3152 approach. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3152 and yield from Future()
Greg, On 2015-04-24 4:13 AM, Greg Ewing wrote: Guido van Rossum wrote: I think this is the nail in PEP 3152's coffin. Seems more like a small tack to me. :-) I've addressed all the issues raised there in earlier posts. I'm sorry, but this is ridiculous. You haven't addressed the issues. We raise issues, saying that your PEP isn't backwards compatible and is breaking existing idioms. You say - there is a workaround for that; or that we can rewrite that and make it like that; or something else that doesn't make any sense for existing asyncio developers. Your PEP isn't backwards compatible. Period. It *will* be harder for people to get, as it *does* introduce a new calling grammar that isn't obvious for at least some people. We, asyncio developers, who write asyncio code, *don't* want to write 'cocall fut()'. I don't understand *why* I'm required to put parentheses there (besides someone just requiring me to do so, because they failed to solve some problem in backwards compatible way). You avoid confusion in one place, but you introduce it in other places. I'm sorry, but your current way of handling the discussion isn't really productive. You don't listen to arguments by Victor Stinner, Andrew Svetlov, and me. At this point, this whole PEP 3152 related discussion isn't helping anyone. Yury P.S. I'm sorry if this sounded harsh, this wasn't my intent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Guido, On 2015-04-24 1:03 PM, Guido van Rossum wrote: *3. syntactic priority of `await`* Yury, could you tweak the syntax for `await` so that we can write the most common usages without parentheses? In particular I'd like to be able to write ``` return await foo() with await foo() as bar: ... foo(await bar(), await bletch()) ``` (I don't care about `await foo() + await bar()` but it would be okay.) ``` I think this is reasonable with some tweaks of the grammar (similar to what Greg did for cocall, but without requiring call syntax at the end). I don't remember the reason why yield requires parentheses in expressions, hopefully it's not something fundamental. This has always annoyed me, so let's try to fix that for await. I'll experiment. Ditto for `__aiter__` and `__anext__`. I guess this means that the async equivalent to obtaining an iterator through `it = iter(xs)` followed by `for x over it` will have to look like `ait = await aiter(xs)` followed by `for x over ait`, where an iterator is required to have an `__aiter__` method that's an async function and returns self immediately. But what if you left out the `await` from the first call? I.e. can this work? ``` ait = aiter(xs) async for x in ait: print(x) With the current semantics that PEP 492 proposes, "await" for "aiter()" is mandatory. You have to write ait = await aiter(xs) async for x in ait: print(c) We can add some logic that will check that the iterator passed to 'async for' is not an unresolved awaitable and resolve it (instead of immediately checking if it has __anext__ method), but that will complicate the implementation. It will also introduce more than one way of doing things. I think that users will recognize "async builtins" (when we add them) by the first letter "a" and use them in "await" expressions consistently. ``` The question here is whether the object returned by aiter(xs) has an `__aiter__` method. Since it was intended to be the target of `await`, it has an `__await__` method. But that itself is mostly an alias for `__iter__`, not `__aiter__`. I guess it can be made to work, the object just has to implement a bunch of different protocols. Correct. And yes, we address this all by having iteration protocols clearly separated. *6. StopAsyncException* I'm not sure about this. The motivation given in the PEP seems to focus on the need for `__anext__` to be async. But is this really the right pattern? What if we required `ait.__anext__()` to return a future, which can either raise good old `StopIteration` or return the next value from the iteration when awaited? I'm wondering if there are a few alternatives to be explored around the async iterator protocol still. __anext__ should return an awaitable (following the terminology of the PEP), which can be a coroutine-object. I'm not sure that with semantics of PEP 479 it can actually raise StopIteration (without some hacks in genobject). I'm also trying to think forward about how we can add generator-coroutines (the ones that combine 'await' and some form of 'yield') to make writing asynchronous iterators easier. I think that reusing StopIteration on that level will be a very hard thing to understand and implement. I'll experiment with reference implementation and update the PEP. Thank you, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Victor, On 2015-04-24 5:32 PM, Victor Stinner wrote: 7. compatibility with asyncio and existing users of it The current state of the PEP makes types.coroutine() mandatory. If a generator-based coroutine is not modified with types.coroutine, await cannot be used on it. To be more concrete: asyncio coroutines not declared with @asyncio.coroutine cannot be used with await. Would it be crazy to allow waiting on a generator-based coroutine (current asyncio coroutines) without having to call types.coroutine() on it? I'd be big -1 on that. The current PEP design is all about strictly prohibiting users from calling regular generators with 'await' expression. And if a generator isn't decorated with @coroutine - then it's a regular generator for us. Maybe I just missed the purpose of disallow this. It's also possible to modify asyncio to detect at runtime when an asyncio coroutine is not decorated by @asyncio.coroutine (emit a warning or even raise an exception). I'd be +1 to add a warning to Task and other places where we accept generator-based coroutines. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Lukasz, On 2015-04-24 5:37 PM, Łukasz Langa wrote: (Though maybe we should consider `await for` and `await with`? That would have the advantage of making it easy to scan for all suspension points by searching for /await/. But being a verb it doesn't read very well.) I’m on the fence here. OT1H, I think “await for something in a_container” and “await with a_context_manager():” also read pretty well. It’s also more consistent with being the one way of finding “yield points”. OTOH, “await with a_context_manager():” suggests the entire statement is awaited on, which is not true. “async with” is more opaque in this way, it simply states “there are going to be implementation-specific awaits inside”. So it’s more consistent with “async def foo()”. This. I also think of 'await with' as I'm awaiting on the whole statement. And I don't even know how to interpret that. 6. StopAsyncException I'm not sure about this. The motivation given in the PEP seems to focus on the need for `__anext__` to be async. But is this really the right pattern? What if we required `ait.__anext__()` to return a future, which can either raise good old `StopIteration` or return the next value from the iteration when awaited? I'm wondering if there are a few alternatives to be explored around the async iterator protocol still. So are you suggesting to pass the returned value in a future? In this case the future would need to be passed to __anext__, so the Cursor example from the PEP would look like this: class Cursor: def __init__(self): self.buffer = collections.deque() def _prefetch(self): ... async def __aiter__(self): return self async def __anext__(self, fut): if not self.buffer: self.buffer = await self._prefetch() if self.buffer: fut.set_result(self.buffer.popleft()) else: fut.set_exception(StopIteration) While this is elegant, my concern is that one-future-per-iteration-step might be bad for performance. Maybe consider the following. The `async def` syntax decouples the concept of a coroutine from the implementation. While it’s still based on generators under the hood, the user no longer considers his “async function” to be a generator or conforming to the generator protocol. From the user’s perpective, it’s obvious that the return below means something different than the exception: async def __anext__(self): if not self.buffer: self.buffer = await self._prefetch() if not self.buffer: raise StopIteration return self.buffer.popleft() So, the same way we added wrapping in RuntimeErrors for generators in PEP 479, we might add transparent wrapping in a _StopAsyncIteration for CO_COROUTINE. FWIW I have to experiment more with the reference implementation, but at the moment I'm big -1 on touching StopIteration for coroutines. It's used for too many things. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Hi Guido, On 2015-04-24 1:03 PM, Guido van Rossum wrote: *3. syntactic priority of `await`* Yury, could you tweak the syntax for `await` so that we can write the most common usages without parentheses? In particular I'd like to be able to write ``` return await foo() with await foo() as bar: ... foo(await bar(), await bletch()) ``` (I don't care about `await foo() + await bar()` but it would be okay.) ``` I think this is reasonable with some tweaks of the grammar (similar to what Greg did for cocall, but without requiring call syntax at the end). I've done some experiments with grammar, and it looks like we indeed can parse await quite differently from yield. Three different options: Option #1. Parse 'await' exactly like we parse 'yield'. This how we parse 'await' in the latest version of reference implementation. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-1-patch Repo to play with: https://github.com/1st1/cpython/tree/await Syntax: await a() res = (await a()) + (await b()) res = (await (await a())) if (await a()): pass return (await a()) print((await a())) func(arg=(await a())) await a() * b() Option #2. Keep 'await_expr' in 'atom' terminal, but don't require parentheses. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-2-patch Repo to play with (parser module is broken atm): https://github.com/1st1/cpython/tree/await_noparens Syntax: await a() res = (await a()) + await b() # w/o parens (a() + await b()) res = await await a() if await a(): pass return await a() print(await a()) func(arg=await a()) await a() * b() Option #3. Create a new terminal for await expression between 'atom' and 'power'. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-3-patch Repo to play with (parser module is broken atm): https://github.com/1st1/cpython/tree/await_noparens2 Syntax: await a() res = await a() + await b() res = await (await a()) # invalid syntax w/o parens if await a(): pass return await a() print(await a()) func(arg=await a()) await (a() * b()) # w/o parens '(await a() * b()) I think that Option #3 is a clear winner. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await in Python; v2
Hi Arnaud, On 2015-04-25 4:47 PM, Arnaud Delobelle wrote: On Tue, 21 Apr 2015 at 18:27 Yury Selivanov wrote: Hi python-dev, I'm moving the discussion from python-ideas to here. The updated version of the PEP should be available shortly at https://www.python.org/dev/peps/pep-0492 and is also pasted in this email. Hi Yury, Having read this very interesting PEP I would like to make two remarks. I apologise in advance if they are points which have already been discussed. 1. About the 'async for' construct. Each iteration will create a new coroutine object (the one returned by Cursor.__anext__()) and it seems to me that it can be wasteful. In the example given of an 'aiterable' Cursor class, probably a large number of rows will fill the cursor buffer in one call of cursor._prefetch(). However each row that is iterated over will involve the creation execution of a new coroutine object. It seems to me that what is desirable in that case is that all the buffered rows will be iterated over as in a plain for loop. I agree that creating a new coroutine object is a little bit wasteful. However, the proposed iteration protocol was designed to: 1. Resemble already existing __iter__/__next__/StopIteration protocol; 2. Pave the road to introduce coroutine-generators in the future. We could, in theory, design the protocol to make __anext__ awaitable return a regular iterators (and raise StopAsyncIteration at the end) to make things more efficient, but that would complicate the protocol tremendously, and make it very hard to program and debug. My opinion is that this has to be addressed in 3.6 with coroutine-generators if there is enough interest from Python users. 2. I think the semantics of the new coroutine objects could be defined more clearly in the PEP. Of course they are pretty obvious when you know that the coroutines are meant to replace asyncio.coroutine as described in [1]. I understand that this PEP is mostly for the benefit of asyncio, hence mainly of interest of people who know it. However I think it would be good for it to be more self-contained. I have often read a PEP as an introduction to a new feature of Python. I feel that if I was not familiar with yield from and asyncio I would not be able to understand this PEP, even though potentially one could use the new constructs without knowing anything about them. I agree. I plan to update the PEP with some new semantics (prohibit passing coroutine-objects to iter(), tuple() and other builtins, as well as using them in 'for .. in coro()' loops). I'll add a section with a more detailed explanation of coroutine-objects. Best, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. Yes, everything that the PEP proposes can be done without new syntax. That's how people use asyncio right now, with only what we have in 3.4. But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. https://www.python.org/dev/peps/pep-0492/#rationale-and-goals https://www.python.org/dev/peps/pep-0492/#debugging-features https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names 3. "await". "await" is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. It can't be replaced by a function. Only if you use greenlets or Stackless Python. 4. Asynchronous with statement. The PEP lists the equivalent as "with (yield from xxx)" which doesn't seem so bad. There is no equivalent to 'async with'. "with (yield from xxx)" only allows you to suspend execution in __enter__ (and it's not actually in __enter__, but in a coroutine that returns a context manager). https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with see "New Syntax" section to see what 'async with' is equivalent too. Please don't add unnecessary new syntax. It is necessary. Perhaps you haven't spent a lot of time maintaining huge code-bases developed with frameworks like asyncio, so I understand why it does look unnecessary to you. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 2015-04-26 5:48 PM, Mark Shannon wrote: On 26/04/15 21:40, Yury Selivanov wrote: Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: I did read the PEP. I do think that clarifying the distinction between coroutines and 'normal' generators is a good idea. Adding stuff to the standard library to help is fine. I just don't think that any new syntax is necessary. Well, unfortunately, I can't explain why the new syntax is necessary better than it is already explained in the PEP, sorry. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. Yes, everything that the PEP proposes can be done without new syntax. That's how people use asyncio right now, with only what we have in 3.4. But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. I don't see why you can't do transactions using a 'with' statement. Because you can't use 'yield from' in __exit__. And allowing it there isn't a very good idea. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. https://www.python.org/dev/peps/pep-0492/#rationale-and-goals states that """ it is not possible to natively define a coroutine which has no yield or yield from statement """ which is just not true. It *is* true. Please note the word "natively". See coroutine decorator from asyncio: https://github.com/python/cpython/blob/master/Lib/asyncio/coroutines.py#L130 To turn a regular function into a coroutine you have three options: 1. wrap the function; 2. use "if 0: yield" terrible hack; 3. flip CO_GENERATOR flag for the code object (which is CPython implementation detail); also terrible hack. @coroutine decorator is great because we can use asyncio even in 3.3. But it's a) easy to forget b) hard to statically analyze c) hard to explain why functions decorated with it must only contain 'yield from' instead of 'yield' d) few more reasons listed in the PEP https://www.python.org/dev/peps/pep-0492/#debugging-features Requires the addition of the CO_COROUTINE flag, not any new keywords. True. But the importance of new keywords is covered in other sections. https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword Seems to be repeating the above. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements Which seems back to front. The argument is that existing syntax constructs cannot be made to work with asynchronous objects. Why not make the asynchronous objects work with the existing syntax? Because a) it's not possible; b) the point is to make suspend points visible in the code. That's one of the key principles of asyncio and other frameworks. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names The argument here relies on the validity of the previous points. 3. "await". "await" is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. It can't be replaced by a function. Only if you use greenlets or Stackless Python. Why not? The implementation of await is here: https://github.com/python/cpython/compare/master...1st1:await#diff-23c87bfada1d01335a3019b9321502a0R642 which clearly could be made into a function. Implementation of 'await' requires YIELD_FROM opcode. As far as I know functions in Python can't push opcodes to the eval loop while running. The only way to do what you propose is to use greenlets or merge stackless into cpython. 4. Asynchronous with statement. The PEP lists the equivalent as "with (yield from xxx)" which doesn't seem so bad. There is no equivalent to 'async with'. "with (yield from xxx)" only allows you to suspend execution in __enter__ (and it's not actually in __enter__, but in a coroutine that returns a context manager). https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with see "New Syntax" section to see what 'async with' is equivalent to
Re: [Python-Dev] PEP 492: No new syntax is required
Brett, On 2015-04-26 6:09 PM, Brett Cannon wrote: How is it a burden for people porting Python 2 code? Because they won't get to name anything 'async' just like anyone supporting older Python 3 versions? Otherwise I don't see how it is of any consequence to people maintaining 2/3 code as it will just be another thing they can't use until they drop Python 2 support. Agree. Also, the PEP states that we'll either keep a modified tokenizer or use __future__ imports till at least 3.7. It's *3 or more years*. And if necessary, we can wait till 3.8 (python 2 won't be supported at that point). Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Paul, On 2015-04-26 6:25 PM, Paul Sokolovsky wrote: Ok, so here're 3 points this link gives, with my concerns/questions: >An alternative idea about new asynchronous iterators and context >managers was to reuse existing magic methods, by adding an async >keyword to their declarations: >[But:] > - it would not be possible to create an object that works in both >with and async with statements; Yes, and I would say, for good. Behavior of sync and async code is different enough to warrant separate classes (if not libraries!) to implement each paradigm. What if, in otherwise async code, someone will miss "async" in "async with" and call sync version of context manager? So, losing ability stated above isn't big practical loss. That's debatable. It might be useful to create hybrid objects that can work in 'with (yield from lock)' and 'async with lock' statements. >- it would look confusing Sorry, "async def __enter__" doesn't look more confusing than "__aenter__" (vs "__enter__"). I'll update the PEP. The argument shouldn't be that it's confusing, the argument is that __aenter__ returns an 'awaitable', which is either a coroutine-object or a future. You can't reuse __enter__, because you'd break backwards compatibility -- it's perfectly normal for context managers in python to return any object from their __enter__. If we assign some special meaning to futures -- we'll break existing code. >and would require some implicit magic behind the scenes in the >interpreter; Interpreter already does a lot of "implicit magic". Can you please elaborate what exactly would need to be done? Async with implies using YIELD_FROM opcodes. If you want to make regular with statements compatible you'd need to add a lot of opcodes that will at runtime make a decision whether to use YIELD_FROM or not. The other point is that you can't just randomly start using YIELD_FROM, you can only do so from generators/coroutines. >one of the main points of this proposal is to make coroutines as >simple and foolproof as possible. And it's possible to agree that "to separate notions, create a dichotomy" is a simple principle on its own. But just taking bunch of stuff - special methods, exceptions - and duplicating it is a bloat a violates another principle - DRY. I'd say that a lot of terrible mistakes has happened in software development precisely because someone followed DRY religiously. Following it here will break backwards compatibility and/or make it harder to understand what is actually going on with your code. You argue that this will make coroutine writing simple. But coroutines are part of the language, and duplicating notions makes language more complex/complicated. Again, it makes reasoning about your code simpler. That what matters. Anyways, I really doubt that you can convince anyone to reuse existing dunder methods for async stuff. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Paul, On 2015-04-26 7:32 PM, Paul Sokolovsky wrote: Hello, On Sun, 26 Apr 2015 18:49:43 -0400 Yury Selivanov wrote: [] - it would look confusing Sorry, "async def __enter__" doesn't look more confusing than "__aenter__" (vs "__enter__"). I'll update the PEP. The argument shouldn't be that it's confusing, the argument is that __aenter__ returns an 'awaitable', which is either a coroutine-object or a future. You can't reuse __enter__, because you'd break backwards compatibility -- it's perfectly normal for context managers in python to return any object from their __enter__. If we assign some special meaning to futures -- we'll break existing code. So, again to make sure I (and hopefully other folks) understand it right. You say "it's perfectly normal for context managers in python to return any object from their __enter__". That's true, but we talk about async context managers. There're no such at all, they yet need to be written. And whoever writes them, would need to return from __enter__ awaitable, because that's the requirement for an async context manager, and it is error to return something else. Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. It's to make sure that it's impossible to accidentally use existing regular context manager that returns a future object from its __enter__ / __exit__ (nobody prohibits you to return a future object from __exit__, although it's pointless) in an 'async with' block. I really don't understand the desire to reuse existing magic methods. Even if it was decided to reuse them, it wouldn't even simplify the implementation in CPython; the code there is already DRY (I don't re-implement opcodes for 'with' statement; I reuse them). Then your implementation will announce that "async context manager lacks __aenter__", whereas "my" approach would announce "Async's manager __enter__ did not return awaitable value". Again, is that the distinction you're shooting for, or do I miss something? [] Anyways, I really doubt that you can convince anyone to reuse existing dunder methods for async stuff. Yeah, but it would be nice to understand why "everyone" and "so easily" agrees to them, after pretty thorough discussion of other aspects. NP :) FWIW, I wasn't trying to dodge the question, but rather stressing that the DRY argument is weak. And thanks for pointing out that this isn't covered in the PEP in enough detail. I'll update the PEP soon. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Paul, On 2015-04-26 8:17 PM, Paul Sokolovsky wrote: Hello, On Sun, 26 Apr 2015 19:45:30 -0400 Yury Selivanov wrote: [] Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. It's to make sure that it's impossible to accidentally use existing regular context manager that returns a future object from its __enter__ / __exit__ (nobody prohibits you to return a future object from __exit__, although it's pointless) in an 'async with' block. I see, so it's just to close the final loophole, unlikely to be hit in real life (unless you can say that there're cases of doing just that in existing asyncio libs). Well, given that Greg Ewing wanted even stricter error-proofness, and you rejected it as such strict as to disallow useful behavior, I've just got to trust you that in this case, you're as strict as needed. Well, backwards compatibility is something that better be preserved. Especially with things like context managers. Things with __aiter__/__iter__ are also different. It's easier for everyone to just clearly separate the protocols to avoid all kind of risks. I really don't understand the desire to reuse existing magic methods. Even if it was decided to reuse them, it wouldn't even simplify the implementation in CPython; the code there is already DRY (I don't re-implement opcodes for 'with' statement; I reuse them). Well, there're 3 levels of this stuff: 1. How "mere" people write their code - everyone would use async def and await, this should be bullet- (and fool-) proof. 2. How "library" code is written - async iterators won't be written by everyone, and only few will write async context managers; it's fair to expect that people doing these know what they do and don't do stupid mistakes. 3. How it all is coded in particular Python implementation. It's clear that __enter__ vs __aenter__ distinction is 1st kind of issue in your list. It is. As for 3rd point, I'd like to remind that CPython is only one Python implementation. And with my MicroPython hat on, I'd like to know if (some of) these new features are "bloat" or "worthy" for the space constraints we have. OT: MicroPython is an amazing project. Kudos for doing it. I really hope that addition of few new magic methods won't make it too hard for you guys to implement PEP 492 in MicroPython one day. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Hi Greg,
I don't want this: "await a() * b()" to be parsed, it's not meaningful.
Likely you'll see "await await a()" only once in your life, so I'm fine
to use parens for it (moreover, I think it reads better with parens)
Yury
On 2015-04-27 8:52 AM, Greg Ewing wrote:
Yury Selivanov wrote:
I've done some experiments with grammar, and it looks like
we indeed can parse await quite differently from yield. Three
different options:
You don't seem to have tried what I suggested, which is
to make 'await' a unary operator with the same precedence
as '-', i.e. replace
factor: ('+'|'-'|'~') factor | power
with
factor: ('+'|'-'|'~'|'await') factor | power
That would allow
await a()
res = await a() + await b()
res = await await a()
if await a(): pass
return await a()
print(await a())
func(arg=await a())
await a() * b()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; v3
Hi python-dev, Another round of updates. Reference implementation has been updated: https://github.com/1st1/cpython/tree/await (includes all things from the below summary of updates + tests). Summary: 1. "PyTypeObject.tp_await" slot. Replaces "tp_reserved". This is to enable implementation of Futures with C API. Must return an iterator if implemented. 2. New grammar for "await" expressions, see 'Syntax of "await" expression' section 3. inspect.iscoroutine() and inspect.iscoroutineobjects() functions. 4. Full separation of coroutines and generators. This is a big one; let's discuss. a) Coroutine objects raise TypeError (is NotImplementedError better?) in their __iter__ and __next__. Therefore it's not not possible to pass them to iter(), tuple(), next() and other similar functions that work with iterables. b) Because of (a), for..in iteration also does not work on coroutines anymore. c) 'yield from' only accept coroutine objects from generators decorated with 'types.coroutine'. That means that existing asyncio generator-based coroutines will happily yield from both coroutines and generators. *But* every generator-based coroutine *must* be decorated with `asyncio.coroutine()`. This is potentially a backwards incompatible change. d) inspect.isgenerator() and inspect.isgeneratorfunction() return `False` for coroutine objects & coroutine functions. e) Should we add a coroutine ABC (for cython etc)? I, personally, think this is highly necessary. First, separation of coroutines from generators is extremely important. One day there won't be generator-based coroutines, and we want to avoid any kind of confusion. Second, we only can do this in 3.5. This kind of semantics change won't be ever possible. asyncio recommends using @coroutine decorator, and most projects that I've seen do use it. Also there is no reason for people to use iter() and next() functions on coroutines when writing asyncio code. I doubt that this will cause serious backwards compatibility problems (asyncio also has provisional status). Thank you, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015 Abstract This PEP introduces new syntax for coroutines, asynchronous ``with`` statements and ``for`` loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); * it is not possible to natively define a coroutine which has no ``yield`` or ``yield from`` statements, again requiring the use of decorators to fix potential refactoring issues; * support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification = This proposal introduces new syntax and semantics to enhance coroutine support in Python, it does not change the internal implementation of coroutines, which are still based on generators. It is strongly suggested that the reader understands how coroutines are implemented in Python (PEP 342 and PEP 380). It is also recommended to read PEP 3156 (asyncio framework) and PEP 3152 (Cofunctions). From this point in this document we use the word *coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. New Coroutine Declaration Syntax -
Re: [Python-Dev] PEP 492: async/await in Python; v3
Hi Walter, On 2015-04-28 10:23 AM, Walter Dörwald wrote: Key properties of coroutines: * ``async def`` functions are always coroutines, even if they do not contain ``await`` expressions. * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. Does this mean it's not possible to implement an async version of os.walk() if we had an async version of os.listdir()? I.e. for async code we're back to implementing iterators "by hand" instead of using generators for it. For now yes. Unfortunately, we don't have time to implement coroutine-generators properly in 3.5. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Hi Stefan, On 2015-04-28 1:43 AM, Stefan Behnel wrote: Should a Generator then inherit from both Iterator and Coroutine, or would that counter your intention to separate coroutines from generators as a concept? I mean, they do share the same interface ... Them sharing the same interface depends on how the discussion goes :) But all in all, I think that it should be totally separate classes, even if they share some methods. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Ethan, On 2015-04-28 11:29 AM, Ethan Furman wrote: On 04/28, Yury Selivanov wrote: On 2015-04-28 1:43 AM, Stefan Behnel wrote: Should a Generator then inherit from both Iterator and Coroutine, or would that counter your intention to separate coroutines from generators as a concept? I mean, they do share the same interface ... Them sharing the same interface depends on how the discussion goes :) But all in all, I think that it should be totally separate classes, even if they share some methods. For those of us who like to meddle, would it be possible to create an object that supports __iter__, __next__, __aiter__, and __anext__? Sure, there is nothing that could prevent you from doing that. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issues with PEP 482 (1)
Mark, I'm sorry but you have to view the proposal as a whole. Discussing it point by point in isolation doesn't make any sense, as with any complex subject. Thanks, Yury On 2015-04-28 2:44 PM, Mark Shannon wrote: Hi, I still think that there are several issues that need addressing with PEP 492. This time, one issue at a time :) "async" The "Rationale and Goals" of PEP 492 states that PEP 380 has 3 shortcomings. The second of which is: """It is not possible to natively define a coroutine which has no yield or yield from statements.""" This is incorrect, although what is meant by 'natively' is unclear. A coroutine without a yield statement can be defined simply and concisely, thus: @coroutine def f(): return 1 This is only a few character longer than the proposed new syntax, perfectly explicit and requires no modification the language whatsoever. A pure-python definition of the "coroutine" decorator is given below. So could the "Rationale and Goals" be correctly accordingly, please. Also, either the "async def" syntax should be dropped, or a new justification is required. Cheers, Mark. #coroutine.py from types import FunctionType, CodeType CO_COROUTINE = 0x0080 CO_GENERATOR = 0x0020 def coroutine(f): 'Converts a function to a generator function' old_code = f.__code__ new_code = CodeType( old_code.co_argcount, old_code.co_kwonlyargcount, old_code.co_nlocals, old_code.co_stacksize, old_code.co_flags | CO_GENERATOR | CO_COROUTINE, old_code.co_code, old_code.co_consts, old_code.co_names, old_code.co_varnames, old_code.co_filename, old_code.co_name, old_code.co_firstlineno, old_code.co_lnotab, old_code.co_freevars, old_code.co_cellvars) return FunctionType(new_code, f.__globals__) P.S. The reverse of this decorator, which unsets the flags, converts a generator function into a normal function. :? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Hi Guido,
Thank you for a very detailed review. Comments below:
On 2015-04-28 5:49 PM, Guido van Rossum wrote:
Inline comments below...
On Mon, Apr 27, 2015 at 8:07 PM, Yury Selivanov
wrote:
Hi python-dev,
Another round of updates. Reference implementation
has been updated: https://github.com/1st1/cpython/tree/await
(includes all things from the below summary of updates +
tests).
Summary:
1. "PyTypeObject.tp_await" slot. Replaces "tp_reserved".
This is to enable implementation of Futures with C API.
Must return an iterator if implemented.
That's fine (though I didn't follow this closely).
My main question here is it OK to reuse 'tp_reserved'
(former tp_compare)?
I had to remove this check:
https://github.com/1st1/cpython/commit/4be6d0a77688b63b917ad88f09d446ac3b7e2ce9#diff-c3cf251f16d5a03a9e7d4639f2d6f998L4906
On the other hand I think that it's a slightly better
solution than adding a new slot.
2. New grammar for "await" expressions, see
'Syntax of "await" expression' section
I like it.
Great!
The current grammar requires parentheses for consequent
await expressions:
await (await coro())
I can change this (in theory), but I kind of like the parens
in this case -- better readability. And it'll be a very
rare case.
3. inspect.iscoroutine() and inspect.iscoroutineobjects()
functions.
What's the use case for these? I wonder if it makes more sense to have a
check for a generalized awaitable rather than specifically a coroutine.
It's important to at least have 'iscoroutine' -- to check that
the object is a coroutine function. A typical use-case would be
a web framework that lets you to bind coroutines to specific
http methods/paths:
@http.get('/spam')
async def handle_spam(request):
...
'http.get' decorator will need a way to raise an error if it's
applied to a regular function (while the code is being imported,
not in runtime).
The idea here is to cover all kinds of python objects in
inspect module, it's Python's reflection API.
The other thing is that it's easy to implement this function
for CPython: just check for CO_COROUTINE flag. For other
Python implementations it might be a different story.
(More arguments for isawaitable() below)
4. Full separation of coroutines and generators.
This is a big one; let's discuss.
a) Coroutine objects raise TypeError (is NotImplementedError
better?) in their __iter__ and __next__. Therefore it's
not not possible to pass them to iter(), tuple(), next() and
other similar functions that work with iterables.
I think it should be TypeError -- what you *really* want is not to define
these methods at all but given the implementation tactic for coroutines
that may not be possible, so the nearest approximation is TypeError. (Also,
NotImplementedError is typically to indicate that a subclass should
implement it.)
Agree.
b) Because of (a), for..in iteration also does not work
on coroutines anymore.
Sounds good.
c) 'yield from' only accept coroutine objects from
generators decorated with 'types.coroutine'. That means
that existing asyncio generator-based coroutines will
happily yield from both coroutines and generators.
*But* every generator-based coroutine *must* be
decorated with `asyncio.coroutine()`. This is
potentially a backwards incompatible change.
See below. I worry about backward compatibility. A lot. Are you saying
that asycio-based code that doesn't use @coroutine will break in 3.5?
I'll experiment with replacing (c) with a warning.
We can disable __iter__ and __next__ for coroutines, but
allow to use 'yield from' on them. Would it be a better
approach?
d) inspect.isgenerator() and inspect.isgeneratorfunction()
return `False` for coroutine objects & coroutine functions.
Makes sense.
(d) can also break something (hypothetically). I'm not
sure why would someone use isgenerator() and
isgeneratorfunction() on generator-based coroutines in
code based on asyncio, but there is a chance that
someone did (it should be trivial to fix the code).
Same for iter() and next(). The chance is slim, but we
may break some obscure code.
Are you OK with this?
e) Should we add a coroutine ABC (for cython etc)?
I, personally, think this is highly necessary. First,
separation of coroutines from generators is extremely
important. One day there won't be generator-based
coroutines, and we want to avoid any kind of confusion.
Second, we only can do this in 3.5. This kind of
semantics change won't be ever possible.
Sounds like Stefan agrees. Are you aware of
http://bugs.python.org/issue24018 (Generator ABC)?
Yes, I saw the issue. I'll review it in more detail
before thinking about Coroutine ABC for the next PEP
update.
asyncio recommends using @coroutine decorator, an
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Looking at the grammar -- the only downside of the current approach is that
you can't do 'await await fut'. I still think that it reads better with
parens. If we put 'await' to 'factor' terminal we would allow
await -fut # await (-fut)
I think I something like
power: atom_expr ['**' factor]
atom_expr: [AWAIT] atom_expr | atom_trailer
atom_trailer: atom trailer*
will fix 'await await' situation, but does it really need to be fixed?
Yury
On 2015-04-27 9:44 AM, Yury Selivanov wrote:
Hi Greg,
I don't want this: "await a() * b()" to be parsed, it's not meaningful.
Likely you'll see "await await a()" only once in your life, so I'm
fine to use parens for it (moreover, I think it reads better with parens)
Yury
On 2015-04-27 8:52 AM, Greg Ewing wrote:
Yury Selivanov wrote:
I've done some experiments with grammar, and it looks like
we indeed can parse await quite differently from yield. Three
different options:
You don't seem to have tried what I suggested, which is
to make 'await' a unary operator with the same precedence
as '-', i.e. replace
factor: ('+'|'-'|'~') factor | power
with
factor: ('+'|'-'|'~'|'await') factor | power
That would allow
await a()
res = await a() + await b()
res = await await a()
if await a(): pass
return await a()
print(await a())
func(arg=await a())
await a() * b()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Hi Stephen, Thanks a lot for the feedback and suggestions. I'll apply them to the PEP. On 2015-04-28 11:03 PM, Stephen J. Turnbull wrote: Literary critic here. In section "Specification" > It is strongly suggested that the reader understands how coroutines > are implemented in Python (PEP 342 and PEP 380). It is also > recommended to read PEP 3156 (asyncio framework) and PEP 3152 > (Cofunctions). The usual phrasing of "strongly suggested" in specifications is "presumes knowledge". Some people think "strongly suggest ing" is presumptuous and condescending, YMMV. Also, the relationship to PEP 3152 should be mentioned IMO. I propose: This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the "Cofunctions" proposal (PEP 3152, now rejected in favor of this specification). Your wording is 100% better and it's time to mention PEP 3152 too. I'm not entirely happy with my phrasing, because there are at least four more or less different concepts that might claim the bare word "coroutine": - this specification - the implementation of this specification - the syntax used to define coroutines via PEPs 342 and 380 - the semantics of PEP 342/380 coroutines In both your original and my rephrasing, the use of "coroutine" violates your convention that it refers to the PEP's proposed syntax for coroutines. Instead it refers to the semantics of coroutines implemented via PEP 342/380. This is probably the same concern that motivated Guido's suggestion to use "native coroutines" for the PEP 492 syntax (but I'm not Dutch, so maybe they're not the same :-). I feel this is a real hindrance to understanding for someone coming to the PEP for the first time. You know which meaning of coroutine you mean, but the new reader needs to think hard enough to disambiguate every time the word occurs. If people agree with me, I could go through the PEP and revise mentions of "coroutine" in "disambiguated" style. I also like Guido's suggestion to use "native coroutine" term. I'll update the PEP (I have several branches of it in the repo that I need to merge before the rename). In section "Comprehensions": > For the sake of restricting the broadness of this PEP there is no > new syntax for asynchronous comprehensions. This should be > considered in a separate PEP, if there is a strong demand for this > feature. Don't invite trouble. How about: Syntax for asynchronous comprehensions could be provided, but this construct is outside of the scope of this PEP. In section "Async lambdas" > Lambda coroutines are not part of this proposal. In this proposal they > would look like ``async lambda(parameters): expression``. Unless there > is a strong demand to have them as part of this proposal, it is > recommended to consider them later in a separate PEP. Same recommendation as for "Comprehensions". I wouldn't mention the tentative syntax, it is both obvious and inviting to trouble. Agree. Do you think it'd be better to combine comprehensions and async lambdas in one section? > Acknowledgments > === > > I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew > Svetlov, and Łukasz Langa for their initial feedback. A partial list of commentators I've found to be notable, YMMV: Sure! I was going to add everybody after the PEP is accepted/rejected/postponed. Greg Ewing for PEP 3152 and his Loyal Opposition to this PEP. Mark Shannon's comments have led to substantial clarifications of motivation for syntax, at least in my mind. Paul Sokolovsky for information about the MicroPython implementation. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
On 2015-04-28 11:59 PM, Greg wrote: On 29/04/2015 9:49 a.m., Guido van Rossum wrote: c) 'yield from' only accept coroutine objects from generators decorated with 'types.coroutine'. That means that existing asyncio generator-based coroutines will happily yield from both coroutines and generators. *But* every generator-based coroutine *must* be decorated with `asyncio.coroutine()`. This is potentially a backwards incompatible change. See below. I worry about backward compatibility. A lot. Are you saying that asycio-based code that doesn't use @coroutine will break in 3.5? That seems unavoidable if the goal is for 'await' to only work on generators that are intended to implement coroutines, and not on generators that are intended to implement iterators. Because there's no way to tell them apart without marking them in some way. Not sure what you mean by "unavoidable". Before the last revision of the PEP it was perfectly fine to use generators in 'yield from' in generator-based coroutines: @asyncio.coroutine def foo(): yield from gen() and yet you couldn't do the same with 'await' (as it has a special opcode instead of GET_ITER that can validate what you're awaiting). With the new version of the PEP - 'yield from' in foo() would raise a TypeError. If we change it to a RuntimeWarning then we're safe in terms of backwards compatibility. I just want to see how exactly warnings will work (i.e. will they occur multiple times at the same 'yield from' expression, etc) Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Guido,
I found a solution how to disable 'yield from',
iter()/tuple() and 'for..in' on native coroutines
with 100% backwards compatibility.
The idea is to add one more code object flag:
CO_NATIVE_COROUTINE, which will be applied, along with
CO_COROUTINE to all 'async def' functions.
This way:
1. old generator-based coroutines from asyncio
are awaitable, because of CO_COROUTINE flag (that
asyncio.coroutine decorator will set with
'types.coroutine').
2. new 'async def' functions are awaitable
because of CO_COROUTINE flag.
3. GenObject __iter__ and __next__ raise error
*only* if it has CO_NATIVE_COROUTINE flag. So
iter(), next(), for..in aren't supported only
for 'async def' functions (but will work ok
on asyncio generator-based coroutines)
4. 'yield from' *only* raises an error if it yields a
*coroutine with a CO_NATIVE_COROUTINE*
from a regular generator.
Thanks,
Yury
On 2015-04-28 7:26 PM, Yury Selivanov wrote:
Hi Guido,
Thank you for a very detailed review. Comments below:
On 2015-04-28 5:49 PM, Guido van Rossum wrote:
Inline comments below...
On Mon, Apr 27, 2015 at 8:07 PM, Yury Selivanov
wrote:
Hi python-dev,
Another round of updates. Reference implementation
has been updated: https://github.com/1st1/cpython/tree/await
(includes all things from the below summary of updates +
tests).
Summary:
1. "PyTypeObject.tp_await" slot. Replaces "tp_reserved".
This is to enable implementation of Futures with C API.
Must return an iterator if implemented.
That's fine (though I didn't follow this closely).
My main question here is it OK to reuse 'tp_reserved'
(former tp_compare)?
I had to remove this check:
https://github.com/1st1/cpython/commit/4be6d0a77688b63b917ad88f09d446ac3b7e2ce9#diff-c3cf251f16d5a03a9e7d4639f2d6f998L4906
On the other hand I think that it's a slightly better
solution than adding a new slot.
2. New grammar for "await" expressions, see
'Syntax of "await" expression' section
I like it.
Great!
The current grammar requires parentheses for consequent
await expressions:
await (await coro())
I can change this (in theory), but I kind of like the parens
in this case -- better readability. And it'll be a very
rare case.
3. inspect.iscoroutine() and inspect.iscoroutineobjects()
functions.
What's the use case for these? I wonder if it makes more sense to
have a
check for a generalized awaitable rather than specifically a coroutine.
It's important to at least have 'iscoroutine' -- to check that
the object is a coroutine function. A typical use-case would be
a web framework that lets you to bind coroutines to specific
http methods/paths:
@http.get('/spam')
async def handle_spam(request):
...
'http.get' decorator will need a way to raise an error if it's
applied to a regular function (while the code is being imported,
not in runtime).
The idea here is to cover all kinds of python objects in
inspect module, it's Python's reflection API.
The other thing is that it's easy to implement this function
for CPython: just check for CO_COROUTINE flag. For other
Python implementations it might be a different story.
(More arguments for isawaitable() below)
4. Full separation of coroutines and generators.
This is a big one; let's discuss.
a) Coroutine objects raise TypeError (is NotImplementedError
better?) in their __iter__ and __next__. Therefore it's
not not possible to pass them to iter(), tuple(), next() and
other similar functions that work with iterables.
I think it should be TypeError -- what you *really* want is not to
define
these methods at all but given the implementation tactic for coroutines
that may not be possible, so the nearest approximation is TypeError.
(Also,
NotImplementedError is typically to indicate that a subclass should
implement it.)
Agree.
b) Because of (a), for..in iteration also does not work
on coroutines anymore.
Sounds good.
c) 'yield from' only accept coroutine objects from
generators decorated with 'types.coroutine'. That means
that existing asyncio generator-based coroutines will
happily yield from both coroutines and generators.
*But* every generator-based coroutine *must* be
decorated with `asyncio.coroutine()`. This is
potentially a backwards incompatible change.
See below. I worry about backward compatibility. A lot. Are you saying
that asycio-based code that doesn't use @coroutine will break in 3.5?
I'll experiment with replacing (c) with a warning.
We can disable __iter__ and __next__ for coroutines, but
allow to use 'yield from' on them. Would it be a better
approach?
d) inspect.isgenerator() and inspect.isgeneratorfunction()
return `False` for coroutine objects & coroutine functions.
Makes sense.
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Greg,
On 2015-04-29 1:40 AM, Greg Ewing wrote:
Yury Selivanov wrote:
I don't want this: "await a() * b()" to be parsed, it's not meaningful.
Why not? If a() is a coroutine that returns a number,
why shouldn't I be able to multiply it by something?
Sorry, I thought you meant parsing "await a()*b()" as
"await (a() * b())".
I don't think your currently proposed grammar prevents
that anyway. We can have
--> '*'
--> '*'
--> '*'
--> 'await' * '*' *
--> 'await' 'a' '(' ')' '*' 'b' '(' ')'
It does, on the other hand, seem to prevent
x = - await a()
This works just fine:
https://github.com/1st1/cpython/commit/33b3cd052243cd71c064eb385c7a557eec3ced4b
Current grammar prevents this: "await -fut", and this:
"await fut ** 2" being parsed as "await (fut ** 2)
which looks perfectly sensible to me.
I don't like the idea of introducing another level
of precedence. Python already has too many of those
to keep in my brain. Being able to tell people "it's
just like unary minus" makes it easy to explain (and
therefore possibly a good idea!).
It's just like unary minus ;)
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
On 2015-04-29 5:13 AM, Greg Ewing wrote: Yury Selivanov wrote: 3. GenObject __iter__ and __next__ raise error *only* if it has CO_NATIVE_COROUTINE flag. So iter(), next(), for..in aren't supported only for 'async def' functions (but will work ok on asyncio generator-based coroutines) What about new 'async def' code called by existing code that expects to be able to use iter() or next() on the future objects it receives? 4. 'yield from' *only* raises an error if it yields a *coroutine with a CO_NATIVE_COROUTINE* from a regular generator. Won't that prevent some existing generator-based coroutines (ones not decorated with @coroutine) from calling ones implemented with 'async def'? It would. But that's not a backwards compatibility issue. Everything will work in 3.5 without a single line change. If you want to use new coroutines - use them, everything will work too. If, however, during the refactoring you've missed several generator-based coroutines *and* they are not decorated with @coroutine - then yes, you will get a runtime error. I see absolutely no problem with that. It's a small price to pay for a better design. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
On 2015-04-29 5:12 AM, Greg Ewing wrote: Yury Selivanov wrote: Looking at the grammar -- the only downside of the current approach is that you can't do 'await await fut'. I still think that it reads better with parens. If we put 'await' to 'factor' terminal we would allow await -fut # await (-fut) Is there really a need to disallow that? It would take a fairly bizarre API to make it meaningful in the first place, but in any case, it's fairly clear what order of operations is intended without the parens. Greg, if grammar can prevent this kind of mistakes - it should. I like my current approach. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Greg,
On 2015-04-29 5:12 AM, Greg Ewing wrote:
Yury Selivanov wrote:
It's important to at least have 'iscoroutine' -- to check that
the object is a coroutine function. A typical use-case would be
a web framework that lets you to bind coroutines to specific
http methods/paths:
@http.get('/spam')
async def handle_spam(request):
...
>
The other thing is that it's easy to implement this function
for CPython: just check for CO_COROUTINE flag.
But isn't that too restrictive? Any function that returns
an awaitable object would work in the above case.
It's just an example. All in all, I think that we should have
full coverage of python objects in the inspect module. There are
many possible use cases besides the one that I used -- runtime
introspection, reflection, debugging etc, where you might need
them.
One of the most frequent mistakes that people make when using
generators as coroutines is forgetting to use ``yield from``::
I think it's a mistake that a lot of beginners may make at some
point (and in this sense it's frequent). I really doubt that
once you were hit by it more than two times you would make it
again.
What about when you change an existing non-suspendable
function to make it suspendable, and have to deal with
the ripple-on effects of that? Seems to me that affects
everyone, not just beginners.
I've been using coroutines on a daily basis for 6 or 7 years now,
long before asyncio we had a coroutine-based framework at my firm
(yield + trampoline). Neither I nor my colleagues had any
problems with refactoring the code. I really try to speak from
my experience when I say that it's not that big of a problem.
Anyways, the PEP provides set_coroutine_wrapper which should solve
the problem.
3. ``yield from`` does not accept coroutine objects from plain Python
generators (*not* generator-based coroutines.)
What exactly are "coroutine objects
from plain Python generators"?)
# *Not* decorated with @coroutine
def some_algorithm_impl():
yield 1
yield from native_coroutine() # <- this is a bug
So what you really mean is "yield-from, when used inside
a function that doesn't have @coroutine applied to it,
will not accept a coroutine object", is that right? If
so, I think this part needs re-wording, because it sounded
like you meant something quite different.
I'm not sure I like this -- it seems weird that applying
a decorator to a function should affect the semantics
of something *inside* the function -- especially a piece
of built-in syntax such as 'yield from'. It's similar
to the idea of replacing 'async def' with a decorator,
which you say you're against.
This is for the transition period. We don't want to break
existing asyncio code. But we do want coroutines to be
a separate concept from generators. It doesn't make any
sense to iterate through coroutines or to yield-from them.
We can deprecate @coroutine decorator in 3.6 or 3.7 and
at some time remove it.
BTW, by "coroutine object", do you mean only objects
returned by an async def function, or any object having
an __await__ method? I think a lot of things would be
clearer if we could replace the term "coroutine object"
with "awaitable object" everywhere.
The PEP clearly separates awaitbale from coroutine objects.
- coroutine object is returned from coroutine call.
- awaitable is either a coroutine object or an object with
__await__.
list(), tuple(), iter(), next(), for..in etc. won't work
on objects with __await__ (unless they implement __iter__).
The problem I was discussing is about specifically
'yield from' and 'coroutine object'.
``yield from`` does not accept *native coroutine objects*
from regular Python generators
It's the "from" there that's confusing -- it sounds
like you're talking about where the argument to
yield-from comes from, rather than where the yield-from
expression resides. In other words, we though you were
proposing to disallow *this*:
# *Not* decorated with @coroutine
def some_algorithm_impl():
yield 1
yield from iterator_implemented_by_generator()
I hope to agree that this is a perfectly legitimate
thing to do, and should remain so?
Sure it's perfectly normal ;) I apologize for the poor
wording.
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Greg, On 2015-04-29 5:13 AM, Greg Ewing wrote: Yury Selivanov wrote: On 2015-04-28 11:59 PM, Greg wrote: On 29/04/2015 9:49 a.m., Guido van Rossum wrote: *But* every generator-based coroutine *must* be decorated with `asyncio.coroutine()`. This is potentially a backwards incompatible change. See below. I worry about backward compatibility. A lot. Are you saying that asycio-based code that doesn't use @coroutine will break in 3.5? That seems unavoidable if the goal is for 'await' to only work on generators that are intended to implement coroutines, Not sure what you mean by "unavoidable". Guido is worried about existing asyncio-based code that doesn't always decorate its generators with @coroutine. If I understand correctly, if you have @coroutine def coro1(): yield from coro2() def coro2(): yield from ... then coro1() would no longer work. In other words, some currently legitimate asyncio-based code will break under PEP 492 even if it doesn't use any PEP 492 features. What you seem to be trying to do here is catch the mistake of using a non-coroutine iterator as if it were a coroutine. By "unavoidable" I mean I can't see a way to achieve that in all possible permutations without giving up some backward compatibility. Please see my reply to Guido in other thread. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Greg, On 2015-04-29 5:12 AM, Greg Ewing wrote: Guido van Rossum wrote: On Mon, Apr 27, 2015 at 8:07 PM, Yury Selivanov mailto:yselivanov...@gmail.com>> wrote: > Why StopAsyncIteration? ''''''''''''''''''''''' I keep wanting to propose to rename this to AsyncStopIteration. +1, that seems more consistent to me too. And since PEP 479 is accepted and enabled by default for coroutines, the following example will have its ``StopIteration`` wrapped into a ``RuntimeError`` I think that's a red herring in relation to the reason for StopAsyncIteration/AsyncStopIteration being needed. The real reason is that StopIteration is already being used to signal returning a value from an async function, so it can't also be used to signal the end of an async iteration. When we start thinking about generator-coroutines (the ones that combine 'await' and 'async yield'-something), we'll have to somehow multiplex them to the existing generator object (at least that's one way to do it). StopIteration is already extremely loaded with different special meanings. [..] Does send() make sense for a native coroutine? Check PEP 380. I think the only way to access the send() argument is by using ``yield`` but that's disallowed. Or is this about send() being passed to the ``yield`` that ultimately suspends the chain of coroutines? That's made me think of something else. Suppose you want to suspend execution in an 'async def' function -- how do you do that if 'yield' is not allowed? You may need something like the suspend() primitive that I was thinking of adding to PEP 3152. We do this in asyncio with Futures. We never combine 'yield' and 'yield from' in a @coroutine. We don't need 'suspend()'. If you need suspend()-like thing in your own framework, implement an object with an __await__ method and await on it. No implicit wrapping in Futures --- There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key difference is that JavaScript "async functions" always return a Promise. While this approach has some advantages, it also implies that a new Promise object is created on each "async function" invocation. I don't see how this is different from an 'async def' function always returning an awaitable object, or a new awaitable object being created on each 'async def' function invocation. Sounds pretty much isomorphic to me. Agree. I'll try to reword that section. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Ethan, On 2015-04-29 11:29 AM, Ethan Furman wrote: Python is not lisp, and await is not a function, so parens should not be needed in the common case. Which common case you mean? Please see this table https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Also please see this: https://hg.python.org/peps/rev/d048458307b7 FWIW, 'await await fut' isn't something that you likely to see in your life; 'await -fut' is 99.999% just a bug. I'm not sure why Greg is pushing his Grammar idea so aggressively. Yury On 2015-04-29 11:33 AM, Yury Selivanov wrote: Ethan, On 2015-04-29 11:29 AM, Ethan Furman wrote: Python is not lisp, and await is not a function, so parens should not be needed in the common case. Which common case you mean? Please see this table https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Ethan, On 2015-04-29 1:25 PM, Ethan Furman wrote: cannot also just work and be the same as the parenthesized version. Because it does not make any sense. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
Hi Jim,
On 2015-04-29 1:43 PM, Jim J. Jewett wrote:
On Tue Apr 28 23:49:56 CEST 2015, Guido van Rossum quoted PEP 492:
Rationale and Goals
===
Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:
* it is easy to confuse coroutines with regular generators, since they
share the same syntax; async libraries often attempt to alleviate
this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);
So? PEP 492 never says what coroutines *are* in a way that explains
why it matters that they are different from generators.
Do you really mean "coroutines that can be suspended while they wait
for something slow"?
As best I can guess, the difference seems to be that a "normal"
generator is using yield primarily to say:
"I'm not done; I have more values when you want them",
but an asynchronous (PEP492) coroutine is primarily saying:
"This might take a while, go ahead and do something else meanwhile."
Correct.
As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.
Does it really permit *making* them, or does it just signal that you
will be waiting for them to finish processing anyhow, and it doesn't
need to be a busy-wait?
I does.
As nearly as I can tell, "async with" doesn't start processing the
managed block until the "asynchronous" call finishes its work -- the
only point of the async is to signal a scheduler that the task is
blocked.
Right.
Similarly, "async for" is still linearized, with each step waiting
until the previous "asynchronous" step was not merely launched, but
fully processed. If anything, it *prevents* within-task parallelism.
It enables cooperative parallelism.
It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
What justifies this limitation?
We want to avoid people passing regular generators and random
objects to 'await', because it is a bug.
Is there anything wrong awaiting something that eventually uses
"return" instead of "yield", if the "this might take a while" signal
is still true?
If it's an 'async def' then sure, you can use it in await.
Is the problem just that the current implementation
might not take proper advantage of task-switching?
Objects with ``__await__`` method are called *Future-like* objects in
the rest of this PEP.
Also, please note that ``__aiter__`` method (see its definition
below) cannot be used for this purpose. It is a different protocol,
and would be like using ``__iter__`` instead of ``__call__`` for
regular callables.
It is a ``TypeError`` if ``__await__`` returns anything but an
iterator.
What would be wrong if a class just did __await__ = __anext__ ?
If the problem is that the result of __await__ should be iterable,
then why isn't __await__ = __aiter__ OK?
For coroutines in PEP 492:
__await__ = __anext__ is the same as __call__ = __next__
__await__ = __aiter__ is the same as __call__ = __iter__
``await`` keyword is defined differently from ``yield`` and ``yield
from``. The main difference is that *await expressions* do not require
parentheses around them most of the times.
Does that mean
"The ``await`` keyword has slightly higher precedence than ``yield``,
so that fewer expressions require parentheses"?
class AsyncContextManager:
async def __aenter__(self):
await log('entering context')
Other than the arbitrary "keyword must be there" limitations imposed
by this PEP, how is that different from:
class AsyncContextManager:
async def __aenter__(self):
log('entering context')
This is OK. The point is that you can use 'await log' in
__aenter__. If you don't need awaits in __aenter__ you can
use them in __aexit__. If you don't need them there too,
then just define a regular context manager.
or even:
class AsyncContextManager:
def __aenter__(self):
log('entering context')
Will anything different happen when calling __aenter__ or log?
Is it that log itself now has more freedom to let other tasks run
in the middle?
__aenter__ must return an awaitable.
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
to use ``async with`` outside of a coroutine.
Why? Does that just mean they won't take advantage of the freedom
you offered them?
Not sure I understand the question.
It doesn't make any sense in using 'async with' outside of a
coroutine. The interpeter won't know what to do with them:
you need an event loop for that.
Or are you concern
Re: [Python-Dev] PEP 492: What is the real goal?
Hi Paul, On 2015-04-29 2:26 PM, Paul Moore wrote: On 29 April 2015 at 18:43, Jim J. Jewett wrote: Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); So? PEP 492 never says what coroutines *are* in a way that explains why it matters that they are different from generators. I agree. While I don't use coroutines/asyncio, and I may never do so, I will say that I find Python's approach very difficult to understand. I'd hope that the point of PEP 492, by making await/async first class language constructs, would be to make async programming more accessible in Python. Whether that will actually be the case isn't particularly clear to me. And whether "async programming" and "coroutines" are the same thing, I'm even less sure of. I haven't really followed the discussions here, because they seem to be about details that are completely confusing to me. It will make it more accessible in Python. asyncio is getting a lot of traction, and with this PEP accepted I can see it only becoming easier to work with it (or any other async frameworks that start using the new syntax/protocols). In principle, I support the PEP, on the basis that working towards better coroutine/async support in Python seems worthwhile to me. But until the whole area is made more accessible to the average programmer, I doubt any of this will be more than a niche area in Python. For example, the PEP says: """ New Coroutine Declaration Syntax The following new syntax is used to declare a coroutine: async def read_data(db): pass """ Looking at the Wikipedia article on coroutines, I see an example of how a producer/consumer process might be written with coroutines: var q := new queue coroutine produce loop while q is not full create some new items add the items to q yield to consume coroutine consume loop while q is not empty remove some items from q use the items yield to produce (To start everything off, you'd just run "produce"). I can't even see how to relate that to PEP 429 syntax. I'm not allowed to use "yield", so should I use "await consume" in produce (and vice versa)? I'd actually expect to just write 2 generators in Python, and use .send() somehow (it's clunky and I can never remember how to write the calls, but that's OK, it just means that coroutines don't have first-class syntax support in Python). This is totally unrelated to asyncio, which is the core use case for all of Python's async support. But it's what I think of when I see the word "coroutine" (and Wikipedia agrees). That Wikipedia page is very generic, and the pseudo-code that it uses does indeed look confusing. Here's how it might look like (this is the same pseudo-code but tailored for PEP 492, not a real something) q = asyncio.Queue(maxsize=100) async def produce(): # you might want to wrap it all in 'while True' while not q.full(): item = create_item() await q.put(item) async def consume(): while not q.empty(): item = await q.get() process_item(item) Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Hi Ethan, On 2015-04-29 2:32 PM, Ethan Furman wrote: On 04/29, Yury Selivanov wrote: On 2015-04-29 1:25 PM, Ethan Furman wrote: cannot also just work and be the same as the parenthesized version. Because it does not make any sense. I obviously don't understand your position that "it does not make any sense" -- perhaps you could explain a bit? What I see is a suspension point that is waiting for the results of coro(), which will be negated (and returned/assigned/whatever). What part of that doesn't make sense? Because you want operators to be resolved in the order you see them, generally. You want '(await -fut)' to: 1. Suspend on fut; 2. Get the result; 3. Negate it. This is a non-obvious thing. I would myself interpret it as: 1. Get fut.__neg__(); 2. await on it. So I want to make this syntactically incorrect: 'await -fut' would throw a SyntaxError. To do what you want, write a pythonic '- await fut'. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Hi Nathaniel, BTW, I'm going to reply to you in the other thread about context-local objects soon. I have some thoughts on the topic. On 2015-04-29 3:14 PM, Nathaniel Smith wrote: On Apr 29, 2015 11:49 AM, "Yury Selivanov" wrote: Hi Ethan, On 2015-04-29 2:32 PM, Ethan Furman wrote: On 04/29, Yury Selivanov wrote: On 2015-04-29 1:25 PM, Ethan Furman wrote: cannot also just work and be the same as the parenthesized version. Because it does not make any sense. I obviously don't understand your position that "it does not make any sense" -- perhaps you could explain a bit? What I see is a suspension point that is waiting for the results of coro(), which will be negated (and returned/assigned/whatever). What part of that doesn't make sense? Because you want operators to be resolved in the order you see them, generally. You want '(await -fut)' to: 1. Suspend on fut; 2. Get the result; 3. Negate it. This is a non-obvious thing. I would myself interpret it as: 1. Get fut.__neg__(); 2. await on it. So I want to make this syntactically incorrect: As a bystander, I don't really care either way about whether await -fut is syntactically valid (since like you say it's semantically nonsense regardless and no one will ever write it). But I would rather like to actually know what the syntax actually is, not just have a list of examples (which kinda gives me perl flashbacks). Is there any simple way to state what the rules for parsing await are? Or do I just have to read the parser code if I want to know that? There is a summary of grammar changes in the PEP: https://www.python.org/dev/peps/pep-0492/#grammar-updates You can also see the actual grammar file from the reference implementation: https://github.com/1st1/cpython/blob/await/Grammar/Grammar Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
Paul, On 2015-04-29 3:19 PM, Paul Moore wrote: On 29 April 2015 at 19:42, Yury Selivanov wrote: Here's how it might look like (this is the same pseudo-code but tailored for PEP 492, not a real something) q = asyncio.Queue(maxsize=100) async def produce(): # you might want to wrap it all in 'while True' I think the "loop" in the Wikipedia pseudocode was intended to be the "while True" here, not part of the "while" on the next line. while not q.full(): item = create_item() await q.put(item) async def consume(): while not q.empty(): item = await q.get() process_item(item) Thanks for that. That does look pretty OK. One question, though - it uses an asyncio Queue. The original code would work just as well with a list, or more accurately, something that wasn't designed for async use. So the translation isn't completely equivalent. Also, can I run the produce/consume just by calling produce()? My impression is that with asyncio I need an event loop - which "traditional" coroutines don't need. Nevertheless, the details aren't so important, it was only a toy example anyway. Well, yes. Coroutine is a generic term. And you can use PEP 492 coroutines without asyncio, in fact that's how most tests for the reference implementation is written. Coroutine objects have .send(), .throw() and .close() methods (same as generator objects in Python). You can work with them without a loop, but loop implementations contain a lot of logic to implement the actual cooperative execution. You can use generators as coroutines, and nothing would prevent you from doing that after PEP 492, moreover, for some use-cases it might be quite a good decision. But a lot of the code -- web frameworks, network applications, etc will hugely benefit from the proposal, streamlined syntax and async for/with statements. [..] But the use of "coroutine" in PEP 492 for the functions introduced by "async def" is confusing - at least to me - because I think of the above, and not of async. Why not just call them "async functions" and leave the term coroutine for the above flow control construct, which is where it originated? But maybe that ship has long sailed - the term "coroutine" is pretty entrenched in the asyncio documentation. If so, then I guess we have to live with the consequences. Everybody is pulling me in a different direction :) Guido proposed to call them "native coroutines". Some people think that "async functions" is a better name. Greg loves his "cofunction" term. I'm flexible about how we name 'async def' functions. I like to call them "coroutines", because that's what they are, and that's how asyncio calls them. It's also convenient to use 'coroutine-object' to explain what is the result of calling a coroutine. Anyways, I'd be OK to start using a new term, if "coroutine" is confusing. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
On 2015-04-29 3:24 PM, Isaac Schwabacher wrote: On 15-04-29, Yury Selivanov wrote: Hi Ethan, [..] So I want to make this syntactically incorrect: Does this need to be a syntax error? -"hello" raises TypeError because str doesn't have a __neg__, but there's no reason a str subclass couldn't define one. "TypeError: bad operand type for unary -: 'asyncio.Future'" is enough to clear up any misunderstandings, and if someone approaching a new language construct doesn't test their code well enough to at least execute all the code paths, the difference between a compile-time SyntaxError and a run-time TypeError is not going to save them. The grammar of the language should match the most common use case. FWIW, I've just updated the pep with a precedence table: https://hg.python.org/peps/rev/d355918bc0d7 Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Greg, On 2015-04-29 6:46 PM, Greg Ewing wrote: Yury Selivanov wrote: I'm not sure why Greg is pushing his Grammar idea so aggressively. Because I believe that any extra complexity in the grammar needs a very strong justification. It's complexity in the core language, like a new keyword, so it puts a burden on everyone's brain. Saying "I don't think anyone would ever need to write this, therefore we should disallow it" is not enough, given that there is a substantial cost to disallowing it. If you don't think there's a cost, consider that we *both* seem to be having trouble predicting the consequences of your proposed syntax, and you're the one who invented it. That's not a good sign! Sorry, but I'm not sure where & when I had any troubles predicting the consequences.. Please take a look at the updated PEP. There is a precedence table there + motivation. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; v3
Greg, On 2015-04-29 6:46 PM, Greg Ewing wrote: Yury Selivanov wrote: Won't that prevent some existing generator-based coroutines (ones not decorated with @coroutine) from calling ones implemented with 'async def'? It would. But that's not a backwards compatibility issue. It seems to go against Guido's desire for the new way to be a 100% drop-in replacement for the old way. There are various ways that old code can end up calling new code -- subclassing, callbacks, etc. It also means that if person A writes a library in the new style, then person B can't make use of it without upgrading all of their code to the new style as well. The new style will thus be "infectious" in a sense. I suppose it's up to Guido to decide whether it's a good or bad infection. But the same kind of reasoning seemed to be at least partly behind the rejection of PEP 3152. It's a drop-in replacement ;) If you run your existing code - it will 100% work just fine. There is a probability that *when* you start applying new syntax something could go wrong -- you're right here. I'm updating the PEP to explain this clearly, and let's see what Guido thinks about that. My opinion is that this is a solvable problem with a clear guidelines on how to transition existing code to the new style. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Nathaniel, On 2015-04-29 7:35 PM, Nathaniel Smith wrote: What I do feel strongly about is that whatever syntax we end up with, there should be*some* accurate human-readable description of*what it is*. AFAICT the PEP currently doesn't have that. How to define human-readable description of how unary minus operator works? Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Guido, On 2015-04-29 7:52 PM, Guido van Rossum wrote: On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov wrote: Nathaniel, On 2015-04-29 7:35 PM, Nathaniel Smith wrote: What I do feel strongly about is that whatever syntax we end up with, there should be*some* accurate human-readable description of*what it is*. AFAICT the PEP currently doesn't have that. How to define human-readable description of how unary minus operator works? In a PEP you should probably give grammar that is not the actual grammar from the implementation, but matches the grammar used in the reference manual on docs.python.org. Will do! Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
Nathaniel, On 2015-04-29 7:58 PM, Nathaniel Smith wrote: On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov wrote: Nathaniel, On 2015-04-29 7:35 PM, Nathaniel Smith wrote: What I do feel strongly about is that whatever syntax we end up with, there should be*some* accurate human-readable description of*what it is*. AFAICT the PEP currently doesn't have that. How to define human-readable description of how unary minus operator works? Hah, good question :-). Of course we all learned how to parse arithmetic in school, so perhaps it's a bit cheating to refer to that knowledge. Except of course basically all our users *do* have that knowledge (or else are forced to figure it out anyway). So I would be happy with a description of "await" that just says "it's like unary minus but higher precedence". Even if we put aside our trained intuitions about arithmetic, I think it's correct to say that the way unary minus is parsed is: everything to the right of it that has a tighter precedence gets collected up and parsed as an expression, and then it takes that expression as its argument. Still pretty simple. Well, await is defined exactly like that ;) Anyways, I'll follow Guido's suggestion to define await in the PEP the same way we define other syntax in python docs. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 quibble and request
Hi Ethan, On 2015-04-29 8:21 PM, Ethan Furman wrote: From the PEP: Why not a __future__ import __future__ imports are inconvenient and easy to forget to add. That is a horrible rationale for not using an import. By that logic we should have everything in built-ins. ;) Working example ... The working example only uses async def and await, not async with nor async for nor __aenter__, etc., etc. Could you put in a more complete example -- maybe a basic chat room with both server and client -- that demonstrated more of the new possibilities? Andrew Svetlov has implemented some new features in his aiomysql driver: https://github.com/aio-libs/aiomysql/blob/await/tests/test_async_iter.py I don't want to cite it in the PEP because it's not complete yet, and some idioms (like 'async with') aren't used to their full potential. Having gone through the PEP again, I am still no closer to understanding what happens here: data = await reader.read(8192) What does the flow of control look like at the interpreter level? 'await' is semantically equivalent to 'yield from' in this line. To really understand all implementation details of this line you need to read PEP 3156 and experiment with asyncio. There is no easier way, unfortunately. I can't add a super detailed explanation how event loops can be implemented in PEP 492, that's not in its scope. The good news is that to use asyncio on a daily basis you don't need to know all details, as you don't need to know how 'ceval.c' works and how 'setTimeout' is implemented in JavaScript. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; version 4
Hi python-dev, New version of the PEP is attached. Summary of updates: 1. Terminology: - *native coroutine* term is used for "async def" functions. - *generator-based coroutine* term is used for PEP 380 coroutines used in asyncio. - *coroutine* is used when *native coroutine* or *generator based coroutine* can be used in the same context. I think that it's not really productive to discuss the terminology that we use in the PEP. Its only purpose is to disambiguate concepts used in the PEP. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Although if you notice that somewhere in the PEP it is not crystal clear what "coroutine" means please give me a heads up! 2. Syntax of await expressions is now thoroghly defined in the PEP. See "Await Expression", "Updated operator precedence table", and "Examples of "await" expressions" sections. I like the current approach. I'm still not convinced that we should make 'await' the same grammatically as unary minus. I don't understand why we should allow parsing things like 'await -fut'; this should be a SyntaxError. I'm fine to modify the grammar to allow 'await await fut' syntax, though. And I'm open to discussion :) 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. 4. asyncio / Migration strategy. Existing code can be used with PEP 492 as is, everything will work as expected. However, since *plain generator*s (not decorated with @asyncio.coroutine) cannot 'yield from' native coroutines (async def functions), it might break some code *while adapting it to the new syntax*. I'm open to just throw a RuntimeWarning in this case in 3.5, and raise a TypeError in 3.6. Please see the "Differences from generators" section of the PEP. 5. inspect.isawaitable() function. And, all new functions are now listed in the "New Standard Library Functions" section. 6. Lot's of small updates and tweaks throughout the PEP. Thanks, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015 Abstract This PEP introduces new syntax for coroutines, asynchronous ``with`` statements and ``for`` loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); * it is not possible to natively define a coroutine which has no ``yield`` or ``yield from`` statements, again requiring the use of decorators to fix potential refactoring issues; * support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification = This proposal introduces new syntax and semantics to enhance coroutine support in Python. This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the "Cofunctions" proposal (PEP 3152, now rejected in favor of this specification). From this point in this document we use the word *native coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based
Re: [Python-Dev] PEP 492: async/await in Python; version 4
One more thing to discuss: 7. StopAsyncIteration vs AsyncStopIteration. I don't have a strong opinion on this, I prefer the former because it reads better. There was no consensus on which one we should use. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 quibble and request
Nick,
On 2015-04-29 9:52 PM, Nick Coghlan wrote:
On 30 April 2015 at 10:33, Yury Selivanov wrote:
To really understand all implementation details of this line
you need to read PEP 3156 and experiment with asyncio. There
is no easier way, unfortunately. I can't add a super detailed
explanation how event loops can be implemented in PEP 492,
that's not in its scope.
This request isn't about understanding the implementation details,
it's about understanding what Python *users* will gain from the PEP
without *needing* to understand the implementation details.
For that, I'd like to see some not-completely-trivial example code in
(or at least linked from) the PEP written using:
* trollius (no "yield from", Python 2 compatible, akin to Twisted's
inlineDeferred's)
* asyncio/tulip ("yield from", Python 3.3+ compatible)
* PEP 492 (async/await, Python 3.5+ only)
I'll see what I can do with aiomysql library to showcase
async for and async with. I have a few outstanding tasks
with reference implementation to test/add though.
I'm not sure that trollius will add anything to it. It's just
like asyncio, but uses 'yield' instead of 'yield from' and was
envisioned by Victor Stinner as a transitional-framework
to ease the porting of openstack to python 3.
The *intent* of PEP 492, like PEP 380 and PEP 342 before it, is "make
asynchronous programming in Python easier". I think it will actually
succeed in that goal, but I don't think it currently does an
especially job of explaining that to folks that aren't both already
deeply invested in the explicitly asynchronous programming model *and*
thoroughly aware of the fact that most of us need asynchronous
programming to look as much like synchronous programming as possible
in order for it to fit our brains. Some folks can fit ravioli code
with callbacks going everywhere in their brains, but I can't, and it's
my experience that most other folks can't either. This lack means the
PEP that gets confused objections from folks that wish explicitly
asynchronous programming models would just go away entirely (they
won't), as well as from folks that already understand it and don't see
why we can't continue treating it as a special case of other features
that folks have to learn how to use from first principles, rather than
saving them that up front learning cost by doing the pattern
extraction to make event driven explicitly asynchronous programming
its own first class concept with dedicated syntax (a hint on that
front: with statements are just particular patterns for using
try/except/finally, decorators are just particular patterns in using
higher order functions, for statements are just particular patterns in
using while statements and builtins, and even imports and classes just
represent particular patterns in combining dictionaries, code
execution and the metaclass machinery - the pattern extraction and
dedicated syntax associated with all of them makes it possible to
learn to *use* these concepts without first having to learn how to
*implement* them)
From my own perspective, I've spent a reasonable amount of time
attempting to explain to folks the "modal" nature of generators, in
that you can use them both as pure iterable data sources *and* as
coroutines (as per PEP 342).
The problem I've found is that our current approach ends up failing
the "conceptually different things should also look superficially
different" test: outside certain data pipeline processing problems,
generators-as-iterators and generators-as-coroutines mostly end up
being fundamentally *different* ways of approaching a programming
problem, but the current shared syntax encourages users to attempt to
place them in the same mental bucket. Those users will remain
eternally confused until they learn to place them in two different
buckets despite the shared syntax (the 5 or so different meanings of
"static" in C++ come to mind at this point...).
Agree. This confusion of trying to fit two fundamentally
different programming models in one syntax is what led me
to start thinking about PEP 492 ideas several years ago.
And the absence of 'async with' and 'async for' statements
forced me to use greenlets, which is another reason for
the PEP.
With explicitly asynchronous development*, this problem of needing to
learn to segment the world into two pieces is particularly important,
and this wonderful rant on red & blue functions published a couple of
months ago helps explain why:
http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/
The dedicated async/await syntax proposed in PEP 492 lets the end user
mostly *not care* about all the messiness that needs to happen under
the covers to make explicitly asynchronous programming as close to
normal synchronous programming as possible.
Re: [Python-Dev] PEP 492 quibble and request
On 2015-04-29 11:01 PM, Nick Coghlan wrote: On 30 April 2015 at 12:31, Guido van Rossum wrote: >On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan wrote: >> >>[...] >>Yeah, I'm coming around to the idea. For the async pseudo-keyword, I >>can see that the proposal only allows its use in cases that were >>previously entirely illegal, but I'm not yet clear on how the PEP >>proposes to avoid changing the meaning of the following code: >> >> x = await(this_is_a_function_call) >> >>Unless I'm misreading the proposed grammar in the PEP (which is >>entirely possible), I believe PEP 492 would reinterpret that as: >> >> x = await this_is_not_a_function_call_any_more > > >Ah, but here's the other clever bit: it's only interpreted this way*inside* >a function declared with 'async def'. Outside such functions, 'await' is not >a keyword, so that grammar rule doesn't trigger. (Kind of similar to the way >that the print_function __future__ disables the keyword-ness of 'print', >except here it's toggled on or off depending on whether the nearest >surrounding scope is 'async def' or not. The PEP could probably be clearer >about this; it's all hidden in the Transition Plan section.) Ah, nice, although even reading the Transition Plan section didn't clue me in to that particular aspect of the idea :) Given that clarification, I think the rationale for "no __future__ statement needed" can be strengthened by focusing on the fact that such a statement would largely be*redundant*, given that: * "async def", "async with", and "async for" are all currently syntax errors, and hence adding them is backwards compatible if "async" is otherwise treated as a normal variable name * "await " only gains its new interpretation when used inside an "async def" statement, so "async def" fills the role that a module level compiler declaration like "from __future__ import async_functions" would otherwise fill Thanks, Nick. I've fixed the Transition Plan section, and rewrote the "why not __future__" one too. https://hg.python.org/peps/rev/552773d7e085 https://hg.python.org/peps/rev/5db3ad3d540b Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-04-30 5:16 AM, Greg Ewing wrote: Yury Selivanov wrote: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I don't think you can honestly claim "full backwards compatibility" as long as there are some combinations of old-style and new-style code that won't work together. You seem to be using your own personal definition of "full" here. Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 vs. PEP 3152, new round
On 2015-04-30 1:56 PM, Ethan Furman wrote: >Why not? Unlike some other languages, Python does not have uniform >priorities for unary operators, so it's reasonable for some unary >operations to have a different priority than others, and certain things >will be SyntaxErrors because of that. E.g. you can write "not -x" but you >can't write "- not x". For one, Yury's answer is "- await x" which looks just as nonsensical as "- not x". "- await x" is a perfectly valid code: result = - await compute_in_db() (same as "result = - (yield from do_something())") For another, an error of some type will be raised if either __neg__ doesn't exist or it doesn't return an awaitable, so a SyntaxError is unnecessary. For a third, by making it a SyntaxError you are forcing the use of parens to get what should be the behavior anyway. I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere. FWIW, I'll fix the 'await (await x)' expression to be parsed without parens. In other words, a SyntaxError is nat any clearer than "AttributeError: obj has no __neg__ method" and it's not any clearer than "AwaitError: __neg__ returned not-awaitable". Those last two errors tell you exactly what you did wrong. This is debatable. "obj has no __neg__ method" isn't obvious to everyone (especially to those people who aren't using operator overloading). [1] https://www.python.org/dev/peps/pep-0492/#examples-of-await-expressions Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
Jim,
On 2015-04-30 2:41 PM, Jim J. Jewett wrote:
[...]
Does it really permit *making* them [asynchronous calls], or does
it just signal that you will be waiting for them to finish processing
anyhow, and it doesn't need to be a busy-wait?
I does.
Bad phrasing on my part. Is there anything that prevents an
asynchronous call (or waiting for one) without the "async with"?
If so, I'm missing something important. Either way, I would
prefer different wording in the PEP.
Yes, you can't use 'yield from' in __exit__/__enter__
in current Python.
It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*,
which can be one of:
What justifies this limitation?
We want to avoid people passing regular generators and random
objects to 'await', because it is a bug.
Why?
Is it a bug just because you defined it that way?
Is it a bug because the "await" makes timing claims that an
object not making such a promise probably won't meet? (In
other words, a marker interface.)
Is it likely to be a symptom of something that wasn't converted
correctly, *and* there are likely to be other bugs caused by
that same lack of conversion?
Same as 'yield from' is expecting an iterable, await
is expecting an awaitable. That's the protocol.
You can't pass random objects to 'with' statements,
'yield from', 'for..in', etc.
If you write
def gen(): yield 1
await gen()
then it's a bug.
For coroutines in PEP 492:
__await__ = __anext__ is the same as __call__ = __next__
__await__ = __aiter__ is the same as __call__ = __iter__
That tells me that it will be OK sometimes, but will usually
be either a mistake or an API problem -- and it explains why.
Please put those 3 lines in the PEP.
There is a line like that:
https://www.python.org/dev/peps/pep-0492/#await-expression
Look for "Also, please note..." line.
This is OK. The point is that you can use 'await log' in
__aenter__. If you don't need awaits in __aenter__ you can
use them in __aexit__. If you don't need them there too,
then just define a regular context manager.
Is it an error to use "async with" on a regular context manager?
If so, why? If it is just that doing so could be misleading,
then what about "async with mgr1, mgr2, mgr3" -- is it enough
that one of the three might suspend itself?
'with' requires an object with __enter__ and __exit__
'async with' requires an object with __aenter__ and __aexit__
You can have an object that implements both interfaces.
class AsyncContextManager:
def __aenter__(self):
log('entering context')
__aenter__ must return an awaitable
Why? Is there a fundamental reason, or it is just to avoid the
hassle of figuring out whether or not the returned object is a
future that might still need awaiting?
The fundamental reason why 'async with' is proposed is because
you can't suspend execution in __enter__ and __exit__.
If you need to suspend it there, use 'async with' and
its __a*__ methods, but they have to return awaitable
(see https://www.python.org/dev/peps/pep-0492/#new-syntax
and look what 'async with' is semantically equivalent to)
Is there an assumption that the scheduler will let the thing-being
awaited run immediately, but look for other tasks when it returns,
and a further assumption that something which finishes the whole
task would be too slow to run right away?
It doesn't make any sense in using 'async with' outside of a
coroutine. The interpeter won't know what to do with them:
you need an event loop for that.
So does the PEP also provide some way of ensuring that there is
an event loop? Does it assume that self-suspending coroutines
will only ever be called by an already-running event loop
compatible with asyncio.get_event_loop()? If so, please make
these contextual assumptions explicit near the beginning of the PEP.
You need some kind of loop, but it doesn't have to the one
from asyncio. There is at least one place in the PEP where
it's mentioned that the PEP introduses a generic concept
that can be used by asyncio *and* other frameworks.
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
outside of a coroutine.
The same questions about why -- what is the harm?
I can imagine that as an implementation detail, the async for wouldn't
be taken advtange of unless it was running under an event loop that
knew to look for "aync for" as suspension points.
Event loop doesn't need to know anything about 'async with'
and 'async for'. For loop it's always one thing -- something
is awaiting somewhere for some result.
I'm not seeing what the actual harm is in either not happening to
suspend (less efficient, but still correct), or in suspending between
every step of a regular iterator (because, why not?)
For debugging this kind of mistakes there is a special debug mode in
asyncio, in which ``@coroutine``
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-04-30 7:24 PM, Greg Ewing wrote: Yury Selivanov wrote: Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. Ah, alright. You quoted this: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I wrote "full backwards compatibility" for that particular point #3 -- existing @asyncio.coroutines will have __iter__ and __next__ working just fine. Sorry if this was misleading. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim "full backwards compatibility". I covered this in point #4. I also touched this in https://www.python.org/dev/peps/pep-0492/#migration-strategy I'm still waiting for feedback on this from Guido. If he decides to go with RuntimeWarnings, then it's 100% backwards compatible. If we keep TypeErrors -- then *existing code will work on 3.5*, but something *might* break during adopting new syntax. I'll update the Backwards Compatibility section. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Stefan,
I don't like the idea of combining __next__ and __anext__.
In this case explicit is better than implicit. __next__
returning coroutines is a perfectly normal thing for a
normal 'for' loop (it wouldn't to anything with them),
whereas 'async for' will interpret that differently, and
will try to await those coroutines.
Yury
On 2015-05-01 1:10 PM, Stefan Behnel wrote:
Guido van Rossum schrieb am 01.05.2015 um 17:28:
On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote:
Yury Selivanov schrieb am 30.04.2015 um 03:30:
Asynchronous Iterators and "async for"
--
An *asynchronous iterable* is able to call asynchronous code in its
*iter* implementation, and *asynchronous iterator* can call
asynchronous code in its *next* method. To support asynchronous
iteration:
1. An object must implement an ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.
What this section does not explain, AFAICT, nor the section on design
considerations, is why the iterator protocol needs to be duplicated
entirely. Can't we just assume (or even validate) that any 'regular'
iterator returned from "__aiter__()" (as opposed to "__iter__()") properly
obeys to the new protocol? Why additionally duplicate "__next__()" and
"StopIteration"?
ISTM that all this really depends on is that "__next__()" returns an
awaitable. Renaming the method doesn't help with that guarantee.
This is an astute observation. I think its flaw (if any) is the situation
where we want a single object to be both a regular iterator and an async
iterator (say when migrating code to the new world). The __next__ method
might want to return a result while __anext__ has to return an awaitable.
The solution to that would be to have __aiter__ return an instance of a
different class than __iter__, but that's not always convenient.
My personal gut feeling is that this case would best be handled by a
generic wrapper class. Both are well defined protocols, so I don't expect
people to change all of their classes and instead return a wrapped object
either from __iter__() or __aiter__(), depending on which they want to
optimise for, or which will eventually turn out to be easier to wrap.
But that's trying to predict the [Ff]uture, obviously. It just feels like
unnecessary complexity for now. And we already have a type slot for
__next__ ("tp_iternext"), but not for __anext__.
Stefan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Let's say it this way: I want to know what I am looking at when I browse through the code -- an asynchronous iterator, or a normal iterator. I want an explicit difference between these protocols, because they are different. Moreover, the below code is a perfectly valid, infinite iterable: class SomeIterable: def __iter__(self): return self async def __next__(self): return 'spam' I'm strong -1 on this idea. Yury On 2015-05-01 3:03 PM, Stefan Behnel wrote: Yury Selivanov schrieb am 01.05.2015 um 20:52: I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. This has nothing to do with implicit vs. explicit. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-01 3:19 PM, Ethan Furman wrote: Sure, but the difference is that one would have called __aiter__() first >and the other __iter__(). Normally, either of the two would not exist, so >using the wrong loop on an object will just fail. However, after we passed >that barrier, we already know that the object that was returned is supposed >to obey to the expected protocol, so it doesn't matter whether we call >__next__() or name it __anext__(), except that the second requires us to >duplicate an existing protocol. If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. And this is a good point too. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-01 3:23 PM, Yury Selivanov wrote:
Let's say it this way: I want to know what I am looking at
when I browse through the code -- an asynchronous iterator,
or a normal iterator. I want an explicit difference between
these protocols, because they are different.
Moreover, the below code is a perfectly valid, infinite
iterable:
class SomeIterable:
def __iter__(self):
return self
async def __next__(self):
return 'spam'
I'm strong -1 on this idea.
To further clarify on the example:
class SomeIterable:
def __iter__(self):
return self
async def __aiter__(self):
return self
async def __next__(self):
print('hello')
raise StopAsyncIteration
If you pass this to 'async for' you will get
'hello' printed and the loop will be over.
If you pass this to 'for', you will get an
infinite loop, because '__next__' will return a
coroutine object (that has to be also awaited,
but it wouldn't, because it's a plain 'for'
statement).
This is something that we shouldn't let happen.
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-01 4:24 PM, Arnaud Delobelle wrote: On 1 May 2015 at 20:24, Yury Selivanov wrote: On 2015-05-01 3:19 PM, Ethan Furman wrote: [...] If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. And this is a good point too. I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. I doubt that it will be a popular thing. But disallowing this by merging two different protocols in one isn't a good idea either. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
On 2015-05-01 5:37 PM, Jim J. Jewett wrote:
On Thu Apr 30 21:27:09 CEST 2015, Yury Selivanov replied:
On 2015-04-30 2:41 PM, Jim J. Jewett wrote:
Bad phrasing on my part. Is there anything that prevents an
asynchronous call (or waiting for one) without the "async with"?
If so, I'm missing something important. Either way, I would
prefer different wording in the PEP.
Yes, you can't use 'yield from' in __exit__/__enter__
in current Python.
I tried it in 3.4, and it worked.
I'm not sure it would ever be sensible, but it didn't raise any
errors, and it did run.
What do you mean by "can't use"?
It probably executed without errors, but it didn't run the
generators.
class Foo:
def __enter__(self):
yield from asyncio.sleep(0)
print('spam')
with Foo(): pass # <- 'spam' won't ever be printed.
For coroutines in PEP 492:
__await__ = __anext__ is the same as __call__ = __next__
__await__ = __aiter__ is the same as __call__ = __iter__
That tells me that it will be OK sometimes, but will usually
be either a mistake or an API problem -- and it explains why.
Please put those 3 lines in the PEP.
There is a line like that:
https://www.python.org/dev/peps/pep-0492/#await-expression
Look for "Also, please note..." line.
It was from reading the PEP that the question came up, and I
just reread that section.
Having those 3 explicit lines goes a long way towards explaining
how an asychio coroutine differs from a regular callable, in a
way that the existing PEP doesn't, at least for me.
This is OK. The point is that you can use 'await log' in
__aenter__. If you don't need awaits in __aenter__ you can
use them in __aexit__. If you don't need them there too,
then just define a regular context manager.
Is it an error to use "async with" on a regular context manager?
If so, why? If it is just that doing so could be misleading,
then what about "async with mgr1, mgr2, mgr3" -- is it enough
that one of the three might suspend itself?
'with' requires an object with __enter__ and __exit__
'async with' requires an object with __aenter__ and __aexit__
You can have an object that implements both interfaces.
I'm not still not seeing why with (let alone await with) can't
just run whichever one it finds. "await with" won't actually let
the BLOCK run until the future is resolved. So if a context
manager only supplies __enter__ instead of __aenter__, then at most
you've lost a chance to switch tasks while waiting -- and that is no
worse than if the context manager just happened to be really slow.
let's say you have a function:
def foo():
with Ctx(): pass
if Ctx.__enter__ is a generator/coroutine, then
foo becomes a generator/coroutine (otherwise how
(and to what) would you yield from/await on __enter__?).
And then suddenly calling 'foo' doesn't do anything
(it will return you a generator/coroutine object).
This isn't transparent or even remotely
understandable.
For debugging this kind of mistakes there is a special debug mode in
Is the intent to do anything more than preface execution with:
import asynchio.coroutines
asynchio.coroutines._DEBUG = True
This won't work, unfortunately. You need to set the
debug flag *before* you import asyncio package (otherwise
we would have an unavoidable performance cost for debug
features). If you enable it after you import asyncio,
then asyncio itself won't be instrumented. Please
see the implementation of asyncio.coroutine for details.
Why does asynchio itself have to wrapped? Is that really something
normal developers need to debug, or is it only for developing the
stdlib itself? If it if only for developing the stdlib, than I
would rather see workarounds like shoving _DEBUG into builtins
when needed, as opposed to adding multiple attributes to sys.
Yes, normal developers need asyncio to be instrumented,
otherwise you won't know what you did wrong when you
called some asyncio code without 'await' for example.
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 and types.coroutine
On 2015-05-02 1:04 PM, Ethan Furman wrote: According to https://www.python.org/dev/peps/pep-0492/#id31: The [types.coroutine] function applies CO_COROUTINE flag to generator-function's code object, making it return a coroutine object. Further down in https://www.python.org/dev/peps/pep-0492/#id32: [await] uses the yield from implementation with an extra step of validating its argument. await only accepts an awaitable, which can be one of: ... - A generator-based coroutine object returned from a generator decorated with types.coroutine(). If I'm understanding this correctly, type.coroutine's only purpose is to add a flag to a generator object so that await will accept it. This raises the question of why can't await simply accept a generator object? There is no code change to the gen obj itself, there is no behavior change in the gen obj, it's the exact same byte code, only a flag is different. Because we don't want 'await' to accept random generators. It can't do anything meaningful with them, in a world where all asyncio code is written with new syntax, passing generator to 'await' is just a bug. 'types.coroutine' is something that we need to ease transition to the new syntax. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492 and types.coroutine
On 2015-05-02 2:14 PM, Ethan Furman wrote: On 05/02, Yury Selivanov wrote: On 2015-05-02 1:04 PM, Ethan Furman wrote: If I'm understanding this correctly, type.coroutine's only purpose is to add a flag to a generator object so that await will accept it. This raises the question of why can't await simply accept a generator object? There is no code change to the gen obj itself, there is no behavior change in the gen obj, it's the exact same byte code, only a flag is different. Because we don't want 'await' to accept random generators. It can't do anything meaningful with them, in a world where all asyncio code is written with new syntax, passing generator to 'await' is just a bug. And yet in current asyncio code, random generators can be accepted, and not even the current asyncio.coroutine wrapper can gaurantee that the generator is a coroutine in fact. This is a flaw in the current Python that we want to fix. For that matter, even the new types.coroutine cannot gaurantee that the returned object is a coroutine and not a generator -- so basically it's just there to tell the compiler, "yeah, I really know what I'm doing, shut up and do what I asked." Well, why would you use it on some generator that is not a generator-based coroutine? 'types.coroutine' is something that we need to ease transition to the new syntax. This doesn't make sense -- either the existing generators are correctly returning coroutines, in which case the decorator adds nothing, or they are returning non-coroutines, in which case the decorator adds nothing. So either way, nothing has been added besides a mandatory boiler-plate requirement. It's not nothing; it's backwards compatibility. Please read https://www.python.org/dev/peps/pep-0492/#await-expression @types.coroutine marks generator function that its generator is awaitable. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; version 5
Hi python-dev, Updated version of the PEP is below. Quick summary of changes: 1. set_coroutine_wrapper and get_coroutine_wrapper functions are now thread-specific (like settrace etc). 2. Updated Abstract & Rationale sections. 3. RuntimeWarning is always raised when a coroutine wasn't awaited on. This is in addition to what 'set_coroutine_wrapper' will/can do. 4. asyncio.async is renamed to asyncio.ensure_future; it will be deprecated in 3.5. 5. Uses of async/await in CPython codebase are documented. 6. Other small edits and updates. Thanks, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015, 05-May-2015 Abstract The growth of Internet and general connectivity has triggered the proportionate need for responsive and scalable code. This proposal aims to answer that need by making writing explicitly asynchronous, concurrent Python code easier and more Pythonic. It is proposed to make *coroutines* a proper standalone concept in Python, and introduce new supporting syntax. The ultimate goal is to help establish a common, easily approachable, mental model of asynchronous programming in Python and make it as close to synchronous programming as possible. We believe that the changes proposed here will help keep Python relevant and competitive in a quickly growing area of asynchronous programming, as many other languages have adopted, or are planning to adopt, similar features: [2]_, [5]_, [6]_, [7]_, [8]_, [10]_. Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * It is easy to confuse coroutines with regular generators, since they share the same syntax; this is especially true for new developers. * Whether or not a function is a coroutine is determined by a presence of ``yield`` or ``yield from`` statements in its *body*, which can lead to unobvious errors when such statements appear in or disappear from function body during refactoring. * Support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification = This proposal introduces new syntax and semantics to enhance coroutine support in Python. This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the "Cofunctions" proposal (PEP 3152, now rejected in favor of this specification). From this point in this document we use the word *native coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. *coroutine* is used in contexts where both definitions are applicable. New Coroutine Declaration Syntax The following new syntax is used to declare a *native coroutine*:: async def read_data(db): pass Key properties of *coroutines*: * ``async def`` functions are always coroutines, even if they do not contain ``await`` expressions. * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. * Internally, two new code object flags were introduced: - ``CO_COROUTINE`` is used to enable runtime detection of *coroutines* (and migrating existing code). - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines* (defined with new syntax.) All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and ``CO_GENERATOR`` flags set. * Regular generators, when called, return a *generator object*; similarly, coroutines return a *coroutine object*. * ``StopIteration`` exceptions are not propagated out of coroutines, and are r
Re: [Python-Dev] PEP 492: What is the real goal?
Hi Oscar, I've updated the PEP with some fixes of the terminology: https://hg.python.org/peps/rev/f156b272f860 I still think that 'coroutine functions' and 'coroutines' is a better pair than 'async functions' and 'coroutines'. First, it's similar to existing terminology for generators. Second, it's less confusing. With pep492 at some point, using generators to implement coroutines won't be a wide spread practice, so 'async def' functions will be the only language construct that returns them. Yury On 2015-05-05 12:01 PM, Oscar Benjamin wrote: On 30 April 2015 at 09:50, Arnaud Delobelle wrote: I'm flexible about how we name 'async def' functions. I like to call them "coroutines", because that's what they are, and that's how asyncio calls them. It's also convenient to use 'coroutine-object' to explain what is the result of calling a coroutine. I'd like the object created by an 'async def' statement to be called a 'coroutine function' and the result of calling it to be called a 'coroutine'. That would be an improvement over the confusing terminology in the PEP atm. The PEP proposes to name the inspect functions inspect.iscoroutine() and inspect.iscoroutinefunction(). According to the PEP iscoroutine() identifies "coroutine objects" and iscoroutinefunction() identifies "coroutine functions" -- a term which is not defined in the PEP but presumably means what the PEP calls a "coroutine" in the glossary. Calling the async def function an "async function" and the object it returns a "coroutine" makes for the clearest terminology IMO (provided the word coroutine is not also used for anything else). It would help to prevent both experienced and new users from confusing the two related but necessarily distinct concepts. Clearly distinct terminology makes it easier to explain/discuss something if nothing else because it saves repeating definitions all the time. -- Oscar ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Hi Wolfgang, On 2015-05-05 7:27 AM, Wolfgang wrote: Hi, [..] Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. "networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Paul, On 2015-05-05 3:14 PM, Paul Moore wrote: On 5 May 2015 at 19:25, Yury Selivanov wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. I agree. We have to improve asyncio docs in this area. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it". Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid. Again, PEP 492 is not only for asyncio. *Any* framework can use it, including Twisted. As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. I can't see any fundamental problem with the PEP behind such discussions. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool. There is aiohttp library [1], which provides a client API similar to requests. And if you want to write high performance networking server in python3 you *will* choose asyncio (or gevent/twisted in python2). And PEP 492 is aimed to make this whole async stuff more accessible to an average user. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? You can use 'loop.run_in_executor' in asyncio. It returns a future that you can await on. You can also provide a nice facade for your Oracle-database code that provides a nice API but uses asyncio thread executor behind the scenes. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. "networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). There are some things (like websockets) that are hard to implement correctly in existing frameworks like django and flask. And these kind of things are becoming more and more important. Languages like Go were designed specifically to allow writing efficient And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probabl
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. (2) The new syntax is intended to make it easier to recognize when a task's execution may be interrupted by arbitrary other tasks, and the interrupted task therefore has to revalidate assumptions about shared data. With threads, CPython can always suspend a task between op-codes, but with a sufficiently comprehensive loop (and sufficiently coooperative tasks), tasks *should* only be suspended when they make an explicit request to *wait* for an answer, and these points *should* be marked syntactically. (3) The new constructs explicitly do NOT support any sort of concurrent execution within a task; they are for use precisely when otherwise parallel subtasks are being linearized by pausing and waiting for the results. Yes. Over-simplifications 4-6 assume a world with standardized futures based on concurrent.futures, where .result either returns the result or raises the exception (or raises another exception about timeout or cancellation). [Note that the actual PEP uses iteration over the results of a new __await__ magic method, rather than .result on the object itself. I couldn't tell whether this was for explicit marking, or just for efficiency in avoiding future creation.] (4) "await EXPR" is just syntactic sugar for EXPR.result except that, by being syntax, it better marks locations where unrelated tasks might have a chance to change shared data. [And that, as currently planned, the result of an await isn't actually the result; it is an iterator of results.] I'm not sure how to comment on (4). Perhaps I don't understand some notation that you're using. If anything, it's more of a syntactic sugar for 'yield from EXPR'. (5) "async def" is just syntactic sugar for "def", except that, by being syntax, it better marks the signatures of functions and methods where unrelated tasks might have a chance to change shared data after execution has already begun. It also sets "CO_COROUTINE | CO_GENERATOR" flags, that are very important. (5A) As the PEP currently stands, it is also a promise that the function will NOT produce a generator used as an iterator; if a generator-iterator needs to wait for something else at some point, that will need to be done differently. I derive this limitation from "It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function." but I don't understand how this limitation works with things like a per-line file iterator that might need to wait for the file to be initially opened. Per-line file iterator can be implemented with __aiter__, __anext__ protocol. __aiter__ is a coroutine, you can open/start reading your file there. (6) async with EXPR as VAR: would be equivalent to: with EXPR as VAR: except that __enter__() would be replaced by next(await __enter__()) # __enter__().result __exit__() would be replaced by next(await __exit__()) # __exit__().result I'm not sure I understand what you mean by "next(await EXPR)" notation. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-05 4:39 PM, Paul Moore wrote: Is there anyone who feels they could write a stripped down but working example of a valid Python event loop*without* the asyncio aspects? Or is that what David Beazley's talk does? Yes, in David's talk, where he starts to use 'yield from' you can simply use new coroutines. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 2015-05-05 5:01 PM, Paul Moore wrote: On 5 May 2015 at 21:00, Yury Selivanov wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? The idea is that you integrate processing of UI events to your event loop of choice. For instance, Twisted has integration for QT and other libraries [1]. This way you can easily combine async network (or OS) calls with your UI logic to avoid "callback hell". Quick search for something like that for asyncio revealed this library: [2]. This small library actually re-implements relevant low-level parts of the asyncio event loop on top of QT primitives (another approach). Yury [1] http://twistedmatrix.com/trac/wiki/QTReactor [2] https://github.com/harvimt/quamash#usage -- see first_50 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-05 5:31 PM, Jim J. Jewett wrote: Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote: >As for terminology, I view this discussion differently. It's >not about the technical details (Python has asymmetric >coroutines, that's it), but rather on how to disambiguate >coroutines implemented with generators and yield-from, from >new 'async def' coroutines. Not just "How?", but "Why?". Why do they*need* to be disambiguated? To clearly show how one interacts with the other, to explain how backwards compatibility is implemented, and to better illustrate some additional (and necessary) restrictions we put on 'async def' coroutines. Otherwise, the PEP would be completely unreadable :) Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
Paul, On 2015-05-05 5:54 PM, Paul Moore wrote: On 5 May 2015 at 22:38, Yury Selivanov wrote: n 2015-05-05 5:01 PM, Paul Moore wrote: On 5 May 2015 at 21:00, Yury Selivanov wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? The idea is that you integrate processing of UI events to your event loop of choice. For instance, Twisted has integration for QT and other libraries [1]. This way you can easily combine async network (or OS) calls with your UI logic to avoid "callback hell". We seem to be talking at cross purposes. You say the PEP is *not* exclusively intended for use with asyncio. You mention UI loops, but when asked how to implement such a loop, you say that I integrate UI events into my event loop of choice. But what options do I have for "my event loop of choice"? Please provide a concrete example that isn't asyncio. Yes, there is no other popular event loop for 3.4 other than asyncio, that uses coroutines based on generators (as far as I know). And yes, the PEP is not exclusively intended for use with asyncio, but asyncio is the only library that ships with Python, and is Python 3 ready, so its users will be the first ones to directly benefit from this proposal. Can I use PEP 492 with Twisted (I doubt it, as Twisted doesn't use yield from, which is Python 3.x only)? I contend that there *is* no concrete example that currently exists, so I'm asking what I'd need to do to write one. You pointed at qamash, but that seems to be subclassing asyncio, so isn't "something that isn't asyncio". When Twisted is ported to Python 3, I'd be really surprised if it doesn't allow to use the new syntax. @inlineCallbacks implements a trampoline to make 'yields' work. This is a much slower approach than using 'yield from' (and 'await' from PEP 492). Not mentioning 'async with' and 'async for' features. (There shouldn't be a problem to support both @inlineCallbacks and PEP 492 approach, if I'm not missing something). Note that I don't have a problem with there being no existing implementation other than asyncio. I'd just like it if we could be clear over exactly what we mean when we say "the PEP is not tied to asyncio". Well, "the PEP is not tied to asyncio" -- this is correct. *The new syntax and new protocols know nothing about asyncio*. asyncio will know about the PEP by implementing new protocols where required etc (but supporting these new features isn't in the scope of the PEP). It feels like the truth currently is "you can write your own async framework that uses the new features introduced by the PEP". I fully expect that *if* there's a need for async frameworks that aren't fundamentally IO multiplexors, then it'll get easier to write them over time (the main problem right now is a lack of good tutorial examples of how to do so). But at the moment, asyncio seems to be the only game in town (and I can imagine that it'll always be the main IO multiplexor, unless existing frameworks like Twisted choose to compete rather than integrate). Agree. But if the existing frameworks choose to compete, or someone decides to write something better than asyncio, they can benefit from PEP 492. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: Please mention the Event Loop
Jim, On 2015-05-05 5:09 PM, Jim J. Jewett wrote: On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote: It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). [..] Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses "yield" as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. Thank you for this suggestion. I've added it to the PEP: https://hg.python.org/peps/rev/7ac132b24f1f Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 492 (async/await)
On 2015-05-05 7:53 PM, Guido van Rossum wrote: Everybody, In order to save myself a major headache I'm hereby accepting PEP 492. I've been following Yury's efforts carefully and I am fully confident that we're doing the right thing here. There is only so much effort we can put into clarifying terminology and explaining coroutines. Somebody should write a tutorial. (I started to write one, but I ran out of time after just describing basic yield.) Thank you, Guido! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
Hi Ben, On 2015-05-06 12:05 AM, Ben Darnell wrote: On Tue, May 5, 2015 at 3:25 PM, Yury Selivanov wrote: Yes, there is no other popular event loop for 3.4 other than asyncio, that uses coroutines based on generators (as far as I know). Tornado supports Python 3.4 and uses generator-based coroutines. We use `yield` instead of `yield from` for compatibility with Python 2. I have a patch to support the new async/await syntax here: https://github.com/bdarnell/tornado/commit/e3b71c3441e9f87a29a9b112901b7644b5b6edb8 I don't know how this happened, especially since I've used Tornado myself! It's amazing that Tornado will have support of async/await when 3.5 is out! Overall, I like the PEP. I've been reluctant to embrace `yield from` for Tornado coroutines (Tornado's Futures do not implement `__iter__`) because I'm worried about confusion between `yield` and `yield from`, but async and await are explicit enough that that's not really a problem. My one request would be that there be a type or ABC corresponding to inspect.isawaitable(). Tornado uses functools.singledispatch to handle interoperability with other coroutine frameworks, so it would be best if we could distinguish awaitables from other objects in a way that is compatible with singledispatch. The patch above simply registers types.GeneratorType which isn't quite correct. Sure. I'll add Awaitable and Coroutine ABCs. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
