[Python-Dev] Timeout for PEP 550 / Execution Context discussion
Hi, It looks like the discussion about the execution context became extremely hard to follow. There are many opinions on how the spec for generators should look like. What seems to be "natural" behaviour/example to one, seems to be completely unreasonable to other people. Recent emails from Guido indicate that he doesn't want to implement execution contexts for generators (at least in 3.7). In another thread Guido said this: "... Because coroutines and generators are similar under the covers, Yury demonstrated the issue with generators instead of coroutines (which are unfamiliar to many people). And then somehow we got hung up about fixing the problem in the example." And Guido is right. My initial motivation to write PEP 550 was to solve my own pain point, have a solution for async code. 'threading.local' is completely unusable there, but complex code bases demand a working solution. I thought that because coroutines and generators are so similar under the hood, I can design a simple solution that will cover all edge cases. Turns out it is not possible to do it in one pass. Therefore, in order to make some progress, I propose to split the problem in half: Stage 1. A new execution context PEP to solve the problem *just for async code*. The PEP will target Python 3.7 and completely ignore synchronous generators and asynchronous generators. It will be based on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an optimization) and borrow some good API decisions from PEP 550 v3+ (contextvars module, ContextVar class). The API (and C-API) will be designed to be future proof and ultimately allow transition to the stage 2. Stage 2. When Python 3.7 is out, we'll see how people use execution contexts for async code and collect feedback. If we recognize that Python users want execution contexts for generators/asynchronous generators, we'll make a new PEP to add support for them in Python 3.8. That future discussion will be focused on generators specifically, and therefore I expect it to be somewhat more focused. I will start working on the new PEP for stage 1 tomorrow. I expect to have a first version by the end of the week. I will also publish PEP 550 v1 as a separate PEP (as v1 is a totally different PEP anyways). Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion
On 16 October 2017 at 11:33, Yury Selivanov wrote: > Stage 2. When Python 3.7 is out, we'll see how people use execution > contexts for async code and collect feedback. If we recognize that > Python users want execution contexts for generators/asynchronous > generators, we'll make a new PEP to add support for them in Python > 3.8. That future discussion will be focused on generators > specifically, and therefore I expect it to be somewhat more focused. > As long as it's made clear that the interaction between context variables and generators is formally undefined in 3.7, I think that's reasonable - folks that want to ensure the current behaviour indefinitely should keep using thread locals rather than switching over to context variables. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion
On Sun, Oct 15, 2017 at 6:33 PM, Yury Selivanov wrote: > Hi, > > It looks like the discussion about the execution context became > extremely hard to follow. There are many opinions on how the spec for > generators should look like. What seems to be "natural" > behaviour/example to one, seems to be completely unreasonable to other > people. Recent emails from Guido indicate that he doesn't want to > implement execution contexts for generators (at least in 3.7). > > In another thread Guido said this: "... Because coroutines and > generators are similar under the covers, Yury demonstrated the issue > with generators instead of coroutines (which are unfamiliar to many > people). And then somehow we got hung up about fixing the problem in > the example." > > And Guido is right. My initial motivation to write PEP 550 was to > solve my own pain point, have a solution for async code. > 'threading.local' is completely unusable there, but complex code bases > demand a working solution. I thought that because coroutines and > generators are so similar under the hood, I can design a simple > solution that will cover all edge cases. Turns out it is not possible > to do it in one pass. > > Therefore, in order to make some progress, I propose to split the > problem in half: > > Stage 1. A new execution context PEP to solve the problem *just for > async code*. The PEP will target Python 3.7 and completely ignore > synchronous generators and asynchronous generators. It will be based > on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an > optimization) and borrow some good API decisions from PEP 550 v3+ > (contextvars module, ContextVar class). The API (and C-API) will be > designed to be future proof and ultimately allow transition to the > stage 2. If you want to ignore generators/async generators, then I think you don't even want PEP 550 v1, you just want something like a {set,get}_context_state API that lets you access the ThreadState's context dict (or rather, an opaque ContextState object that holds the context dict), and then task schedulers can call them at appropriate moments. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion
On Sun, Oct 15, 2017 at 8:12 PM, Nick Coghlan wrote: > On 16 October 2017 at 11:33, Yury Selivanov > wrote: > >> Stage 2. When Python 3.7 is out, we'll see how people use execution >> contexts for async code and collect feedback. If we recognize that >> Python users want execution contexts for generators/asynchronous >> generators, we'll make a new PEP to add support for them in Python >> 3.8. That future discussion will be focused on generators >> specifically, and therefore I expect it to be somewhat more focused. >> > > As long as it's made clear that the interaction between context variables > and generators is formally undefined in 3.7, I think that's reasonable - > folks that want to ensure the current behaviour indefinitely should keep > using thread locals rather than switching over to context variables. > It shouldn't be formally undefined. It should have the semantics it acquires when you combine the existing (well-defined) formal semantics of generators with the (to be defined) formal semantics of context variables. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion
On Sun, Oct 15, 2017 at 8:17 PM, Nathaniel Smith wrote: > On Sun, Oct 15, 2017 at 6:33 PM, Yury Selivanov > wrote: > > Stage 1. A new execution context PEP to solve the problem *just for > > async code*. The PEP will target Python 3.7 and completely ignore > > synchronous generators and asynchronous generators. It will be based > > on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an > > optimization) and borrow some good API decisions from PEP 550 v3+ > > (contextvars module, ContextVar class). The API (and C-API) will be > > designed to be future proof and ultimately allow transition to the > > stage 2. > > If you want to ignore generators/async generators, then I think you > don't even want PEP 550 v1, you just want something like a > {set,get}_context_state API that lets you access the ThreadState's > context dict (or rather, an opaque ContextState object that holds the > context dict), and then task schedulers can call them at appropriate > moments. > Yes, that's what I meant by "ignoring generators". And I'd like there to be a "current context" that's a per-thread MutableMapping with ContextVar keys. Maybe there's not much more to it apart from naming the APIs for getting and setting it? To be clear, I am fine with this being a specific subtype of MutableMapping. But I don't see much benefit in making it more abstract than that. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion
On Sun, Oct 15, 2017 at 10:10 PM, Guido van Rossum wrote: > On Sun, Oct 15, 2017 at 8:17 PM, Nathaniel Smith wrote: >> >> On Sun, Oct 15, 2017 at 6:33 PM, Yury Selivanov >> wrote: >> > Stage 1. A new execution context PEP to solve the problem *just for >> > async code*. The PEP will target Python 3.7 and completely ignore >> > synchronous generators and asynchronous generators. It will be based >> > on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an >> > optimization) and borrow some good API decisions from PEP 550 v3+ >> > (contextvars module, ContextVar class). The API (and C-API) will be >> > designed to be future proof and ultimately allow transition to the >> > stage 2. >> >> If you want to ignore generators/async generators, then I think you >> don't even want PEP 550 v1, you just want something like a >> {set,get}_context_state API that lets you access the ThreadState's >> context dict (or rather, an opaque ContextState object that holds the >> context dict), and then task schedulers can call them at appropriate >> moments. > > > Yes, that's what I meant by "ignoring generators". And I'd like there to be > a "current context" that's a per-thread MutableMapping with ContextVar keys. > Maybe there's not much more to it apart from naming the APIs for getting and > setting it? To be clear, I am fine with this being a specific subtype of > MutableMapping. But I don't see much benefit in making it more abstract than > that. We don't need it to be abstract (it's fine to have a single concrete mapping type that we always use internally), but I think we do want it to be opaque (instead of exposing the MutableMapping interface, the only way to get/set specific values should be through the ContextVar interface). The advantages are: - This allows C level caching of values in ContextVar objects (in particular, funneling mutations through a limited API makes cache invalidation *much* easier) - It gives us flexibility to change the underlying data structure without breaking API, or for different implementations to make different choices -- in particular, it's not clear whether a dict or HAMT is better, and it's not clear whether a regular dict or WeakKeyDict is better. The first point (caching) I think is the really compelling one: in practice decimal and numpy are already using tricky caching code to reduce the overhead of accessing the ThreadState dict, and this gets even trickier with context-local state which has more cache invalidation points, so if we don't do this in the interpreter then it could actually become a blocker for adoption. OTOH it's easy for the interpreter itself to do this caching, and it makes everyone faster. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com