[Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-15 Thread Yury Selivanov
Hi,

It looks like the discussion about the execution context became
extremely hard to follow.  There are many opinions on how the spec for
generators should look like.  What seems to be "natural"
behaviour/example to one, seems to be completely unreasonable to other
people.  Recent emails from Guido indicate that he doesn't want to
implement execution contexts for generators (at least in 3.7).

In another thread Guido said this: "... Because coroutines and
generators are similar under the covers, Yury demonstrated the issue
with generators instead of coroutines (which are unfamiliar to many
people). And then somehow we got hung up about fixing the problem in
the example."

And Guido is right.  My initial motivation to write PEP 550 was to
solve my own pain point, have a solution for async code.
'threading.local' is completely unusable there, but complex code bases
demand a working solution.  I thought that because coroutines and
generators are so similar under the hood, I can design a simple
solution that will cover all edge cases.  Turns out it is not possible
to do it in one pass.

Therefore, in order to make some progress, I propose to split the
problem in half:

Stage 1. A new execution context PEP to solve the problem *just for
async code*.  The PEP will target Python 3.7 and completely ignore
synchronous generators and asynchronous generators.  It will be based
on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an
optimization) and borrow some good API decisions from PEP 550 v3+
(contextvars module, ContextVar class).  The API (and C-API) will be
designed to be future proof and ultimately allow transition to the
stage 2.

Stage 2. When Python 3.7 is out, we'll see how people use execution
contexts for async code and collect feedback.  If we recognize that
Python users want execution contexts for generators/asynchronous
generators, we'll make a new PEP to add support for them in Python
3.8.  That future discussion will be focused on generators
specifically, and therefore I expect it to be somewhat more focused.

I will start working on the new PEP for stage 1 tomorrow.  I expect to
have a first version by the end of the week.

I will also publish PEP 550 v1 as a separate PEP (as v1 is a totally
different PEP anyways).

Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-15 Thread Nick Coghlan
On 16 October 2017 at 11:33, Yury Selivanov  wrote:

> Stage 2. When Python 3.7 is out, we'll see how people use execution
> contexts for async code and collect feedback.  If we recognize that
> Python users want execution contexts for generators/asynchronous
> generators, we'll make a new PEP to add support for them in Python
> 3.8.  That future discussion will be focused on generators
> specifically, and therefore I expect it to be somewhat more focused.
>

As long as it's made clear that the interaction between context variables
and generators is formally undefined in 3.7, I think that's reasonable -
folks that want to ensure the current behaviour indefinitely should keep
using thread locals rather than switching over to context variables.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-15 Thread Nathaniel Smith
On Sun, Oct 15, 2017 at 6:33 PM, Yury Selivanov  wrote:
> Hi,
>
> It looks like the discussion about the execution context became
> extremely hard to follow.  There are many opinions on how the spec for
> generators should look like.  What seems to be "natural"
> behaviour/example to one, seems to be completely unreasonable to other
> people.  Recent emails from Guido indicate that he doesn't want to
> implement execution contexts for generators (at least in 3.7).
>
> In another thread Guido said this: "... Because coroutines and
> generators are similar under the covers, Yury demonstrated the issue
> with generators instead of coroutines (which are unfamiliar to many
> people). And then somehow we got hung up about fixing the problem in
> the example."
>
> And Guido is right.  My initial motivation to write PEP 550 was to
> solve my own pain point, have a solution for async code.
> 'threading.local' is completely unusable there, but complex code bases
> demand a working solution.  I thought that because coroutines and
> generators are so similar under the hood, I can design a simple
> solution that will cover all edge cases.  Turns out it is not possible
> to do it in one pass.
>
> Therefore, in order to make some progress, I propose to split the
> problem in half:
>
> Stage 1. A new execution context PEP to solve the problem *just for
> async code*.  The PEP will target Python 3.7 and completely ignore
> synchronous generators and asynchronous generators.  It will be based
> on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an
> optimization) and borrow some good API decisions from PEP 550 v3+
> (contextvars module, ContextVar class).  The API (and C-API) will be
> designed to be future proof and ultimately allow transition to the
> stage 2.

If you want to ignore generators/async generators, then I think you
don't even want PEP 550 v1, you just want something like a
{set,get}_context_state API that lets you access the ThreadState's
context dict (or rather, an opaque ContextState object that holds the
context dict), and then task schedulers can call them at appropriate
moments.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-15 Thread Guido van Rossum
On Sun, Oct 15, 2017 at 8:12 PM, Nick Coghlan  wrote:

> On 16 October 2017 at 11:33, Yury Selivanov 
> wrote:
>
>> Stage 2. When Python 3.7 is out, we'll see how people use execution
>> contexts for async code and collect feedback.  If we recognize that
>> Python users want execution contexts for generators/asynchronous
>> generators, we'll make a new PEP to add support for them in Python
>> 3.8.  That future discussion will be focused on generators
>> specifically, and therefore I expect it to be somewhat more focused.
>>
>
> As long as it's made clear that the interaction between context variables
> and generators is formally undefined in 3.7, I think that's reasonable -
> folks that want to ensure the current behaviour indefinitely should keep
> using thread locals rather than switching over to context variables.
>

It shouldn't be formally undefined. It should have the semantics it
acquires when you combine the existing (well-defined) formal semantics of
generators with the (to be defined) formal semantics of context variables.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-15 Thread Guido van Rossum
On Sun, Oct 15, 2017 at 8:17 PM, Nathaniel Smith  wrote:

> On Sun, Oct 15, 2017 at 6:33 PM, Yury Selivanov 
> wrote:
> > Stage 1. A new execution context PEP to solve the problem *just for
> > async code*.  The PEP will target Python 3.7 and completely ignore
> > synchronous generators and asynchronous generators.  It will be based
> > on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an
> > optimization) and borrow some good API decisions from PEP 550 v3+
> > (contextvars module, ContextVar class).  The API (and C-API) will be
> > designed to be future proof and ultimately allow transition to the
> > stage 2.
>
> If you want to ignore generators/async generators, then I think you
> don't even want PEP 550 v1, you just want something like a
> {set,get}_context_state API that lets you access the ThreadState's
> context dict (or rather, an opaque ContextState object that holds the
> context dict), and then task schedulers can call them at appropriate
> moments.
>

Yes, that's what I meant by "ignoring generators". And I'd like there to be
a "current context" that's a per-thread MutableMapping with ContextVar
keys. Maybe there's not much more to it apart from naming the APIs for
getting and setting it? To be clear, I am fine with this being a specific
subtype of MutableMapping. But I don't see much benefit in making it more
abstract than that.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-15 Thread Nathaniel Smith
On Sun, Oct 15, 2017 at 10:10 PM, Guido van Rossum  wrote:
> On Sun, Oct 15, 2017 at 8:17 PM, Nathaniel Smith  wrote:
>>
>> On Sun, Oct 15, 2017 at 6:33 PM, Yury Selivanov 
>> wrote:
>> > Stage 1. A new execution context PEP to solve the problem *just for
>> > async code*.  The PEP will target Python 3.7 and completely ignore
>> > synchronous generators and asynchronous generators.  It will be based
>> > on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an
>> > optimization) and borrow some good API decisions from PEP 550 v3+
>> > (contextvars module, ContextVar class).  The API (and C-API) will be
>> > designed to be future proof and ultimately allow transition to the
>> > stage 2.
>>
>> If you want to ignore generators/async generators, then I think you
>> don't even want PEP 550 v1, you just want something like a
>> {set,get}_context_state API that lets you access the ThreadState's
>> context dict (or rather, an opaque ContextState object that holds the
>> context dict), and then task schedulers can call them at appropriate
>> moments.
>
>
> Yes, that's what I meant by "ignoring generators". And I'd like there to be
> a "current context" that's a per-thread MutableMapping with ContextVar keys.
> Maybe there's not much more to it apart from naming the APIs for getting and
> setting it? To be clear, I am fine with this being a specific subtype of
> MutableMapping. But I don't see much benefit in making it more abstract than
> that.

We don't need it to be abstract (it's fine to have a single concrete
mapping type that we always use internally), but I think we do want it
to be opaque (instead of exposing the MutableMapping interface, the
only way to get/set specific values should be through the ContextVar
interface). The advantages are:

- This allows C level caching of values in ContextVar objects (in
particular, funneling mutations through a limited API makes cache
invalidation *much* easier)

- It gives us flexibility to change the underlying data structure
without breaking API, or for different implementations to make
different choices -- in particular, it's not clear whether a dict or
HAMT is better, and it's not clear whether a regular dict or
WeakKeyDict is better.

The first point (caching) I think is the really compelling one: in
practice decimal and numpy are already using tricky caching code to
reduce the overhead of accessing the ThreadState dict, and this gets
even trickier with context-local state which has more cache
invalidation points, so if we don't do this in the interpreter then it
could actually become a blocker for adoption. OTOH it's easy for the
interpreter itself to do this caching, and it makes everyone faster.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com