Re: [Python-Dev] Best Python API for exposing posix_spawn

[Pablo Galindo Salgado]

Thank you everyone that commented in this thread about the best interface
for posix_spawn. I have finished implementing Antoine's suggestion in the
PR:

https://github.com/python/cpython/pull/5109

I think it would be good if we can have this merged before the feature lock
at the end of the month if possible.

Thanks you very much everyone for your time and suggestions!

On Wed, 10 Jan 2018, 09:17 Pablo Galindo Salgado, 
wrote:

> I think I really like Antoine's suggestion so I'm going to finish
> implementing it that way. I think this keeps the API simple, does not bring
> in the os module new dependencies, keeps the C implementation clean and is
> consistent with the rest of the posix module. I will post an update when is
> ready.
>
> Thank you everyone for sharing your view and advice!
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >=2.5.3

[Christian Heimes]

On 2018-01-16 08:08, Steve Dower wrote:
> From my perspective, we can’t keep an OpenSSL-like API and use Windows
> platform libraries (we *could* do a requests-like API easily enough, but
> even urllib3 is painfully low-level).
> 
>  
> 
> We have to continue shipping our own copy of OpenSSL on Windows. Nothing
> to negotiate here except whether OpenSSL releases should trigger a
> Python release, and I think that decision can stay with the RM.

3.7 will no longer use static linking. We can offer out-of-bounds
updates of the OpenSSL DLLs. And by "we", I'm talking about you. :)

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Wes Turner]

On Tuesday, January 16, 2018, Steve Dower  wrote:

> From my perspective, we can’t keep an OpenSSL-like API and use Windows
> platform libraries (we *could* do a requests-like API easily enough, but
> even urllib3 is painfully low-level).
>
> Support for Windows SChannel and Apple SecureTransport is part of the TLS
module.

IDK how far along that work is (whether it'll be ready for 3.7 beta 1)? Or
where those volunteering to help with the TLS module can send PRs?

https://github.com/python/peps/blob/master/pep-0543.rst

https://www.python.org/dev/peps/pep-0543/

http://markmail.org/search/?q=list%3Aorg.python+PEP+543+TLS

https://www.python.org/dev/peps/pep-0543/#interfaces

>
>
> We have to continue shipping our own copy of OpenSSL on Windows. Nothing
> to negotiate here except whether OpenSSL releases should trigger a Python
> release, and I think that decision can stay with the RM.
>
>
>
> Good luck solving macOS :o)
>
>
>
> Cheers,
>
> Steve
>
>
>
> Top-posted from my Windows phone
>
>
>
> *From: *Stephen J. Turnbull 
> *Sent: *Tuesday, January 16, 2018 17:45
> *To: *Matt Billenstein 
> *Cc: *Christian Heimes ; python-dev@python.org
> *Subject: *Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 /
> LibreSSL >=2.5.3
>
>
>
> Matt Billenstein writes:
>
>
>
> > In my mind it becomes easier to bundle deps in a binary installer
>
> > across the board (Linux, OSX, Windows) rather than rely on whatever
>
> > version the operating system provides.
>
>
>
> Thing is, as Christian points out, TLS is a rapidly moving target.
>
> Every Mac OS or iOS update seems to link to a dozen CVEs for TLS
>
> support.  We can go there if we have to, but it's often hard to go
>
> back when vendor support catches up to something reasonable.  I think
>
> this is something for Ned and Christian and Steve to negotiate, since
>
> they're the ones who are most aware of the tradeoffs and bear the
>
> costs.
>
>
>
>
>
>
>
> ___
>
> Python-Dev mailing list
>
> Python-Dev@python.org
>
> https://mail.python.org/mailman/listinfo/python-dev
>
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve.dower%40python.org
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Christian Heimes]

On 2018-01-16 12:28, Wes Turner wrote:
> 
> 
> On Tuesday, January 16, 2018, Steve Dower  > wrote:
> 
> From my perspective, we can’t keep an OpenSSL-like API and use
> Windows platform libraries (we *could* do a requests-like API easily
> enough, but even urllib3 is painfully low-level).
> 
> Support for Windows SChannel and Apple SecureTransport is part of the
> TLS module.
> 
> IDK how far along that work is (whether it'll be ready for 3.7 beta 1)?
> Or where those volunteering to help with the TLS module can send PRs?

You are misunderstanding the goal of PEP 543. It's not about providing
implementations of various backends. The PEP merely defines an minimal
abstraction layer. Neither the PEP nor the API are finalized or complete
yet, too Some parts of the PEP must be changed before it can be
finalized. Cory and I are discussion the matter.

Python 3.7's ssl module won't be compatible with PEP 543. For 3.8 it
*might* be possible to provide a 543 compatible implementation on top of
the ssl module.

I will not work on SChannel or SecureTransport, since I have neither
expertise, knowledge, interest or resources to work on other
implementations. AFAIK Steve would rather plug in Windows' cert
validation API into OpenSSL than to provide another TLS implementation.
For Apple ... no clue. How about you contact Apple support?

Regards,
Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Wes Turner]

On Tuesday, January 16, 2018, Christian Heimes  wrote:

> On 2018-01-16 12:28, Wes Turner wrote:
> >
> >
> > On Tuesday, January 16, 2018, Steve Dower  > > wrote:
> >
> > From my perspective, we can’t keep an OpenSSL-like API and use
> > Windows platform libraries (we *could* do a requests-like API easily
> > enough, but even urllib3 is painfully low-level).
> >
> > Support for Windows SChannel and Apple SecureTransport is part of the
> > TLS module.
> >
> > IDK how far along that work is (whether it'll be ready for 3.7 beta 1)?
> > Or where those volunteering to help with the TLS module can send PRs?
>
> You are misunderstanding the goal of PEP 543. It's not about providing
> implementations of various backends. The PEP merely defines an minimal
> abstraction layer. Neither the PEP nor the API are finalized or complete
> yet, too Some parts of the PEP must be changed before it can be
> finalized. Cory and I are discussion the matter.
>
> Python 3.7's ssl module won't be compatible with PEP 543. For 3.8 it
> *might* be possible to provide a 543 compatible implementation on top of
> the ssl module.

Got it. Thanks!

>
> I will not work on SChannel or SecureTransport, since I have neither
> expertise, knowledge, interest or resources to work on other
> implementations. AFAIK Steve would rather plug in Windows' cert
> validation API into OpenSSL than to provide another TLS implementation.
> For Apple ... no clue. How about you contact Apple support?


A HUP to their seclist about this work awhile back doesn't seem to have
upgraded OpenSSL.

Presumably there's another mailing list thread or GitHub issue for PEP 543
interface and implementation development.


>
> Regards,
> Christian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> wes.turner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Christian Heimes]

FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have
created a daily cronjob to populate Travis' cache with OpenSSL builds.
Until the cache is filled, Linux CI will take an extra 5 minute.

Christian

[1] https://github.com/python/cpython/pull/5180

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >=2.5.3

[Steve Dower]

Honestly, I’d rather plug into the WinHTTP API and just not even bother with 
sockets :)

Certificate validation is about the only thing broken in OpenSSL on Windows (as 
far as not working well with system config), and it’s relatively easy to 
replace with a couple of API calls. Now that we don’t statically link OpenSSL 
anymore, it can be done easily with ctypes, so I’ll probably put out a package 
for it sometime soon.

Top-posted from my Windows phone

From: Christian Heimes
Sent: Tuesday, January 16, 2018 22:52
To: python-dev@python.org
Subject: Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >=2.5.3

On 2018-01-16 12:28, Wes Turner wrote:
> 
> 
> On Tuesday, January 16, 2018, Steve Dower  > wrote:
> 
> From my perspective, we can’t keep an OpenSSL-like API and use
> Windows platform libraries (we *could* do a requests-like API easily
> enough, but even urllib3 is painfully low-level).
> 
> Support for Windows SChannel and Apple SecureTransport is part of the
> TLS module.
> 
> IDK how far along that work is (whether it'll be ready for 3.7 beta 1)?
> Or where those volunteering to help with the TLS module can send PRs?

You are misunderstanding the goal of PEP 543. It's not about providing
implementations of various backends. The PEP merely defines an minimal
abstraction layer. Neither the PEP nor the API are finalized or complete
yet, too Some parts of the PEP must be changed before it can be
finalized. Cory and I are discussion the matter.

Python 3.7's ssl module won't be compatible with PEP 543. For 3.8 it
*might* be possible to provide a 543 compatible implementation on top of
the ssl module.

I will not work on SChannel or SecureTransport, since I have neither
expertise, knowledge, interest or resources to work on other
implementations. AFAIK Steve would rather plug in Windows' cert
validation API into OpenSSL than to provide another TLS implementation.
For Apple ... no clue. How about you contact Apple support?

Regards,
Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL>=2.5.3

[Steve Dower]

I think you mean out-of-band updates, and by “you” I'm going to pretend you 
mean PyCA ;)

Top-posted from my Windows phone

From: Christian Heimes
Sent: Tuesday, January 16, 2018 21:42
To: python-dev@python.org
Subject: Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL>=2.5.3

On 2018-01-16 08:08, Steve Dower wrote:
> From my perspective, we can’t keep an OpenSSL-like API and use Windows
> platform libraries (we *could* do a requests-like API easily enough, but
> even urllib3 is painfully low-level).
> 
>  
> 
> We have to continue shipping our own copy of OpenSSL on Windows. Nothing
> to negotiate here except whether OpenSSL releases should trigger a
> Python release, and I think that decision can stay with the RM.

3.7 will no longer use static linking. We can offer out-of-bounds
updates of the OpenSSL DLLs. And by "we", I'm talking about you. :)

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 567 v3

[Yury Selivanov]

Hi,

This is a third version of PEP 567.

Changes from v2:

1. PyThreadState now references Context objects directly (instead of
referencing _ContextData).  This fixes out of sync Context.get() and
ContextVar.get().

2. Added a new Context.copy() method.

3. Renamed Token.old_val property to Token.old_value

4. ContextVar.reset(token) now raises a ValueError if the token was
created in a different Context.

5. All areas of the PEP were updated to be more precise. Context is
*no longer* defined as a read-only or an immutable mapping;
ContextVar.get() behaviour is fully defined; the immutability is only
mentioned in the Implementation section to avoid confusion; etc.

6. Added a new Examples section.

The reference implementation has been updated to include all these changes.

The only open question I personally have is whether ContextVar.reset()
should be idempotent or not.  Maybe we should be strict and raise an
error if a user tries to reset a variable more than once with the same
token object?

Other than that, I'm pretty happy with this version.  Big thanks to
everybody helping with the PEP!


PEP: 567
Title: Context Variables
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Dec-2017
Python-Version: 3.7
Post-History: 12-Dec-2017, 28-Dec-2017, 16-Jan-2018


Abstract


This PEP proposes a new ``contextvars`` module and a set of new
CPython C APIs to support context variables.  This concept is
similar to thread-local storage (TLS), but, unlike TLS, it also allows
correctly keeping track of values per asynchronous task, e.g.
``asyncio.Task``.

This proposal is a simplified version of :pep:`550`.  The key
difference is that this PEP is concerned only with solving the case
for asynchronous tasks, not for generators.  There are no proposed
modifications to any built-in types or to the interpreter.

This proposal is not strictly related to Python Context Managers.
Although it does provide a mechanism that can be used by Context
Managers to store their state.


Rationale
=

Thread-local variables are insufficient for asynchronous tasks that
execute concurrently in the same OS thread.  Any context manager that
saves and restores a context value using ``threading.local()`` will
have its context values bleed to other code unexpectedly when used
in async/await code.

A few examples where having a working context local storage for
asynchronous code is desirable:

* Context managers like ``decimal`` contexts and ``numpy.errstate``.

* Request-related data, such as security tokens and request
  data in web applications, language context for ``gettext``, etc.

* Profiling, tracing, and logging in large code bases.


Introduction


The PEP proposes a new mechanism for managing context variables.
The key classes involved in this mechanism are ``contextvars.Context``
and ``contextvars.ContextVar``.  The PEP also proposes some policies
for using the mechanism around asynchronous tasks.

The proposed mechanism for accessing context variables uses the
``ContextVar`` class.  A module (such as ``decimal``) that wishes to
use the new mechanism should:

* declare a module-global variable holding a ``ContextVar`` to
  serve as a key;

* access the current value via the ``get()`` method on the
  key variable;

* modify the current value via the ``set()`` method on the
  key variable.

The notion of "current value" deserves special consideration:
different asynchronous tasks that exist and execute concurrently
may have different values for the same key.  This idea is well-known
from thread-local storage but in this case the locality of the value is
not necessarily bound to a thread.  Instead, there is the notion of the
"current ``Context``" which is stored in thread-local storage.
Manipulation of the current context is the responsibility of the
task framework, e.g. asyncio.

A ``Context`` is a mapping of ``ContextVar`` objects to their values.
The ``Context`` itself exposes the ``abc.Mapping`` interface
(not ``abc.MutableMapping``!), so it cannot be modified directly.
To set a new value for a context variable in a ``Context`` object,
the user needs to:

* make the ``Context`` object "current" using the ``Context.run()``
  method;

* use ``ContextVar.set()`` to set a new value for the context
  variable.

The ``ContextVar.get()`` method looks for the variable in the current
``Context`` object using ``self`` as a key.

It is not possible to get a direct reference to the current ``Context``
object, but it is possible to obtain a shallow copy of it using the
``contextvars.copy_context()`` function.  This ensures that the
caller of ``Context.run()`` is the sole owner of its ``Context``
object.


Specification
=

A new standard library module ``contextvars`` is added with the
following APIs:

1. ``copy_context() -> Context`` function is used to get a copy of
   the current ``Context`` object for the current OS 

Re: [Python-Dev] PEP 567 v3

[Victor Stinner]

Hi Yury,

Thanks for the updated PEP v3, it's now much better than PEP v2!

I have no more complain against your PEP. I vote +1 for PEP 567 contextvars!

> The only open question I personally have is whether ContextVar.reset()
> should be idempotent or not.  Maybe we should be strict and raise an
> error if a user tries to reset a variable more than once with the same
> token object?

I don't think that it's worth it to prevent misuage of reset(). IMHO
it's fine if calling reset() twice reverts the variable state twice.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

On Tue, Jan 16, 2018 at 3:26 PM, Victor Stinner 
wrote:

> Thanks for the updated PEP v3, it's now much better than PEP v2!
>
> I have no more complain against your PEP. I vote +1 for PEP 567
> contextvars!
>

Yeah!

> The only open question I personally have is whether ContextVar.reset()
> > should be idempotent or not.  Maybe we should be strict and raise an
> > error if a user tries to reset a variable more than once with the same
> > token object?
>
> I don't think that it's worth it to prevent misuage of reset(). IMHO
> it's fine if calling reset() twice reverts the variable state twice.
>

Maybe the effect of calling it twice should be specified as undefined --
the implementation can try to raise in simple cases.

Unless Yury has a use case for the idempotency? (But with
__enter__/__exit__ as the main use case for reset() I wouldn't know what
the use case for idempotency would be.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Antoine Pitrou]

On Tue, 16 Jan 2018 17:44:14 -0500
Yury Selivanov  wrote:

> Offloading execution to other threads
> -
> 
> It is possible to run code in a separate OS thread using a copy
> of the current thread context::
> 
> executor = ThreadPoolExecutor()
> current_context = contextvars.copy_context()
> 
> executor.submit(
> lambda: current_context.run(some_function))

Does it also support offloading to a separate process (using
ProcessPoolExecutor in the example above)?  This would require the
Context to support pickling.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou  wrote:

> On Tue, 16 Jan 2018 17:44:14 -0500
> Yury Selivanov  wrote:
>
> > Offloading execution to other threads
> > -
> >
> > It is possible to run code in a separate OS thread using a copy
> > of the current thread context::
> >
> > executor = ThreadPoolExecutor()
> > current_context = contextvars.copy_context()
> >
> > executor.submit(
> > lambda: current_context.run(some_function))
>
> Does it also support offloading to a separate process (using
> ProcessPoolExecutor in the example above)?  This would require the
> Context to support pickling.
>

I don't think that's a requirement. The transparency between the two
different types of executor is mostly misleading anyway -- it's like the
old RPC transparency problem, which was never solved IIRC. There are just
too many things you need to be aware of before you can successfully offload
something to a different process.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Tue, Jan 16, 2018 at 6:53 PM, Guido van Rossum  wrote:
> On Tue, Jan 16, 2018 at 3:26 PM, Victor Stinner 
[..]
>> I don't think that it's worth it to prevent misuage of reset(). IMHO
>> it's fine if calling reset() twice reverts the variable state twice.
>
>
> Maybe the effect of calling it twice should be specified as undefined -- the
> implementation can try to raise in simple cases.
>
> Unless Yury has a use case for the idempotency? (But with __enter__/__exit__
> as the main use case for reset() I wouldn't know what the use case for
> idempotency would be.)

I don't have any use case for idempotent reset, so I'd change it to
raise an error on second call. We can always relax this in 3.8 if
people request it to be idempotent.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

Thanks, Victor!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum  wrote:
> On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou  wrote:
>>
>> On Tue, 16 Jan 2018 17:44:14 -0500
>> Yury Selivanov  wrote:
>>
>> > Offloading execution to other threads
>> > -
>> >
>> > It is possible to run code in a separate OS thread using a copy
>> > of the current thread context::
>> >
>> > executor = ThreadPoolExecutor()
>> > current_context = contextvars.copy_context()
>> >
>> > executor.submit(
>> > lambda: current_context.run(some_function))
>>
>> Does it also support offloading to a separate process (using
>> ProcessPoolExecutor in the example above)?  This would require the
>> Context to support pickling.
>
>
> I don't think that's a requirement. The transparency between the two
> different types of executor is mostly misleading anyway -- it's like the old
> RPC transparency problem, which was never solved IIRC. There are just too
> many things you need to be aware of before you can successfully offload
> something to a different process.

I agree.

I think it would be a very fragile thing In practice: if you have even
one variable in the context that isn't pickleable, your code that uses
a ProcessPool would stop working.  I would defer Context pickleability
to 3.8+.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nathaniel Smith]

On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov  wrote:
> On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum  wrote:
>> On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou  wrote:
>>>
>>> On Tue, 16 Jan 2018 17:44:14 -0500
>>> Yury Selivanov  wrote:
>>>
>>> > Offloading execution to other threads
>>> > -
>>> >
>>> > It is possible to run code in a separate OS thread using a copy
>>> > of the current thread context::
>>> >
>>> > executor = ThreadPoolExecutor()
>>> > current_context = contextvars.copy_context()
>>> >
>>> > executor.submit(
>>> > lambda: current_context.run(some_function))
>>>
>>> Does it also support offloading to a separate process (using
>>> ProcessPoolExecutor in the example above)?  This would require the
>>> Context to support pickling.
>>
>>
>> I don't think that's a requirement. The transparency between the two
>> different types of executor is mostly misleading anyway -- it's like the old
>> RPC transparency problem, which was never solved IIRC. There are just too
>> many things you need to be aware of before you can successfully offload
>> something to a different process.
>
> I agree.
>
> I think it would be a very fragile thing In practice: if you have even
> one variable in the context that isn't pickleable, your code that uses
> a ProcessPool would stop working.  I would defer Context pickleability
> to 3.8+.

There's also a more fundamental problem: you need some way to match up
the ContextVar objects across the two processes, and right now they
don't have an attached __module__ or __qualname__.

I guess we could do like namedtuple and (a) capture the module where
the ContextVar was instantiated, on the assumption that that's where
it will be stored, (b) require that users pass in the name of variable
where it will be stored as the 'name' argument to ContextVar.__init__.
I tend to agree that this is something to worry about for 3.8 though.
(If we need to retrofit pickle support, we could add a
pickleable=False argument to ContextVar, and require people to pass
pickleable=True to signal that they've done the appropriate setup to
make the ContextVar identifiable across processes, and that its
contents are safe to pickle.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nathaniel Smith]

On Tue, Jan 16, 2018 at 2:44 PM, Yury Selivanov  wrote:
> 4. ContextVar.reset(token) now raises a ValueError if the token was
> created in a different Context.

A minor bit of polish: given that Token objects have to track the
associated ContextVar anyway, I think it'd be cleaner if instead of
doing:

token = cvar.set(...)
cvar.reset(token)

we made the API be:

token = cvar.set(...)
token.reset()

In the first version, we use 'cvar' twice, and it's a mandatory
invariant that the same ContextVar object gets used in both places;
you had to add extra code to check this and raise an error if that's
violated. It's level 5 on Rusty's scale
(http://sweng.the-davies.net/Home/rustys-api-design-manifesto)

In the second version, the ContextVar is only mentioned once, so the
invariant is automatically enforced by the API -- you can't even
express the broken version. That's level 10 on Rusty's scale, and
gives a simpler implementation too.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Tue, Jan 16, 2018 at 8:27 PM, Nathaniel Smith  wrote:
[..]
> token = cvar.set(...)
> token.reset()

I see the point, but I think that having the 'reset' method defined on
the ContextVar class is easier to grasp.  It also feels natural that a
pair of set/reset methods is defined on the same class.  This is
highly subjective though, so let's see which option Guido likes more.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

On Tue, Jan 16, 2018 at 5:33 PM, Yury Selivanov 
wrote:

> On Tue, Jan 16, 2018 at 8:27 PM, Nathaniel Smith  wrote:
> [..]
> > token = cvar.set(...)
> > token.reset()
>
> I see the point, but I think that having the 'reset' method defined on
> the ContextVar class is easier to grasp.  It also feels natural that a
> pair of set/reset methods is defined on the same class.  This is
> highly subjective though, so let's see which option Guido likes more.
>

I think this came up in one of the previous reviews of the PEP. I like
Yury's (redundant) version -- it makes it clear to the human reader of the
code which variable is being reset. And it's not like it's going to be used
that much -- it'll be likely hidden inside a context manager.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nick Coghlan]

On 17 January 2018 at 11:27, Nathaniel Smith  wrote:
> On Tue, Jan 16, 2018 at 2:44 PM, Yury Selivanov  
> wrote:
>> 4. ContextVar.reset(token) now raises a ValueError if the token was
>> created in a different Context.
>
> A minor bit of polish: given that Token objects have to track the
> associated ContextVar anyway, I think it'd be cleaner if instead of
> doing:
>
> token = cvar.set(...)
> cvar.reset(token)
>
> we made the API be:
>
> token = cvar.set(...)
> token.reset()

As a counterpoint to this, consider the case where you're working with
*two* cvars:


token1 = cvar1.set(...)
token2 = cvar2.set(...)
...
cvar1.reset(token1)
...
cvar2.reset(token2)

At the point where the resets happen, you know exactly which cvar is
being reset, even if you don't know where the token was created.

With reset-on-the-token, you're entirely reliant on variable naming to
know which ContextVar is going to be affected:

token1 = cvar1.set(...)
token2 = cvar2.set(...)
...
token1.reset() # Resets cvar1
...
token2.reset() # Resets cvar2

If someone really does want an auto-reset API, it's also fairly easy
to build atop the more explicit one:

def set_cvar(cvar, value):
token = cvar.set(value)
return functools.partial(cvar.reset, token)

reset_cvar1 = set_cvar(cvar1, ...)
...
reset_cvar1()

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com