[Python-Dev] Best Python API for exposing posix_spawn

[Pablo Galindo Salgado]

Hi,

I'm currently working on exposing posix_spawn in the posix module (and by
extension in the os module). You can find the initial implementation in
this PR:

https://github.com/python/cpython/pull/5109

As pointed out by Gregory P. Smith, some changes are needed in the way the
file_actions arguments is passed from Python. For context, posix_spawn has
the following declaration:

int posix_spawn(pid_t *pid, const char *path,
const posix_spawn_file_actions_t *file_actions,
const posix_spawnattr_t *attrp,
char *const argv[], char *const envp[]);

Here, file_actions is an object that represents a list of file actions
(open, close or dup2) that is populated using helper functions on the C API.

The question is: what is the best way to deal with this argument?

Following Gregory's comment on the PR I understand that he is proposing to
have three objects in the os module representing each action and pass a
sequence of these objects to the Python API. What I am not sure about this
is that there is no previous example of such classes in the os module for
other similar APIs and therefore I am not sure if there is a better
approach.

Thanks you very much for your time!

Pablo Galindo
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 567 pre v3

[Yury Selivanov]

Hi,

Thanks to everybody participating in the PEP 567 discussion!  I want
to summarize a few topics to make sure that we are all on the same
page (and maybe provoke more discussion).


1. Proposal: ContextVar has default set to None.

>From the typing point of view that would mean that if a context
variable is declared without an explicit default, its type would be
Optional.  E.g. say we have a hypothetical web framework that allows
to access the current request object through a context variable:

  request_var: ContextVar[Optional[Request]] = \
  ContextVar('current_request')

When we need to get the current request object, we would write:

  request: Optional[Request] = request_var.get()

And we'd also need to explicitly handle when 'request' is set to None.
Of course we could create request_var with its default set to some
"InvalidRequest" object, but that would complicate things.  It would
be easier to just state that the framework always sets the current
request and it's a bug if it's not set.

Therefore, in my opinion, it's better to keep the current behaviour:
if a context variable was created without a default value,
ContextVar.get() can raise a LookupError.


2. Context.__contains__, Context.__getitem__ and ContexVar.default

So if we keep the current PEP 567 behaviour w.r.t. defaults,
ContextVar.get() might return a different value from Context.get():

v = ContextVar('v', default=42)
ctx = contextvars.copy_context()

ctx.get(v)   # returns None
v.get()   # returns 42
v in ctx  # returns False

I think this discrepancy is OK.  Context is a mapping-like object and
it reflects the contents of the underlying _ContextData mapping
object.

ContextVar.default is meant to be used only by ContextVar.get().
Context objects should not use it.

Maybe we can rename ContextVar.get() to ContextVar.lookup()?  This
would help to avoid potential confusion between Context.get() and
ContextVar.get().


3. Proposal: Context.get() and __getitem__() should always return up
to date values.

The issue with the current PEP 567 design is that PyThreadState points
to a _ContextData object, and not to the current Context.  The
following code illustrates how this manifests in Python code:

v = ContextVar('v')

def foo():
v.set(42)
print(v.get(), ctx.get(v, 'missing'))

ctx = Context()
ctx.run(foo)

The above code will print "42 missing", because 'ctx' points to an
outdated _ContextData.

This is easily fixable if we make PyThreadState to point to the
current Context object (instead of it pointing to a _ContextData).
This change will also make "contextvars.copy_context()" easier to
understand--it will actually return a copy of the current context that
the thread state points to.

Adding a private Context._in_use attribute would allow us to make sure
that Context.run() cannot be simultaneously called in two OS threads.
As Nathaniel points out, this will also simplify cache implementation
in ContextVar.get().  So let's do this.


4. Add Context.copy().

I was actually going to suggest this addition myself.  With the
current PEP 567 design, Context.copy() can be implemented with
"ctx.run(contextvars.copy_context)", but this is very cumbersome.

An example of when a copy() method could be useful is capturing the
current context and executing a few functions with it using
ThreadPoolExecutor.map().  Copying the Context object will ensure that
every mapped function executes in its own context copy (i.e.
isolated).  So I'm +1 for this one.


5. PEP language.

I agree that PEP is vague about some details and is incorrect in some
places (like calling Context objects immutable, which is not really
true, because .run() can modify them). I'll fix the language in v3
once I'm back home.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Brett Cannon]

On Mon, 8 Jan 2018 at 07:57 Pablo Galindo Salgado 
wrote:

> Hi,
>
> I'm currently working on exposing posix_spawn in the posix module (and by
> extension in the os module). You can find the initial implementation in
> this PR:
>
> https://github.com/python/cpython/pull/5109
>
> As pointed out by Gregory P. Smith, some changes are needed in the way the
> file_actions arguments is passed from Python. For context, posix_spawn has
> the following declaration:
>
> int posix_spawn(pid_t *pid, const char *path,
> const posix_spawn_file_actions_t *file_actions,
> const posix_spawnattr_t *attrp,
> char *const argv[], char *const envp[]);
>
> Here, file_actions is an object that represents a list of file actions
> (open, close or dup2) that is populated using helper functions on the C API.
>
> The question is: what is the best way to deal with this argument?
>
> Following Gregory's comment on the PR I understand that he is proposing to
> have three objects in the os module representing each action and pass a
> sequence of these objects to the Python API. What I am not sure about this
> is that there is no previous example of such classes in the os module for
> other similar APIs and therefore I am not sure if there is a better
> approach.
>
> Thanks you very much for your time!
>

Any chance of using an enum?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Serhiy Storchaka]

08.01.18 11:11, Pablo Galindo Salgado пише:
Following Gregory's comment on the PR I understand that he is proposing 
to have three objects in the os module representing each action and pass 
a sequence of these objects to the Python API. What I am not sure about 
this is that there is no previous example of such classes in the os 
module for other similar APIs and therefore I am not sure if there is a 
better approach.


I would pass a sequence like:

[(os.close, 0),
 (os.open, 1, '/tmp/mylog', os.O_WRONLY, 0o700),
 (os.dup2, 1, 2),
]

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[eryk sun]

On Sun, Jan 7, 2018 at 6:48 PM, Christian Tismer  wrote:
> That is true.
> list2cmdline escapes partially, but on NT and Windows10, the "^" must
> also be escaped, but is not. The "|" pipe symbol must also be escaped
> by "^", as many others as well.
>
> The effect was that passing a rexexp as parameter to a windows program
> gave me strange effects, and I recognized that "^" was missing.
>
> So I was asking for a coherent solution:
> Escape things completely or omit "shell=True".
>
> Yes, there is a list of chars to escape, and it is Windows version
> dependent. I can provide it if it makes sense.

subprocess.list2cmdline is meant to help support cross-platform code,
since Windows uses a command-line instead of an argv array. The
command-line parsing rules used by VC++ (and CommandLineToArgvW) are
the most common in practice. list2cmdline is intended for this set of
applications. Otherwise pass args as a string instead of a list.

In CMD we can quote part of a command line in double quotes to escape
special characters. The quotes are preserved in the application
command line. This can get complicated when we need to preserve
literal quotes in the command line of an application that uses VC++
backslash escaping. CMD doesn't recognize backslash as an escape
character, which gives rise to a quoting conflict between CMD and the
application. Some applications support translating single quotes to
double quotes in this case (e.g. schtasks.exe). Single quotes
generally aren't used in CMD, except in a `for /f` loop, but this can
be forced to use backquotes instead via `usebackq`.

Quoting doesn't escape the percent character that's used for
environment variables. In batch scripts percent can be escaped by
doubling it, but not in /c commands. Some applications can translate a
substitute character in this case, such as "~" (e.g. setx.exe).
Otherwise, we can usually disrupt matching an existing variable by
adding a "^" character after the first percent character. The "^"
escape character gets consumed later on in parsing -- as long as it's
not quoted (see the previous paragraph for complications).
Nonetheless, "^" is a valid name character, so there's still a
possibility of matching an environment variable (perhaps a malicious
one).  For example:

C:\>python -c "print('"%^"time%')"
%time%

C:\>set "^"time=spam"
C:\>python -c "print('"%^"time%')"
spam

Anyway, we're supposed to pass args as a string when using the shell
in POSIX, so we may as well stay consistent with this in Windows.
Practically no one wants the resulting behavior when passing a shell
command as a list in POSIX. For example:

>>> subprocess.call(['echo \\$0=$0 \\$1=$1', 'spam', 'eggs'], shell=True)
$0=spam $1=eggs

It's common to discourage using `shell=True` because it's considered
insecure. One of the reasons to use CMD in Windows is that it tries
ShellExecuteEx if CreateProcess fails. ShellExecuteEx supports "App
Paths" commands, file actions (open, edit, print), UAC elevation (via
"runas" or if requested by the manifest), protocols (including
"shell:"), and opening folders in Explorer. It isn't a scripting
language, however, so it doesn't pose the same risk as using CMD.
Calling ShellExecuteEx could be integrated in subprocess as a new
Popen parameter, such as `winshell` or `shellex`.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Steve Dower]

On 09Jan2018 0744, eryk sun wrote:

It's common to discourage using `shell=True` because it's considered
insecure. One of the reasons to use CMD in Windows is that it tries
ShellExecuteEx if CreateProcess fails. ShellExecuteEx supports "App
Paths" commands, file actions (open, edit, print), UAC elevation (via
"runas" or if requested by the manifest), protocols (including
"shell:"), and opening folders in Explorer. It isn't a scripting
language, however, so it doesn't pose the same risk as using CMD.
Calling ShellExecuteEx could be integrated in subprocess as a new
Popen parameter, such as `winshell` or `shellex`.


This can also be used directly as os.startfile, the only downside being 
that you can't wait for the process to complete (but that's due to the 
underlying API, which may not end up starting a process but rather 
sending a message to an existing long-running one such as explorer.exe). 
I'd certainly recommend it for actions like "open this file with its 
default editor" or "browse to this web page with the default browser".


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Victor Stinner]

Le 8 janv. 2018 8:36 PM, "Yury Selivanov"  a
écrit :

2. Context.__contains__, Context.__getitem__ and ContexVar.default

So if we keep the current PEP 567 behaviour w.r.t. defaults,
ContextVar.get() might return a different value from Context.get():

v = ContextVar('v', default=42)
ctx = contextvars.copy_context()

ctx.get(v)   # returns None
v.get()   # returns 42
v in ctx  # returns False

I think this discrepancy is OK.  Context is a mapping-like object and
it reflects the contents of the underlying _ContextData mapping
object.


ctx[var] raises an exception but ctx.get(var) returns None in such case. My
point is just that Context.get() behaves differently than dict.get(). If
dict[key] raises, I expect that dict.get() raises too and that I have to
write explicitely dict.get(default=None).

I suggest to modify Context.get() to raise an exception or require to
explicitely write ctx.get(var, default=None).

ContextVar.default is meant to be used only by ContextVar.get().
Context objects should not use it.


I now agree. The difference between ContextVar.get() and Context.get() is
fine and can be explained.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Nathaniel Smith]

On Mon, Jan 8, 2018 at 2:35 PM, Victor Stinner  wrote:
> ctx[var] raises an exception but ctx.get(var) returns None in such case. My
> point is just that Context.get() behaves differently than dict.get(). If
> dict[key] raises, I expect that dict.get() raises too and that I have to
> write explicitely dict.get(default=None).

But that's not how dict.get works?

In [1]: d = {}

In [2]: print(d.get(1))
None

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Gregory P. Smith]

On Mon, Jan 8, 2018 at 12:36 PM Serhiy Storchaka 
wrote:

> 08.01.18 11:11, Pablo Galindo Salgado пише:
> > Following Gregory's comment on the PR I understand that he is proposing
> > to have three objects in the os module representing each action and pass
> > a sequence of these objects to the Python API. What I am not sure about
> > this is that there is no previous example of such classes in the os
> > module for other similar APIs and therefore I am not sure if there is a
> > better approach.
>
> I would pass a sequence like:
>
> [(os.close, 0),
>   (os.open, 1, '/tmp/mylog', os.O_WRONLY, 0o700),
>   (os.dup2, 1, 2),
> ]
>

i agree with just a list of tuples, but i suggest creating namedtuple
instances in the posix module for the purpose (one each for close, dup2,
open) .  Don't put a reference to a function in the tuple as Serhiy
suggested as, while obvious what it means, it gives the wrong impression to
the user: nothing is calling the Python functions.  This is a posix API
that takes a list of arguments for a specific set of system calls for _it_
to make for us in a specific order.

-gps


>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Victor Stinner]

Hum, now I'm confused. I was probably confused by ContextVar.get()
differences with Context.get().

It's fine if it behaves with a dict.

Victor

Le 9 janv. 2018 12:02 AM, "Nathaniel Smith"  a écrit :

> On Mon, Jan 8, 2018 at 2:35 PM, Victor Stinner 
> wrote:
> > ctx[var] raises an exception but ctx.get(var) returns None in such case.
> My
> > point is just that Context.get() behaves differently than dict.get(). If
> > dict[key] raises, I expect that dict.get() raises too and that I have to
> > write explicitely dict.get(default=None).
>
> But that's not how dict.get works?
>
> In [1]: d = {}
>
> In [2]: print(d.get(1))
> None
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Guido van Rossum]

I am +1 on everything Yury says here.

On Mon, Jan 8, 2018 at 11:34 AM, Yury Selivanov 
wrote:

> Hi,
>
> Thanks to everybody participating in the PEP 567 discussion!  I want
> to summarize a few topics to make sure that we are all on the same
> page (and maybe provoke more discussion).
>
>
> 1. Proposal: ContextVar has default set to None.
>
> From the typing point of view that would mean that if a context
> variable is declared without an explicit default, its type would be
> Optional.  E.g. say we have a hypothetical web framework that allows
> to access the current request object through a context variable:
>
>   request_var: ContextVar[Optional[Request]] = \
>   ContextVar('current_request')
>
> When we need to get the current request object, we would write:
>
>   request: Optional[Request] = request_var.get()
>
> And we'd also need to explicitly handle when 'request' is set to None.
> Of course we could create request_var with its default set to some
> "InvalidRequest" object, but that would complicate things.  It would
> be easier to just state that the framework always sets the current
> request and it's a bug if it's not set.
>
> Therefore, in my opinion, it's better to keep the current behaviour:
> if a context variable was created without a default value,
> ContextVar.get() can raise a LookupError.
>
>
> 2. Context.__contains__, Context.__getitem__ and ContexVar.default
>
> So if we keep the current PEP 567 behaviour w.r.t. defaults,
> ContextVar.get() might return a different value from Context.get():
>
> v = ContextVar('v', default=42)
> ctx = contextvars.copy_context()
>
> ctx.get(v)   # returns None
> v.get()   # returns 42
> v in ctx  # returns False
>
> I think this discrepancy is OK.  Context is a mapping-like object and
> it reflects the contents of the underlying _ContextData mapping
> object.
>
> ContextVar.default is meant to be used only by ContextVar.get().
> Context objects should not use it.
>
> Maybe we can rename ContextVar.get() to ContextVar.lookup()?  This
> would help to avoid potential confusion between Context.get() and
> ContextVar.get().
>
>
> 3. Proposal: Context.get() and __getitem__() should always return up
> to date values.
>
> The issue with the current PEP 567 design is that PyThreadState points
> to a _ContextData object, and not to the current Context.  The
> following code illustrates how this manifests in Python code:
>
> v = ContextVar('v')
>
> def foo():
> v.set(42)
> print(v.get(), ctx.get(v, 'missing'))
>
> ctx = Context()
> ctx.run(foo)
>
> The above code will print "42 missing", because 'ctx' points to an
> outdated _ContextData.
>
> This is easily fixable if we make PyThreadState to point to the
> current Context object (instead of it pointing to a _ContextData).
> This change will also make "contextvars.copy_context()" easier to
> understand--it will actually return a copy of the current context that
> the thread state points to.
>
> Adding a private Context._in_use attribute would allow us to make sure
> that Context.run() cannot be simultaneously called in two OS threads.
> As Nathaniel points out, this will also simplify cache implementation
> in ContextVar.get().  So let's do this.
>
>
> 4. Add Context.copy().
>
> I was actually going to suggest this addition myself.  With the
> current PEP 567 design, Context.copy() can be implemented with
> "ctx.run(contextvars.copy_context)", but this is very cumbersome.
>
> An example of when a copy() method could be useful is capturing the
> current context and executing a few functions with it using
> ThreadPoolExecutor.map().  Copying the Context object will ensure that
> every mapped function executes in its own context copy (i.e.
> isolated).  So I'm +1 for this one.
>
>
> 5. PEP language.
>
> I agree that PEP is vague about some details and is incorrect in some
> places (like calling Context objects immutable, which is not really
> true, because .run() can modify them). I'll fix the language in v3
> once I'm back home.
>
>
> Yury
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Random832]

On Mon, Jan 8, 2018, at 18:05, Gregory P. Smith wrote:
> i agree with just a list of tuples, but i suggest creating namedtuple
> instances in the posix module for the purpose (one each for close, dup2,
> open) .  Don't put a reference to a function in the tuple as Serhiy
> suggested as, while obvious what it means, it gives the wrong impression to
> the user: nothing is calling the Python functions.  This is a posix API
> that takes a list of arguments for a specific set of system calls for _it_
> to make for us in a specific order.

Instead of a sequence of functions to call, it'd be nice if a higher-level API 
could allow just passing in a mapping of file descriptor numbers to what they 
should point to in the new process, and the implementation figures out what 
sequence is necessary to get that result.

And at that point we could just extend the subprocess API to allow redirection 
of file descriptors other than 0/1/2, and have an implementation of it in terms 
of posix_spawn.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Gregory P. Smith]

On Mon, Jan 8, 2018 at 4:03 PM Random832  wrote:

> On Mon, Jan 8, 2018, at 18:05, Gregory P. Smith wrote:
> > i agree with just a list of tuples, but i suggest creating namedtuple
> > instances in the posix module for the purpose (one each for close, dup2,
> > open) .  Don't put a reference to a function in the tuple as Serhiy
> > suggested as, while obvious what it means, it gives the wrong impression
> to
> > the user: nothing is calling the Python functions.  This is a posix API
> > that takes a list of arguments for a specific set of system calls for
> _it_
> > to make for us in a specific order.
>
> Instead of a sequence of functions to call, it'd be nice if a higher-level
> API could allow just passing in a mapping of file descriptor numbers to
> what they should point to in the new process, and the implementation
> figures out what sequence is necessary to get that result.
>
> And at that point we could just extend the subprocess API to allow
> redirection of file descriptors other than 0/1/2, and have an
> implementation of it in terms of posix_spawn.
>

sure, but high level APIs don't belong in the os/posix module.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Brett Cannon]

On Mon, 8 Jan 2018 at 15:06 Gregory P. Smith  wrote:

> On Mon, Jan 8, 2018 at 12:36 PM Serhiy Storchaka 
> wrote:
>
>> 08.01.18 11:11, Pablo Galindo Salgado пише:
>> > Following Gregory's comment on the PR I understand that he is proposing
>> > to have three objects in the os module representing each action and pass
>> > a sequence of these objects to the Python API. What I am not sure about
>> > this is that there is no previous example of such classes in the os
>> > module for other similar APIs and therefore I am not sure if there is a
>> > better approach.
>>
>> I would pass a sequence like:
>>
>> [(os.close, 0),
>>   (os.open, 1, '/tmp/mylog', os.O_WRONLY, 0o700),
>>   (os.dup2, 1, 2),
>> ]
>>
>
> i agree with just a list of tuples, but i suggest creating namedtuple
> instances in the posix module for the purpose (one each for close, dup2,
> open) .
>

I a namedtuple really necessary for this versus a simple object? There is
no backwards-compatibility here with an old tuple-based interface so
supporting both tuples and named access doesn't seem necessary to me.

-Brett


>   Don't put a reference to a function in the tuple as Serhiy suggested as,
> while obvious what it means, it gives the wrong impression to the user:
> nothing is calling the Python functions.  This is a posix API that takes a
> list of arguments for a specific set of system calls for _it_ to make for
> us in a specific order.
>
> -gps
>
>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Nick Coghlan]

On 9 January 2018 at 05:34, Yury Selivanov  wrote:
> Maybe we can rename ContextVar.get() to ContextVar.lookup()?  This
> would help to avoid potential confusion between Context.get() and
> ContextVar.get().

I think this would also tie in nicely with the PEP 568 draft, where
"ContextVar.lookup()" may end up scanning a chain of Context mappings
before falling back on the given default value.

That said, I do wonder if this may be a case where a dual API might be
appropriate (ala dict.__getitem__ vs dict.get), such that you have:

ContextVar.get(default=None) -> Optional[T] # Missing -> None
ContextVar.lookup() -> T # Missing -> raise LookupError

If you set a default on the ContextVar itself, they'd always be
identical (since you'll never hit the "Missing" case), but they'd
mimic the dict.__getitem__ vs dict.get split if no var level default
was specified.

The conservative option would be to start with only the
`ContextVar.lookup` method, and then add `ContextVar.get` later if
it's absence proved sufficiently irritating.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Nick Coghlan]

On 8 January 2018 at 19:11, Pablo Galindo Salgado  wrote:
> Following Gregory's comment on the PR I understand that he is proposing to
> have three objects in the os module representing each action and pass a
> sequence of these objects to the Python API. What I am not sure about this
> is that there is no previous example of such classes in the os module for
> other similar APIs and therefore I am not sure if there is a better
> approach.

Probably the closest prior art would be the os.DirEntry objects used
for the items yielded from os.scandir - that is the same general idea
(a dedicated Python class to represent a C struct), just in the other
direction.

As with DirEntry, I don't see any obvious value in making the new
objects iterable though - we should be able to just use named field
access in both the C and Python APIs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Guido van Rossum]

When I +1'ed Yury's message I forgot about this issue. I actually prefer
the current PEP 567 version -- .get() raises an error if there's no default
on the ContextVar, and .get(None) returns None if there's no default. The
idea here is that by far the most common use will be .get(), so it should
be a short name. (In that sense it's similar to the Queue API.)



On Mon, Jan 8, 2018 at 7:22 PM, Nick Coghlan  wrote:

> On 9 January 2018 at 05:34, Yury Selivanov 
> wrote:
> > Maybe we can rename ContextVar.get() to ContextVar.lookup()?  This
> > would help to avoid potential confusion between Context.get() and
> > ContextVar.get().
>
> I think this would also tie in nicely with the PEP 568 draft, where
> "ContextVar.lookup()" may end up scanning a chain of Context mappings
> before falling back on the given default value.
>
> That said, I do wonder if this may be a case where a dual API might be
> appropriate (ala dict.__getitem__ vs dict.get), such that you have:
>
> ContextVar.get(default=None) -> Optional[T] # Missing -> None
> ContextVar.lookup() -> T # Missing -> raise LookupError
>
> If you set a default on the ContextVar itself, they'd always be
> identical (since you'll never hit the "Missing" case), but they'd
> mimic the dict.__getitem__ vs dict.get split if no var level default
> was specified.
>
> The conservative option would be to start with only the
> `ContextVar.lookup` method, and then add `ContextVar.get` later if
> it's absence proved sufficiently irritating.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 pre v3

[Nathaniel Smith]

On Mon, Jan 8, 2018 at 11:34 AM, Yury Selivanov  wrote:
> 1. Proposal: ContextVar has default set to None.
>
> From the typing point of view that would mean that if a context
> variable is declared without an explicit default, its type would be
> Optional.  E.g. say we have a hypothetical web framework that allows
> to access the current request object through a context variable:
>
>   request_var: ContextVar[Optional[Request]] = \
>   ContextVar('current_request')
>
> When we need to get the current request object, we would write:
>
>   request: Optional[Request] = request_var.get()
>
> And we'd also need to explicitly handle when 'request' is set to None.
> Of course we could create request_var with its default set to some
> "InvalidRequest" object, but that would complicate things.  It would
> be easier to just state that the framework always sets the current
> request and it's a bug if it's not set.
>
> Therefore, in my opinion, it's better to keep the current behaviour:
> if a context variable was created without a default value,
> ContextVar.get() can raise a LookupError.

All the different behaviors here can work, so I don't want to make a
huge deal about this. But the current behavior is bugging me, and I
don't think anyone has brought up the reason why, so here goes :-).

Right now, the set of valid states for a ContextVar are: it can hold
any Python object, or it can be undefined. However, the only way it
can be in the "undefined" state is in a new Context where it has never
had a value; once it leaves the undefined state, it can never return
to it.

This makes me itch. It's very weird to have a mutable variable with a
valid state that you can't reach by mutating it. I see two
self-consistent ways to make me stop itching: (a) double-down on
undefined as being part of ContextVar's domain, or (b) reduce the
domain so that undefined is never a valid state.

# Option 1

In the first approach, we conceptualize ContextVar as being a
container that either holds a value or is empty (and then there's one
of these containers for each context). We also want to be able to
define an initial value that the container takes on when a new context
materializes, because that's really convenient. And then after that we
provide ways to get the value (if present), or control the value
(either set it to a particular value or unset it). So something like:

var1 = ContextVar("var1")  # no initial value
var2 = ContextVar("var2", initial_value="hello")

with assert_raises(SomeError):
var1.get()
# get's default lets us give a different outcome in cases where it
would otherwise raise
assert var1.get(None) is None
assert var2.get() == "hello"
# If get() doesn't raise, then the argument is ignored
assert var2.get(None) == "hello"

# We can set to arbitrary values
for var in [var1, var2]:
var.set("new value")
assert var.get() == "new value"

# We can unset again, so get() will raise
for var in [var1, var2]:
var.unset()
with assert_raises(SomeError):
var.get()
assert var.get(None) is None

To fulfill all that, we need an implementation like:

MISSING = make_sentinel()

class ContextVar:
def __init__(self, name, *, initial_value=MISSING):
self.name = name
self.initial_value = initial_value

def set(self, value):
if value is MISSING: raise TypeError
current_context()._dict[self] = value
# Token handling elided because it's orthogonal to this issue
return Token(...)

def unset(self):
current_context()._dict[self] = MISSING
# Token handling elided because it's orthogonal to this issue
return Token(...)

def get(self, default=_NOT_GIVEN):
value = current_context().get(self, self.initial_value)
if value is MISSING:
if default is _NOT_GIVEN:
raise ...
else:
return default
else:
return value

Note that the implementation here is somewhat tricky and non-obvious.
In particular, to preserve the illusion of a simple container with an
optional initial value, we have to encode a logically undefined
ContextVar as one that has Context[var] set to MISSING, and a missing
entry in Context encodes the presence of the inital value. If we
defined unset() as 'del current_context._dict[self]', then we'd have:

var2.unset()
assert var2.get() is None

which would be very surprising to users who just want to think about
ContextVars and ignore all that stuff about Contexts. This, in turn,
means that we need to expose the MISSING sentinel in general, because
anyone introspecting Context objects directly needs to know how to
recognize this magic value to interpret things correctly.

AFAICT this is the minimum complexity required to get a complete and
internally-consistent set of operations for a ContextVar that's
conceptualized as being a container that either holds an arbitrary
value or is empty.

# Option 2

The other complete and coherent conc

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Serhiy Storchaka]

09.01.18 05:31, Nick Coghlan пише:

On 8 January 2018 at 19:11, Pablo Galindo Salgado  wrote:

Following Gregory's comment on the PR I understand that he is proposing to
have three objects in the os module representing each action and pass a
sequence of these objects to the Python API. What I am not sure about this
is that there is no previous example of such classes in the os module for
other similar APIs and therefore I am not sure if there is a better
approach.


Probably the closest prior art would be the os.DirEntry objects used
for the items yielded from os.scandir - that is the same general idea
(a dedicated Python class to represent a C struct), just in the other
direction.

As with DirEntry, I don't see any obvious value in making the new
objects iterable though - we should be able to just use named field
access in both the C and Python APIs.


Do you suggest to add a class corresponding to 
posix_spawn_file_actions_t with methods corresponding to 
posix_spawn_file_* functions?


class posix_spawn_file_actions:
def __init__(self): ...
def addclose(self, fildes): ...
def addopen(self, fildes, path, oflag, mode): ...
def adddup2(self, fildes, newfildes): ...
def destroy(self): pass
__del__ = destroy

This maximally corresponds the C API. But doesn't look Pythonic.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v2

[Nathaniel Smith]

On Thu, Jan 4, 2018 at 9:42 PM, Guido van Rossum  wrote:
> On Thu, Jan 4, 2018 at 7:58 PM, Nathaniel Smith  wrote:
>> This does make me think that I should write up a short PEP for
>> extending PEP 567 to add context lookup, PEP 550 style: it can start
>> out in Status: deferred and then we can debate it properly before 3.8,
>> but at least having the roadmap written down now would make it easier
>> to catch these details. (And it might also help address Paul's
>> reasonable complaint about "unstated requirements".)
>
> Anything that will help us kill a 550-pound gorilla sounds good to me. :-)
>
> It might indeed be pretty short if we follow the lead of ChainMap (even
> using a different API than MutableMapping to mutate it). Maybe
> copy_context() would map to new_child()? Using ChainMap as a model we might
> even avoid the confusion between Lo[gi]calContext and ExecutionContext which
> was the nail in PEP 550's coffin. The LC associated with a generator in PEP
> 550 would be akin to a loose dict which can be pushed on top of a ChainMap
> using cm = cm.new_child(). (Always taking for granted that instead of
> an actual dict we'd use some specialized mutable object implementing the
> Mapping protocol and a custom mutation protocol so it can maintain
> ContextVar cache consistency.)

The approach I took in PEP 568 is even simpler, I think. The PEP is a
few pages long because I wanted to be exhaustive to make sure we
weren't missing any details, but the tl;dr is: The ChainMap lives
entirely inside the threadstate, so there's no need to create a LC/EC
distinction -- users just see Contexts, or there's the one stack
introspection API, get_context_stack(), which returns a List[Context].
Instead of messing with new_child, copy_context is just
Context(dict(chain_map)) -- i.e., it creates a flattened copy of the
current mapping. (If we used new_child, then we'd have to have a way
to return a ChainMap, reintroducing the LC/EC mess. Plus it would
allow for changes to the Context's inside one ChainMap to "leak" into
the child or vice-versa.) And Context push/pop is just push/pop on the
threadstate's ChainMap's '.maps' attribute.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Serhiy Storchaka]

09.01.18 01:05, Gregory P. Smith пише:
On Mon, Jan 8, 2018 at 12:36 PM Serhiy Storchaka > wrote:


08.01.18 11:11, Pablo Galindo Salgado пише:
 > Following Gregory's comment on the PR I understand that he is
proposing
 > to have three objects in the os module representing each action
and pass
 > a sequence of these objects to the Python API. What I am not sure
about
 > this is that there is no previous example of such classes in the os
 > module for other similar APIs and therefore I am not sure if
there is a
 > better approach.

I would pass a sequence like:

[(os.close, 0),
   (os.open, 1, '/tmp/mylog', os.O_WRONLY, 0o700),
   (os.dup2, 1, 2),
]


i agree with just a list of tuples, but i suggest creating namedtuple 
instances in the posix module for the purpose (one each for close, dup2, 
open) .  Don't put a reference to a function in the tuple as Serhiy 
suggested as, while obvious what it means, it gives the wrong impression 
to the user: nothing is calling the Python functions.  This is a posix 
API that takes a list of arguments for a specific set of system calls 
for _it_ to make for us in a specific order.


Creating three new classes has higher cost than creating three 
singletones. There are some advantages of using existing functions as tags.


But this is not the only possible interface. If there is a single order 
of actions (first close, then open, finally dup2), actions can be 
specified as three keyword-only arguments taking sequences of integers 
or tuples:


posix_spawn(..., close=[0],
open=[(1, '/tmp/mylog', os.O_WRONLY, 0o700)],
dup2=[(1, 2)])

But this perhaps is not able to express all useful cases.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com