[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-22 Thread Victor Stinner
Le sam. 21 mars 2020 à 04:14, Stephen J. Turnbull a écrit : > But ... this sounds to me like work that should be done on a branch. > > I'm sure you considered that, but I also expect others will feel the > same way. Perhaps this is a good opportunity to document why it's not > being done on a bra

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-21 Thread Nick Coghlan
On Sat, 21 Mar 2020 at 13:15, Stephen J. Turnbull wrote: > That makes a lot of sense, as a strategy for doing the work. It > should be pretty straightforward to convert the tstate argument to a > thread-local tstate. Note that the thread locals are already there, as that's how the public API alr

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-21 Thread Antoine Pitrou
On Fri, 20 Mar 2020 19:24:22 +0100 Victor Stinner wrote: > > One good example is Py_AddPendingCall(). The documentation says that > it's safe to call it without holding the GIL. Except that right now, > there is no reliable way to get the correct interpreter in this case > (correct me if I'm wron

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-21 Thread Nathaniel Smith
On Fri, Mar 20, 2020 at 11:27 AM Victor Stinner wrote: > I would prefer to continue to experiment passing tstate explicitly in > internal C APIs until most blocker issues will be fixed. Once early > work on running two subinterpreters in parallel will start working > (one "GIL" per interpreter), I

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-20 Thread Nick Coghlan
On Wed., 18 Mar. 2020, 8:36 pm Mark Shannon, wrote: > > > On 17/03/2020 7:00 pm, Steve Dower wrote: > > On 17Mar2020 1803, Chris Angelico wrote: > >> On Wed, Mar 18, 2020 at 3:50 AM Mark Shannon wrote: > >>> The accessibility of a thread-local variable is a strict superset of > >>> that of a fun

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-20 Thread Stephen J. Turnbull
Victor Stinner writes: > Le jeu. 19 mars 2020 à 02:17, Kyle Stanley a écrit : > > Agreed; a PEP (even if it's just informational) would go a long way in > > helping to clear up some misunderstandings. > > I am still moving blindly in the darkness of CPython internals and so > I don't feel

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-20 Thread Victor Stinner
Le jeu. 19 mars 2020 à 02:17, Kyle Stanley a écrit : > Agreed; a PEP (even if it's just informational) would go a long way in > helping to clear up some misunderstandings. I am still moving blindly in the darkness of CPython internals and so I don't feel comfortable to write a PEP which would me

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Kyle Stanley
Antoine Pitrou wrote: > In any case, the amount of disagreement and/or misunderstanding in this > discussion is a strong hint that it needs a PEP to hash things out and > explain them clearly, IMHO. Agreed; a PEP (even if it's just informational) would go a long way in helping to clear up some mis

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Kyle Stanley
Mark Shannon wrote: > The point I'm making is that adding `tstate` parameters everywhere is > unnecessary. Using a thread local variable is much less intrusive and is > at least as capable. Objectively speaking, what would be the specific difference(s) in behavior and performance between using a t

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Barry Scott
> On 17 Mar 2020, at 16:43, Mark Shannon wrote: > > > > On 17/03/2020 3:38 pm, Steve Dower wrote: >> On 17Mar2020 1447, Mark Shannon wrote: >>> On 16/03/2020 3:04 pm, Victor Stinner wrote: In short, the answer is yes. >>> >>> I said "no" then and gave reasons. AFAICT no one has faulted

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Anders Munch
Antoine Pitrou [mailto:solip...@pitrou.net]: > This example is mixing up the notion of interpreter state and thread state. Sorry about that, I was making a more general point and not paying so much attention to the specific names. The general point is this: A thread-local variable can work as an

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Antoine Pitrou
On Wed, 18 Mar 2020 13:35:16 + Anders Munch wrote: > Chris Angelico [mailto:ros...@gmail.com]: > > And by that logic, globals are even more capable. I don't understand your > > point. Isn't the purpose of the tstate parameters to avoid the problem of > > being unable to have multiple tstates w

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Anders Munch
Chris Angelico [mailto:ros...@gmail.com]: > And by that logic, globals are even more capable. I don't understand your > point. Isn't the purpose of the tstate parameters to avoid the problem of > being unable to have multiple tstates within the same OS thread? I think I've > missed something here.

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-18 Thread Mark Shannon
On 17/03/2020 7:00 pm, Steve Dower wrote: On 17Mar2020 1803, Chris Angelico wrote: On Wed, Mar 18, 2020 at 3:50 AM Mark Shannon wrote: The accessibility of a thread-local variable is a strict superset of that of a function-local variable. Therefore storing the thread state in a thread-loca

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Steve Dower
On 17Mar2020 1803, Chris Angelico wrote: On Wed, Mar 18, 2020 at 3:50 AM Mark Shannon wrote: The accessibility of a thread-local variable is a strict superset of that of a function-local variable. Therefore storing the thread state in a thread-local variable is at least as capable as passing t

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Chris Angelico
On Wed, Mar 18, 2020 at 3:50 AM Mark Shannon wrote: > The accessibility of a thread-local variable is a strict superset of > that of a function-local variable. > > Therefore storing the thread state in a thread-local variable is at > least as capable as passing thread-state as a parameter. > And

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Mark Shannon
On 17/03/2020 3:38 pm, Steve Dower wrote: On 17Mar2020 1447, Mark Shannon wrote: On 16/03/2020 3:04 pm, Victor Stinner wrote: In short, the answer is yes. I said "no" then and gave reasons. AFAICT no one has faulted my reasoning. I said "yes" then and was also not faulted. I'll do tha

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Victor Stinner
Le mar. 17 mars 2020 à 15:49, Mark Shannon a écrit : > > * https://bugs.python.org/issue10915 > > * https://bugs.python.org/issue15751 > > > > It's unclear to me if fixing this issue would require to add a lock, > > nor if it would make PyGILState_GetThisThreadState() or > > _PyThreadState_GET() s

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Victor Stinner
Le mar. 17 mars 2020 à 15:47, Mark Shannon a écrit : > > There is no PEP but scatted documents. I wrote a short article to > > elaborate the context of this work: > > https://vstinner.github.io/cpython-pass-tstate.html > > > > One motivation is to ease the implementation of subinterpreters (PEP >

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Steve Dower
On 17Mar2020 1447, Mark Shannon wrote: On 16/03/2020 3:04 pm, Victor Stinner wrote: In short, the answer is yes. I said "no" then and gave reasons. AFAICT no one has faulted my reasoning. I said "yes" then and was also not faulted. Let me reiterate why using a thread-local variable is bette

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Antoine Pitrou
On Tue, 17 Mar 2020 14:47:19 + Mark Shannon wrote: > On 16/03/2020 3:04 pm, Victor Stinner wrote: > > Hi, > > > >> Changes on this scale merit a PEP and proper discussion, rather than > >> being added piecemeal without proper review. > > > > Last November, I asked explicitly on python-de

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Mark Shannon
On 16/03/2020 6:21 pm, Victor Stinner wrote: There were quick discussions about using thread local storage (TLS) to get and set the current Python thread state ("tstate"), instead of reading/setting an atomic variable (_PyRuntime.gilstate.tstate_current). In fact, TLS already exists as "PyGILS

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-17 Thread Mark Shannon
On 16/03/2020 3:04 pm, Victor Stinner wrote: Hi, Changes on this scale merit a PEP and proper discussion, rather than being added piecemeal without proper review. Last November, I asked explicitly on python-dev if we should "Pass the Python thread state to internal C functions": https://mai

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-16 Thread Victor Stinner
There were quick discussions about using thread local storage (TLS) to get and set the current Python thread state ("tstate"), instead of reading/setting an atomic variable (_PyRuntime.gilstate.tstate_current). In fact, TLS already exists as "PyGILState" and PyGILState_GetThisThreadState() can alr

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-16 Thread Victor Stinner
Mark opened https://bugs.python.org/issue39978 "Vectorcall implementation should conform to PEP 590" where he wrote that passing tstate explicitly slows down PyObject_Vectorcall(). Victor Le lun. 16 mars 2020 à 15:16, Mark Shannon a écrit : > > Hi, > > There seems to be a proliferation of `PyThr

[Python-Dev] Re: Proliferation of tstate arguments.

2020-03-16 Thread Victor Stinner
Hi, > Changes on this scale merit a PEP and proper discussion, rather than > being added piecemeal without proper review. Last November, I asked explicitly on python-dev if we should "Pass the Python thread state to internal C functions": https://mail.python.org/archives/list/python-dev@python.or