[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Change by Pavel Minaev :


--
nosy: +int19h

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Pavel Minaev  added the comment:

This is a bit tricky to explain... There's no easy way to achieve this effect 
"normally". It manifests due to the way some Python debuggers (specifically, 
pydevd and ptvsd - as used by PyCharm, PyDev, and VSCode) implement 
non-cooperative attaching to a running Python process by PID.

A TL;DR take is that those debuggers have to inject a new thread into a running 
process from the outside, and then run some Python code on that thread. There 
are OS APIs for such thread injection - e.g. CreateRemoteThread on Windows. 
There are various tricks that they then have to use to safely acquire GIL and 
invoke PyEval_InitThreads, but ultimately it comes down to running Python code.

That is the point where this can manifest. Basically, as soon as that injected 
code (i.e. the actual debugger) imports threading, things break. And advanced 
debuggers do need background threads for some functionality...

Here are the technical details - i.e. how thread injection happens exactly, and 
what kind of code it might run - if you're interested.
https://github.com/microsoft/ptvsd/issues/1542

I think that a similar problem can also occur in an embedded Python scenario 
with multithreading. Consider what happens if the hosted interpreter is 
initialized from the main thread of the host app - but some Python code is then 
run from the background thread, and that code happens to be the first in the 
process to import threading. Then that background thread becomes the "main 
thread" for threading, with the same results as described above.

The high-level problem, I think, is that there's an inconsistency between what 
Python itself considers "main thread" (i.e. main_thread in ceval.c, as set by 
PyEval_InitThreads), and what threading module considers "main thread" (i.e. 
_main_thread in threading.py). Logically, these should be in sync.

If PyEval_InitThreads is the source of truth, then the existing thread 
injection technique will "just work" as implemented already in all the 
aforementioned debuggers. It makes sure to invoke PyEval_InitThreads via 
Py_AddPendingCall, rather than directly from the background thread, precisely 
so that the interpreter doesn't get confused. 

Furthermore, on 3.7+, PyEval_InitThreads is already automatically invoked by 
Py_Initialize, and hence when used by python.exe, will mark the actual first 
thread in the process as the main thread. So, using it a the source of truth 
would guarantee that attach by thread injection works correctly in all 
non-embedded Python scenarios.

Apps hosting Python would still need to ensure that they always call 
Py_Initialize on what they want to be the main thread, as they already have to 
do; but they wouldn't need to worry about "import threading" anymore.

--

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Pavel Minaev  added the comment:

It's also possible to hit if using some native code that starts a background 
thread without going via threading, and runs Python code on that background 
thread. In that case, if that Python code then does "import threading", and 
threading hasn't been imported yet, then you have this same problem.

Here's a pure Python repro using ctypes on Win32:

#--
import sys, time
from ctypes import *

ThreadProc = WINFUNCTYPE(c_uint32, c_void_p)

@ThreadProc
def thread_proc(_):
import threading
print(threading.current_thread() is threading.main_thread())
return 0

assert "threading" not in sys.modules
#import threading  # uncomment to fix

windll.kernel32.CreateThread(None, c_size_t(0), thread_proc, None, c_uint32(0), 
None)
time.sleep(1)

assert "threading" in sys.modules
import threading
print(threading.current_thread() is threading.main_thread())
#--

Here's the output:

>py -3 main.py
True
False
Exception ignored in: 
Traceback (most recent call last):
  File "C:\Python\3.7-64\lib\threading.py", line 1276, in _shutdown
assert tlock.locked()
AssertionError:

--

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Pavel Minaev  added the comment:

Debuggers will have to work around this in past Python versions that they 
support (which still includes Python 2 for pretty much all of them), so this is 
solely about resolving the inconsistency for the future. No point rushing it 
for 3.8 specifically.

(The most likely immediate workaround will be that, instead of invoking 
PyEval_InitThreads on the main thread via Py_AddPendingCall, we will simply use 
the same facility to exec "import threading" on the main thread.)

--

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com