[Python-Dev] thread issues when embedding Python
I've successfully embedded Python for a single thread I tried to extend the implementation for multiple threads (a worker thread scenario) and I'm encountering either deadlocks or seg faults depending upon how I got about it. There seems to be some inconsistency between what is covered in the docs here: http://docs.python.org/2/c-api/init.html#non-python-created-threads and the experiences of other users trying the same thing, e.g. http://bugs.python.org/issue19576 http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads Can anybody comment on the situation, in particular, Is the non-python-created-threads documentation accurate for v2.7? If a main thread does things like importing a module and obtaining a reference to a Python method, can those things be used by other C++ threads or do they have to repeat those lookups? Is there any logic that needs to be executed once only as each thread is started? (the doc suggests just PyGILState_Ensure/PyGILState_Release each time a thread accesses Python methods - is there anything else?) Given the bug 19576, what is the most portable way to code this to work reliably on unfixed Python versions? (e.g. should users always explicitly call PyEval_InitThreads() in their main thread or worker threads or both?) Here is my actual source code: https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/ (see example.py for a trivial example of what it does) The problem that I encounter: - the init stuff runs fine in PyRoutePlugin.cxx, it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule and looks up the "provide_route" method in the module it creates a PyRouteWorker object, giving it a reference to "provide_route" it creates a thread pool to run the worker - the PyRouteWorker::process() method is invoked in one of those threads - it crashes when trying to call the "provide_route" method PyRouteWorker.cxx: routes = mAction.apply(args); Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x730b8700 (LWP 23965)] 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 (gdb) bt #0 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 #1 0x73d6b647 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.7.so.1.0 #2 0x7414885a in apply (args=..., this=) at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215 #3 repro::PyRouteWorker::process (this=0x6f00a0, msg=) at PyRouteWorker.cxx:98 #4 0x77b879e1 in repro::WorkerThread::thread (this=0x68e110) at WorkerThread.cxx:36 #5 0x770b7a2f in threadIfThreadWrapper (threadParm=) at ThreadIf.cxx:51 #6 0x765ffb50 in start_thread (arg=) at pthread_create.c:304 #7 0x75999a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #8 0x in ?? () ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] thread issues when embedding Python
I've successfully embedded Python for a single thread I tried to extend the implementation for multiple threads (a worker thread scenario) and I'm encountering either deadlocks or seg faults depending upon how I got about it. There seems to be some inconsistency between what is covered in the docs here: http://docs.python.org/2/c-api/init.html#non-python-created-threads and the experiences of other users trying the same thing, e.g. http://bugs.python.org/issue19576 http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads Can anybody comment on the situation, in particular, Is the non-python-created-threads documentation accurate for v2.7? If a main thread does things like importing a module and obtaining a reference to a Python method, can those things be used by other C++ threads or do they have to repeat those lookups? Is there any logic that needs to be executed once only as each thread is started? (the doc suggests just PyGILState_Ensure/PyGILState_Release each time a thread accesses Python methods - is there anything else?) Given the bug 19576, what is the most portable way to code this to work reliably on unfixed Python versions? (e.g. should users always explicitly call PyEval_InitThreads() in their main thread or worker threads or both?) Here is my actual source code: https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/ (see example.py for a trivial example of what it does) The problem that I encounter: - the init stuff runs fine in PyRoutePlugin.cxx, it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule and looks up the "provide_route" method in the module it creates a PyRouteWorker object, giving it a reference to "provide_route" it creates a thread pool to run the worker - the PyRouteWorker::process() method is invoked in one of those threads - it crashes when trying to call the "provide_route" method PyRouteWorker.cxx: routes = mAction.apply(args); Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x730b8700 (LWP 23965)] 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 (gdb) bt #0 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 #1 0x73d6b647 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.7.so.1.0 #2 0x7414885a in apply (args=..., this=) at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215 #3 repro::PyRouteWorker::process (this=0x6f00a0, msg=) at PyRouteWorker.cxx:98 #4 0x77b879e1 in repro::WorkerThread::thread (this=0x68e110) at WorkerThread.cxx:36 #5 0x770b7a2f in threadIfThreadWrapper (threadParm=) at ThreadIf.cxx:51 #6 0x765ffb50 in start_thread (arg=) at pthread_create.c:304 #7 0x75999a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #8 0x in ?? () ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
Another link that fills in some gaps and finally helped me make this work: http://www.codevate.com/blog/7-concurrency-with-embedded-python-in-a-multi-threaded-c-application In particular, I found that PyGILState_Ensure/PyGILState_Release as described in the Python docs is not sufficient - as described in that blog link, I had to a) obtain PyInterpreterState from the first thread where Py_Initialize() was called b) when each worker thread starts, call PyThreadState_New(mInterpreterState) and save the result in a thread local mPyThreadState c) use the mPyThreadState with PyEval_RestoreThread and PyEval_SaveThread before and after calling Python methods Is this a bug in PyGILState_Ensure or is it a deficiency in the documentation? I also found one bug in my own code, although that was not related to the problem just described with PyGILState_Ensure and I had to fix both problems to make it work. Specifically, the PyWorkerThread constructor was taking an object argument when it should have taken a reference argument and this was creating an invalid Py::Callable member in my worker. On 18/12/13 00:19, Daniel Pocock wrote: > > I've successfully embedded Python for a single thread > > I tried to extend the implementation for multiple threads (a worker > thread scenario) and I'm encountering either deadlocks or seg faults > depending upon how I got about it. > > There seems to be some inconsistency between what is covered in the docs > here: > > http://docs.python.org/2/c-api/init.html#non-python-created-threads > > and the experiences of other users trying the same thing, e.g. > > http://bugs.python.org/issue19576 > http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads > > Can anybody comment on the situation, in particular, > > Is the non-python-created-threads documentation accurate for v2.7? > > If a main thread does things like importing a module and obtaining a > reference to a Python method, can those things be used by other C++ > threads or do they have to repeat those lookups? > > Is there any logic that needs to be executed once only as each thread is > started? (the doc suggests just PyGILState_Ensure/PyGILState_Release > each time a thread accesses Python methods - is there anything else?) > > Given the bug 19576, what is the most portable way to code this to work > reliably on unfixed Python versions? (e.g. should users always > explicitly call PyEval_InitThreads() in their main thread or worker > threads or both?) > > > > > Here is my actual source code: > > https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/ > > (see example.py for a trivial example of what it does) > > The problem that I encounter: > > - the init stuff runs fine in PyRoutePlugin.cxx, > it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule > and looks up the "provide_route" method in the module > it creates a PyRouteWorker object, > giving it a reference to "provide_route" > it creates a thread pool to run the worker > > - the PyRouteWorker::process() method is invoked in one of those threads > > - it crashes when trying to call the "provide_route" method > PyRouteWorker.cxx: > routes = mAction.apply(args); > > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x730b8700 (LWP 23965)] > 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 > (gdb) bt > #0 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 > #1 0x73d6b647 in PyEval_CallObjectWithKeywords () >from /usr/lib/libpython2.7.so.1.0 > #2 0x7414885a in apply (args=..., this=) > at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215 > #3 repro::PyRouteWorker::process (this=0x6f00a0, msg=) > at PyRouteWorker.cxx:98 > #4 0x77b879e1 in repro::WorkerThread::thread (this=0x68e110) > at WorkerThread.cxx:36 > #5 0x770b7a2f in threadIfThreadWrapper (threadParm=) > at ThreadIf.cxx:51 > #6 0x765ffb50 in start_thread (arg=) > at pthread_create.c:304 > #7 0x75999a7d in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > #8 0x in ?? () > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/daniel%40pocock.com.au ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On 18/12/13 16:02, Chris Angelico wrote:
> On Wed, Dec 18, 2013 at 9:26 PM, Daniel Pocock wrote:
>> b) when each worker thread starts, call
>> PyThreadState_New(mInterpreterState) and save the result in a thread
>> local mPyThreadState
>>
>> c) use the mPyThreadState with PyEval_RestoreThread and
>> PyEval_SaveThread before and after calling Python methods
>>
>> Is this a bug in PyGILState_Ensure or is it a deficiency in the
>> documentation?
> It doesn't surprise me that you would need to do step b - I do seem to
> recall the need to call that for each new thread. Not so sure about c.
> Once you fixed the unrelated bug, do you still need that step? (Been a
> while since I last embedded Python though, and I might well be wrong.)
Yes, I definitely needed to use this PyThreadState_New call even after
my unrelated bug fix
Should it be added to the documentation?
I created a C++ wrapper around this logic, it is here:
https://github.com/resiprocate/resiprocate/blob/master/repro/plugins/pyroute/PyThreadSupport.hxx
and the use case is something like:
// in constructor:
PyExternalUser* mPyUser = new PyExternalUser(mInterpreterState);
and each time Python calls are made, just do:
{
PyExternalUser::Use use(*mPyUser);
// now call Python code
}
When the PyExternalUser::Use instance is created it does PyEval_RestoreThread()
When the PyExternalUser::Use instance goes out of scope it is destroyed and
PyEval_SaveThread() is called
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On 18/12/13 16:29, Victor Stinner wrote: > 2013/12/18 Antoine Pitrou : >> You only need to call PyEval_InitThreads() once in the main Python >> thread. > > This is not well documented. For your information, PyGILState_Ensure() > now calls PyEval_InitThreads() in Python 3.4, see: > http://bugs.python.org/issue19576 I did see that - but from my own experience, I do not believe it is calling PyThreadState_New(..) and it is not even checking if PyThreadState_New(..) has ever been called for the active thread Consequently, the thread is blocked or there is a seg fault I've now written up a much more thorough overview of my experience on my blog: http://danielpocock.com/embedding-python-multi-threaded-cpp ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On 19/12/13 12:22, Nick Coghlan wrote: > On 19 December 2013 07:58, Daniel Pocock wrote: >> >> On 18/12/13 16:29, Victor Stinner wrote: >>> 2013/12/18 Antoine Pitrou : >>>> You only need to call PyEval_InitThreads() once in the main Python >>>> thread. >>> This is not well documented. For your information, PyGILState_Ensure() >>> now calls PyEval_InitThreads() in Python 3.4, see: >>> http://bugs.python.org/issue19576 >> >> I did see that - but from my own experience, I do not believe it is >> calling PyThreadState_New(..) and it is not even checking if >> PyThreadState_New(..) has ever been called for the active thread >> >> Consequently, the thread is blocked or there is a seg fault >> >> I've now written up a much more thorough overview of my experience on my >> blog: >> >> http://danielpocock.com/embedding-python-multi-threaded-cpp > You absolutely should *NOT* have to call PyThreadState_New before > calling PyGILState_Ensure, as it is designed to call it for you (see > http://hg.python.org/cpython/file/2.7/Python/pystate.c#l598). If > calling PyThreadState_New works, but calling PyGILState_Ensure does > not, then something else is broken (such as not initialising the > thread local storage for the GIL state APIs). > > I don't see anything in your article about how you ensure that the > main thread of the application *before anything else related to the > embedded Python happens* calls both Py_Initialize() and > PyEval_InitThreads(). If you don't do that, then all bets are off in > terms of multithreading support. I definitely do both of those things in the method PyRoutePlugin::init(..) It is in PyRoutePlugin.cxx: http://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/PyRoutePlugin.cxx?view=markup#l88 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interested in GSoC 2014
On 24/12/13 07:41, Prasad Joshi wrote: > Hello All, > > I am interested in participating GSoC 2014. I have went through last > year's eligibility criterion, I think I am qualified to participate. I > know GSoC 2014 still has more than 2-3 months, however I would like to > start early. Please let me know suggestions. Hi Prasad, We recently introduced a Python scripting API into reSIProcate You would be welcome to contribute some code to exercise the API. Here are some details: http://www.resiprocate.org/Python It would be particularly interesting to try and integrate the new WebRTC / WebSocket cookie support with some existing Python-based web framework, e.g. Django. This would allow a user with a Django login, shopping cart or some other long running session to pass their state through the cookies. Regards, Daniel ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
