On 5 December 2011 17:48, Mark Wiebe <[email protected]> wrote: > On Mon, Dec 5, 2011 at 9:37 AM, mark florisson <[email protected]> > wrote: >> >> On 5 December 2011 17:25, Mark Wiebe <[email protected]> wrote: >> > On Sun, Dec 4, 2011 at 11:37 PM, Geoffrey Irving <[email protected]> wrote: >> >> >> >> <snip> >> >> >> >> >> >> Back to the bugs: here's a branch with all the changes I needed to get >> >> rational arithmetic to work: >> >> >> >> https://github.com/girving/numpy >> >> >> >> I discovered two more after the last email. One is another simple 0 >> >> vs. 1 bug, and another is somewhat optional: >> >> >> >> commit 730b05a892371d6f18d9317e5ae6dc306c0211b0 >> >> Author: Geoffrey Irving <[email protected]> >> >> Date: Sun Dec 4 20:03:46 2011 -0800 >> >> >> >> After loops, check for PyErr_Occurred() even if needs_api is 0 >> >> >> >> For certain types of user defined classes, casting and ufunc loops >> >> normally run without the Python API, but occasionally need to throw >> >> an error. Currently we assume that !needs_api means no error occur. >> >> However, the fastest way to implement such loops is to run without >> >> the GIL normally and use PyGILState_Ensure/Release if an error >> >> occurs. >> >> >> >> In order to support this usage pattern, change all post-loop checks >> >> from >> >> >> >> needs_api && PyErr_Occurred() >> >> >> >> to simply >> >> >> >> PyErr_Occurred() >> > >> > >> > To support this properly, I think we would need to convert needs_api >> > into an >> > enum with this hybrid mode as another case. While it isn't done >> > currently, I >> > was imagining using a thread pool to multithread the trivially >> > data-parallel >> > operations when needs_api is false, and I suspect the >> > PyGILState_Ensure/Release would trigger undefined behavior in a thread >> > created entirely outside of the Python system. >> >> PyGILState_Ensure/Release can be safely used by non-python threads >> with the only requirement that the GIL has been initialized previously >> in the main thread (PyEval_InitThreads). > > > Is there a way this could efficiently be used to propagate any errors back > to the main thread, for example using TBB as the thread pool? The innermost > task code which calls the inner loop can't call PyErr_Occurred() without > first calling PyGILState_Ensure itself, which would kill utilization.
No, there is no way these things can be efficient, as the GIL is likely contented anyway (I wasn't making a point for these functions, just wanted to clarify). There is in fact the additional problem that PyGILState_Ensure would initialize a threadstate, you set an exception, and when you call PyGILState_Release the threadstate gets deleted along with the exception, before you will even have a chance to check with PyErr_Occurred(). For cython.parallel I worked around this by calling PyGILState_Ensure (to initialize the thread state), followed immediately by Py_BEGIN_ALLOW_THREADS before starting any work. You then have to fetch the exception and restore it in another thread when you want to propagate it. It's a total mess, it's inefficient and if you can avoid it you should. > Maybe this is an ABI problem in NumPy that needs to be fixed, to mandate > that inner loops always return an error code and disallow them from setting > the Python exception state without returning failure. That would likely be the best thing. > -Mark > >> >> >> > For comparison, I created a >> > special mechanism for simplified multi-threaded exceptions in the nditer >> > in >> > the 'errmsg' parameter: >> > >> > >> > http://docs.scipy.org/doc/numpy/reference/c-api.iterator.html#NpyIter_GetIterNext >> > >> > Worth considering is also the fact that the PyGILState API is >> > incompatible >> > with multiple embedded interpreters. Maybe that's not something anyone >> > does >> > with NumPy, though. >> > >> > -Mark >> > >> >> >> >> >> >> Geoffrey >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> [email protected] >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > [email protected] >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> [email protected] >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
