[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: The documentation states that thread states are supported within a single interpreter and not supported across other interpreters (specifically for the GIL functions which are just wrapper functions around the PyEval_ functions). So I would have to conclude then that the condition should check to see if the swapping thread is within the current interpreter state otherwise "fatal error", as such... The condition: check->interp == newts->interp should be: check->interp != newts->interp In otherwords if there is a previous thread state and it's interpreter is NOT the same as the one being swapped in then do the fatal error. Just my opinion. I ran into this problem when using the PyThreadState_Swap function directly (low level) to do the thread handling within a single interpreter state (Debug mode only). -- nosy: +fdirosa ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: I believe PyThreadState_Swap function in ceval.c has a bug as I stated earlier. However, I have not seen it included in the latest patches so now I wonder... The following line in PyThreadState_Swap... if (check && check->interp == newts->interp && check != newts) should read as follows... if (check && check->interp != newts->interp && check != newts) since this condition, if true, raises an error. Why should it raise an error if all the interpreters are equal across multiple thread states? If we have one interpreter with multiple thread states (i.e. multi- threaded application) this function will error when switching between the thread states within the same interpreter (in DEBUG compile mode only since this code is commented out otherwise). In the forums it describes the use of thread states to handle multiple python threads running simultaneously and not by using multiple interpreters but only one (the main interpreter). Also the interpreters have be equal because in the documentation for the GIL functions it says it doesn't support multiple interpreters. I think this is a typo/bug in the code. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment:
Thanks Adam
but
I'm still confused because...
There is a new rule in version 2.3.5. Which is one interpreter with many
thread states are supported for the GIL functions. So this code breaks that
rule since this if statement is checking if the interpreters are different
for the current GIL state and the new ts which it can't be (i.e.
unsupported). See this email that points to the python documentation for
2.3.5 regarding this "new" rule...
http://mail.python.org/pipermail/python-dev/2005-May/053840.html
Here is the extract of the email pertaining to this issue...
The documentation (http://docs.python.org/api/threads.html) states
"Note that the PyGILState_*() functions assume there is only one
global interpreter (created automatically by Py_Initialize()). Python
still supports the creation of additional interpreters (using
Py_NewInterpreter()), but mixing multiple interpreters and the
PyGILState_*() API is unsupported. ", so it looks like that using the
PyGilState_XXX functions in the core threadmodule.c means the
Py_NewInterpreter() call (i.e. multiple interpreters) is no longer
supported when threads are involved.
So regardless if we use the GIL functions or the lower level functions it
all eventually boils down to this Swap function which has this condition
that doesn't match what the documentation is stating. So which way is it?
Can't have it both ways.
It seems since 2.3.5 they don't want you to use multiple interpreters is my
guess when threading is involved.
- Franco
- Original Message -
From: "Adam Olsen" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, July 08, 2008 6:25 PM
Subject: [issue1758146] Crash in PyObject_Malloc
Adam Olsen <[EMAIL PROTECTED]> added the comment:
Franco, you need to look at the line above that check:
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check->interp == newts->interp && check != newts)
Py_FatalError("Invalid thread state for this thread");
PyGILState_GetThisThreadState returns the original tstate *for that
thread*. What it's asserting is that, if there's a second tstate *in
that thread*, it must be in a different subinterpreter.
It doesn't prevent your second and third tstate from sharing the same
subinterpreter, but it probably should, as this check implies it's an
invariant.
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1758146>
___
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1758146>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: Thanks Adam but I'm still confused because... There is a new rule in version 2.3.5. Which is one interpreter with many thread states are supported for the GIL functions. So this code breaks that rule since this if statement is checking if the interpreters are different for the current GIL state and the new ts which it can't be (i.e. unsupported). See this email that points to the python documentation for 2.3.5 regarding this "new" rule... http://mail.python.org/pipermail/python-dev/2005-May/053840.html Here is the extract of the email pertaining to this issue... The documentation (http://docs.python.org/api/threads.html) states "Note that the PyGILState_*() functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python still supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_*() API is unsupported. ", so it looks like that using the PyGilState_XXX functions in the core threadmodule.c means the Py_NewInterpreter() call (i.e. multiple interpreters) is no longer supported when threads are involved. So regardless if we use the GIL functions or the lower level functions it all eventually boils down to this Swap function which has this condition that doesn't match what the documentation is stating. So which way is it? Can't have it both ways. It seems since 2.3.5 they don't want you to use multiple interpreters is my guess when threading is involved. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: OK, I think I found my problem. I was using the main interpreter state (the one created by Py_Initialize) to create new thread states with. It seems that this interpreter state is reserved for GIL functions so one will need to create a new interpreter state with PyInterpeterState_New and use that interpreter state when creating the cooperating threads. So now this check makes sense. If one is swapping in a ts belonging to the main interpreter state, it best be the GIL thread state. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: "Also, that Py_DEBUG check effectively says that if you use simplified GIL API for a particular thread against the first interpreter, you are prohibited from creating additional thread states for that thread." I found that you cannot create additional thread states against the first interpreter and swap between them w/o this assertion occurring. I didn't use the GIL functions at all and had this issue in debug. PyInitialize initializes the GIL and hijacks the main interpreter. We always call PyInitialize so does that mean we can only use the GIL functions with the main interpreter and nothing else when locking/unlocking the global lock as you seem to infer? Does that mean there is a backward compatibility issue here with those who used the main interpreter only and created thread states from it to handle multi- threading, like I did (thru the use of PyEval_Acquire/Release & PyThreadState_Swap)? ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: By the way. I switched to using the GIL functions on the main interpreter and everything works great now. It is a better solution to use the GIL functions because I also had my own code that prevented dead lock from occuring when a python script calls back into the extension module that ends up calling PyEval_Acquire again (deadlock) even though it is the same thread. Now with the GIL functions I don't need that code. It is a good feature but it broke my previous implementation and it is not obvious why. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1758146] Crash in PyObject_Malloc
Franco DiRosa <[EMAIL PROTECTED]> added the comment: I'm unsure if you are understanding what I'm doing so here is the story... I stepped through Py_Initialize and this function takes the main interpreter and it's initial thread state and makes that the GIL thread state. The following code in Py_Initialize hijacks the main interpreter and thread state for GIL use... /* auto-thread-state API, if available */ #ifdef WITH_THREAD _PyGILState_Init(interp, tstate); #endif /* WITH_THREAD */ WITH_THREAD is defined since I'm using multithreading in my application. So now if you create thread states from the main interpeter and use the PyEval_Acquire/Release and PyThreadState_Swap you will get the assertion when compiled with the DEBUG option. If you use the PyGILState_Ensure and PyGILState_Release functions you don't. What I'm doing is that I have a Windows application with embedded python. The application spawns multiple threads each running a python script. Each application thread has its own unique PyThreadState created from the main interpreter because I wanted all the modules loaded only once for resource conservation purposes (thus use only one interpreter). I used PyEval_Acquire/Release and PyThreadState_Swap to handle swapping in each application thread's thread state when each one uses the python API. This worked great in RELEASE compilation but in DEBUG it asserted. Now that I use the GIL functions it works well and not only that, I removed the code I had put in myself to handle python callback's into the application and avoiding deadlocks by calling PyEval_Acquire onto itself (since it uses mutexes which doesn't do reference counting so it could deadlock waiting on itself to complete) ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1758146> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
