[Python-Dev] Bug in PyErr_WriteUnraisable ?
I'd hit an access violation inside PyErr_WriteUnraisable when a non-exception instance was raised. The call to PyExceptionClass_Name with a non-exception instance is yielding an invalid pointer. We are embedding Python 2.5 and a string instance is raised using PyThreadState_SetAsyncExc. I can fix that in my code, by raising an appropiate exception instance, but I think PyErr_WriteUnraisable lacks some checks. -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in PyErr_WriteUnraisable ?
Brett Cannon wrote: > On 2/22/07, Gabriel Becedillas <[EMAIL PROTECTED]> wrote: >> I'd hit an access violation inside PyErr_WriteUnraisable when a >> non-exception instance was raised. The call to PyExceptionClass_Name >> with a non-exception instance is yielding an invalid pointer. >> We are embedding Python 2.5 and a string instance is raised using >> PyThreadState_SetAsyncExc. I can fix that in my code, by raising an >> appropiate exception instance, but I think PyErr_WriteUnraisable lacks >> some checks. >> > > Please use the the bug tracker at > http://sourceforge.net/tracker/?group_id=5470&atid=105470 to report > issues with Python. > > -Brett > >> -- >> >> >> Gabriel Becedillas >> Developer >> CORE SECURITY TECHNOLOGIES >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org >> > I've submitted the bug report with a snippet of code to reproduce the crash. http://sourceforge.net/tracker/index.php?func=detail&aid=1669182&group_id=5470&atid=105470 -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyThreadState_SetAsyncExc and native extensions
Does anyone know if PyThreadState_SetAsyncExc stops a thread while its inside a native extension ? I'm trying to stop a testing script that boils down to this: while True: print "aaa" native_extension_call() print "bbb" Using PyThreadState_SetAsyncExc the module doesn't stop but if I add more print statements to increase the chance that PyThreadState_SetAsyncExc is called when the module is executing Python code, the module stops. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyThreadState_SetAsyncExc, PyErr_Clear and native extensions
I use PyThreadState_SetAsyncExc to stop a python thread but there are
situations when the thread doesn't stop and continues executing
normally. After some debugging, I realized that the problem is that
PyThreadState_SetAsyncExc was called when the thread was inside a
native extension, that for some reason calls PyErr_Clear. That code
happens to be inside boost::python.
I do need to stop the thread from executing Python code as soon as
possible (as soon as it returns from a native function is also
acceptable).
Because we have embedded Python's VM in our product, I'm thinking of
modifying PyErr_Clear() to return immediately if the thread was
stopped (we determine if the thread should stop using our own
functions).
Example:
void PyErr_Clear(void)
{
if (!stop_executing_this_thread())
PyErr_Restore(NULL, NULL, NULL);
}
Does anybody see any problem with this approach ?, Does anybody have a
cleaner/better solution ?
I've allready posted this in comp.lang.python a couple of days ago but
got no answers, that's why I'm posting here, coz I think the only ones
that could give me a solution/workaround are Python core developers.
Thanks a lot.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Syscall Proxying in Python
Hi, We embbeded Python 2.0.1 in our product a few years ago and we'd like to upgrade to Python 2.4.1. This was not a simple task, because we needed to execute syscalls on a remote host. We modified Python's source code in severall places to call our own versions of some functions. For example, instead of calling fopen(...), the source code was modified to call remote_fopen(...), and the same was done with other libc functions. Socket functions where hooked too (we modified socket.c), Windows Registry functions, etc.. There are some syscalls that we don't want to execute remotely. For example when importing a module. That has to be local, and we didn't modified that. Python scripts are executed locally, but syscalls are executed on a remote host, thus giving the illusion that the script is executing on the remote host. As I said before, we're in the process of upgrading and we don't want to make such unmaintainable changes to Python's code. We'd like to make as few changes as possible. The aproach we're trying this time is far less intrusive: We'd like to link Python with special libraries that override those functions that we want to execute remotely. This way the only code that has to be changed is the one that has to be executed locally. I wrote this mail to ask you guys for any useful advice in making this changes to Python's core. The only places I figure out right now that have to execute locally all the time are import.c and pythonrun.c, but I'm not sure at all. Maybe you guys figure out another way to achieve what we need. Thanks in advance. -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES Florida 141 - 2º cuerpo - 7º piso C1005AAC Buenos Aires - Argentina Tel/Fax: (54 11) 5032-CORE (2673) http://www.corest.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Syscall Proxying in Python
Donovan Baarda wrote: > On Mon, 2005-08-01 at 10:36, Gabriel Becedillas wrote: > >>Hi, >>We embbeded Python 2.0.1 in our product a few years ago and we'd like to >>upgrade to Python 2.4.1. This was not a simple task, because we needed >>to execute syscalls on a remote host. We modified Python's source code >>in severall places to call our own versions of some functions. For >>example, instead of calling fopen(...), the source code was modified to >>call remote_fopen(...), and the same was done with other libc functions. >>Socket functions where hooked too (we modified socket.c), Windows >>Registry functions, etc.. > > > Wow... you guys sure did it the hard way. If you had done it at the > Python level, you would have had a much easier time of both implementing > and updating it. > > As an example, have a look at my osVFS stuff. This is a replacement for > the os module and open() that tricks Python into using a virtual file > system; > > http://minkirri.apana.org.au/~abo/projects/osVFS > > Hi, thanks for your reply. The problem I see with the aproach you're sugesting is that I have to rewrite a lot of code to make it work the way I want. We allready have the syscall proxying stuff with an stdio layer on top of it. I should have to rewrite some parts of some modules and use my own versions of stdio functions, and that is pretty much the same as we have done before. There are also native objects that use stdio functions, and I should replace those ones too, or modules that have some native code that uses stdio, or sockets. I should duplicate those files, and make the same kind of search/replace work that we have done previously and that we'd like to avoid. Please let me know if I misunderstood you. Thanks again. -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES Florida 141 - 2º cuerpo - 7º piso C1005AAC Buenos Aires - Argentina Tel/Fax: (54 11) 5032-CORE (2673) http://www.corest.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Sharing between multiple interpreters and restricted mode
Hi, At the company I work for we've embedded Python 2.4.1 in a C++ application. We execute multiple scripts concurrenlty, each one in its own interpreter (created using Py_NewInterpreter()). We are sharing a certain instance between interpreters because its to expensive to instantiate that class every time an interpreter is created. The class is instantiated in the main interpreter (that is always alive) and every time a new interpreter is created, that instance is added to the interpreter's __builtins__ dict. The problem I'm having is that some methods of that class (when executed in an interpreter different from the one it was created in) complain about being in restricted execution mode. Assuming that there are no issues with this instance lifetime (coz it will always be referenced by the main interpreter), is there a safe way to do some sharing between interpreters ?. Thanks. -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES Florida 141 - 2º cuerpo - 7º piso C1005AAC Buenos Aires - Argentina Tel/Fax: (54 11) 5032-CORE (2673) http://www.corest.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyThreadState_Delete vs PyThreadState_DeleteCurrent
Hi,
I started hitting Py_FatalError("Invalid thread state for this thread")
in pystate.c when a thread bootstraps since my upgrade to Python 2.4.2
(was using 2.4.1 previously).
I'm embedding Python in C++, using multiple interpreters and threads.
I think the problem is that PyThreadState_Delete (which is used when I
destroy thread states and interpreters) is not making the same clean up
that PyThreadState_DeleteCurrent is doing:
if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
PyThread_delete_key_value(autoTLSkey);
If a thread id is reused (and this often happens), and the previous
thread used PyThreadState_Delete instead of PyThreadState_DeleteCurrent,
an old and invalid value for autoTLSkey is stored, and that is
triggering the Py_FatalError.
Thanks.
--
Gabriel Becedillas
Developer
CORE SECURITY TECHNOLOGIES
Florida 141 - 2º cuerpo - 7º piso
C1005AAC Buenos Aires - Argentina
Tel/Fax: (54 11) 5032-CORE (2673)
http://www.corest.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pystate.c changes for Python 2.4.2
Hi,
At the company I work for, we've embedded Python in C++ application we
develop. Since our upgrade to Python 2.4.2 from 2.4.1 we started hitting
Py_FatalError("Invalid thread state for this thread") when using debug
builds.
We use both multiple interpreters and thread states.
I think the problem is that PyThreadState_Delete (which is used when I
destroy thread states and interpreters) is not making the same clean up
that PyThreadState_DeleteCurrent is doing (which is used when we create
threads from python code). If a thread id is reused (obviously not
between two running threads), and the previous thread used
PyThreadState_Delete instead of PyThreadState_DeleteCurrent, then an old
and invalid value for autoTLSkey is still stored, and that is
triggering the Py_FatalError when a call to PyThreadState_Swap is made.
If I add this code at the end of PyThreadState_Delete:
if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
PyThread_delete_key_value(autoTLSkey);
then everything works fine.
Could you please confirm if this is a bug ?
Thanks a lot and have a nice day.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pystate.c changes for Python 2.4.2
Michael Hudson wrote:
> Gabriel Becedillas <[EMAIL PROTECTED]> writes:
>
>
>>Hi,
>>At the company I work for, we've embedded Python in C++ application we
>>develop. Since our upgrade to Python 2.4.2 from 2.4.1 we started hitting
>>Py_FatalError("Invalid thread state for this thread") when using debug
>>builds.
>>We use both multiple interpreters and thread states.
>
>
> I know you've said this to me in email already, but I have no idea how
> you managed to get this to work with 2.4.1 :)
>
>
>>I think the problem is that PyThreadState_Delete (which is used when I
>>destroy thread states and interpreters) is not making the same clean up
>>that PyThreadState_DeleteCurrent is doing (which is used when we create
>>threads from python code). If a thread id is reused (obviously not
>>between two running threads), and the previous thread used
>>PyThreadState_Delete instead of PyThreadState_DeleteCurrent, then an old
>>and invalid value for autoTLSkey is still stored, and that is
>>triggering the Py_FatalError when a call to PyThreadState_Swap is made.
>>If I add this code at the end of PyThreadState_Delete:
>>
>>if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
>> PyThread_delete_key_value(autoTLSkey);
>>
>>then everything works fine.
>
>
> I think I begin to understand the problem... but this is still
> fragile: it relies on the fact that you delete a thread state from the
> OS level thread that created it, but while a thread belonging to a
> different *interpreter state* has the GIL (or at least: the
> interpreter state of the thread state being deleted *doesn't* hold the
> GIL). Can you guarantee that?
Mmm... it doesn't have to do with a race condition or something. Let me
try to show you with an example (keep in mind that this relies on the
fact that at some moment the operating system gives you a thread with
the same id of another thread that allready finished executing):
// Initialize python.
Py_Initialize();
PyEval_InitThreads();
PyThreadState* p_main = PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
/*
Asume this block is executed in a separate thread, and that has been
asigned thread id 10.
It doesn't matter what the main thread (the thread that initialized
Python) is doing. Lets assume its waiting for this one to finish.
*/
{
PyEval_AcquireLock();
PyThreadState_Swap(p_main);
PyThreadState* p_child = PyThreadState_New(p_main->interp);
PyThreadState_Swap(p_child);
PyRun_SimpleString("print 'mi vieja mula ya no es lo que era'");
PyThreadState_Clear(p_child);
PyThreadState_Swap(NULL);
PyThreadState_Delete(p_child);
PyEval_ReleaseLock();
// The os level thread exits here.
}
/*
When this thread state gets deleted (using PyThreadState_Delete), the
autoTLSkey stored for thread id number 10 is not removed, because
PyThreadState_Delete doesn't have the necesary cleanup code (check
pystate.c):
if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
PyThread_delete_key_value(autoTLSkey);
PyThreadState_DeleteCurrent behaves correctly because it does have this
code.
I think that this code should be added to tstate_delete_common (I've
done this and everything worked just fine).
If a new thread executes the same code, and that thread is assigned the
same thread id, you get the Py_FatalError I'm talking about.
A workaround for this would be to use PyThreadState_DeleteCurrent
instead of PyThreadState_Delete.
If you use the following code instead of the one above, the you have no
way out to the Py_FatalError:
*/
{
PyEval_AcquireLock();
PyThreadState* ret = Py_NewInterpreter();
PyRun_SimpleString("print 'mi vieja mula ya no es lo que era'");
Py_EndInterpreter(ret);
PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
}
/*
When this interpreter gets deleted (using Py_EndInterpreter), its thread
state gets deleted using PyThreadState_Delete, and you are back to the
beginning of the problem.
*/
I hope this helps to clarify the problem.
Thanks a lot and have a nice day.
> It seems to me that to be robust, you need a way of saying "delete the
> thread local value of autoTLSKey from thread $FOO". But this is all
> very confusing...
>
>
>>Could you please confirm if this is a bug ?
>
>
> Yes, I think it's a bug.
>
> Cheers,
> mwh
>
--
Gabriel Becedillas
Developer
CORE SECURITY TECHNOLOGIES
Florida 141 - 2º cuerpo - 7º piso
C1005AAC Buenos Aires - Argentina
Tel/Fax: (54 11) 5032-CORE (2673)
http://www.corest.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pystate.c changes for Python 2.4.2
Gabriel Becedillas wrote:
> Michael Hudson wrote:
>> Gabriel Becedillas <[EMAIL PROTECTED]> writes:
>>
>>
>>> Hi,
>>> At the company I work for, we've embedded Python in C++ application
>>> we develop. Since our upgrade to Python 2.4.2 from 2.4.1 we started
>>> hitting Py_FatalError("Invalid thread state for this thread") when
>>> using debug builds.
>>> We use both multiple interpreters and thread states.
>>
>>
>> I know you've said this to me in email already, but I have no idea how
>> you managed to get this to work with 2.4.1 :)
>>
>>
>>> I think the problem is that PyThreadState_Delete (which is used when I
>>> destroy thread states and interpreters) is not making the same clean up
>>> that PyThreadState_DeleteCurrent is doing (which is used when we
>>> create threads from python code). If a thread id is reused (obviously
>>> not between two running threads), and the previous thread used
>>> PyThreadState_Delete instead of PyThreadState_DeleteCurrent, then an
>>> old and invalid value for autoTLSkey is still stored, and that is
>>> triggering the Py_FatalError when a call to PyThreadState_Swap is made.
>>> If I add this code at the end of PyThreadState_Delete:
>>>
>>> if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
>>> PyThread_delete_key_value(autoTLSkey);
>>>
>>> then everything works fine.
>>
>>
>> I think I begin to understand the problem... but this is still
>> fragile: it relies on the fact that you delete a thread state from the
>> OS level thread that created it, but while a thread belonging to a
>> different *interpreter state* has the GIL (or at least: the
>> interpreter state of the thread state being deleted *doesn't* hold the
>> GIL). Can you guarantee that?
>
> Mmm... it doesn't have to do with a race condition or something. Let me
> try to show you with an example (keep in mind that this relies on the
> fact that at some moment the operating system gives you a thread with
> the same id of another thread that allready finished executing):
>
> // Initialize python.
> Py_Initialize();
> PyEval_InitThreads();
> PyThreadState* p_main = PyThreadState_Swap(NULL);
> PyEval_ReleaseLock();
>
> /*
> Asume this block is executed in a separate thread, and that has been
> asigned thread id 10.
> It doesn't matter what the main thread (the thread that initialized
> Python) is doing. Lets assume its waiting for this one to finish.
> */
> {
> PyEval_AcquireLock();
> PyThreadState_Swap(p_main);
> PyThreadState* p_child = PyThreadState_New(p_main->interp);
>
> PyThreadState_Swap(p_child);
> PyRun_SimpleString("print 'mi vieja mula ya no es lo que era'");
>
> PyThreadState_Clear(p_child);
> PyThreadState_Swap(NULL);
> PyThreadState_Delete(p_child);
> PyEval_ReleaseLock();
> // The os level thread exits here.
> }
> /*
> When this thread state gets deleted (using PyThreadState_Delete), the
> autoTLSkey stored for thread id number 10 is not removed, because
> PyThreadState_Delete doesn't have the necesary cleanup code (check
> pystate.c):
>
> if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
> PyThread_delete_key_value(autoTLSkey);
>
> PyThreadState_DeleteCurrent behaves correctly because it does have this
> code.
> I think that this code should be added to tstate_delete_common (I've
> done this and everything worked just fine).
> If a new thread executes the same code, and that thread is assigned the
> same thread id, you get the Py_FatalError I'm talking about.
> A workaround for this would be to use PyThreadState_DeleteCurrent
> instead of PyThreadState_Delete.
>
> If you use the following code instead of the one above, the you have no
> way out to the Py_FatalError:
> */
> {
> PyEval_AcquireLock();
> PyThreadState* ret = Py_NewInterpreter();
>
> PyRun_SimpleString("print 'mi vieja mula ya no es lo que era'");
>
> Py_EndInterpreter(ret);
> PyThreadState_Swap(NULL);
> PyEval_ReleaseLock();
> }
> /*
> When this interpreter gets deleted (using Py_EndInterpreter), its thread
> state gets deleted using PyThreadState_Delete, and you are back to the
> beginning of the problem.
> */
>
> I hope this helps to clarify the problem.
> Thanks a lot and have a nice day.
>
>> It seems to me that to be robust, you need a way of saying "delete the
>> threa
Re: [Python-Dev] pystate.c changes for Python 2.4.2
Thanks to everyone for the great support. I've submitted a patch for this: http://sourceforge.net/tracker/index.php?func=detail&aid=1413181&group_id=5470&atid=305470. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
