Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Martin v. Löwis
>> Most Py_DECREF calls are probably okay but it's going to be hard >> to find the ones that are not. > > I suppose Py_DECREF is not the only source of trouble. Many calls > to the Python API can end up calling arbitrary user code (via > __getattr__, __getitem__, etc.). Whenever an object does t

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Daniel Stutzbach
On Feb 18, 2008 4:52 PM, Neil Schemenauer <[EMAIL PROTECTED]> wrote: > That sucks. Most Py_DECREF calls are probably okay but it's going > to be hard to find the ones that are not. I can't think of anything > we can do to make this trap harder to fall into. Even using > Py_CLEAR as a blunt tool

[Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Jim Jewett
> A simple way to do this would be to push objects whose > refcounts had reached 0 onto a list instead of finalizing them > immediately, and have PyEval_EvalFrameEx periodically swap > in a new to-delete list and delete the objects on the old one. Some of the memory management threads discussed so

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Neil Schemenauer
I wrote: > Most Py_DECREF calls are probably okay but it's going to be hard > to find the ones that are not. I suppose Py_DECREF is not the only source of trouble. Many calls to the Python API can end up calling arbitrary user code (via __getattr__, __getitem__, etc.). Whenever an object does th

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Martin v. Löwis
> That sucks. Most Py_DECREF calls are probably okay but it's going > to be hard to find the ones that are not. Methinks that egrep 'DECREF\([a-zA-Z0-9_]->[a-zA-Z0-9_]+\)' */*.c gives a good overview - you should never DECREF a variable on heap. In the trunk, this command finds 36 candidates. N

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Neil Schemenauer
On Mon, Feb 18, 2008 at 05:48:57PM +0100, Amaury Forgeot d'Arc wrote: > For example, in exception.c, BaseException_init() starts with the instruction: > Py_DECREF(self->args); > this may call __del__ on self->args Ah, I understand now. We are not talking about tp_dealloc methods (the GC takes

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Jeffrey Yasskin
On Feb 17, 2008 12:29 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Jeffrey Yasskin wrote: > > On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote: > >> Should we however intensively search and correct all of them? > >> Is there a clever way to prevent these problems global

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Amaury Forgeot d'Arc
Hello, Neil Schemenauer wrote: > Nick Coghlan <[EMAIL PROTECTED]> wrote: > > The problem is calls to Py_DECREF(self->attr) where some of the code > > invoked by __del__ manages to find a way back around to reference > > self->attr and gets access to a half-deleted object. > > Don't you mean "__del_

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Neil Schemenauer
Nick Coghlan <[EMAIL PROTECTED]> wrote: > The problem is calls to Py_DECREF(self->attr) where some of the code > invoked by __del__ manages to find a way back around to reference > self->attr and gets access to a half-deleted object. Don't you mean "__del__ manages to find a way back around to s

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-18 Thread Nick Coghlan
Greg Ewing wrote: > Martin v. Löwis wrote: >> when some caller of PyEval_EvalFrameEx still carries >> a pointer to some object that got deleted, and then still some code can >> get hold of the then-deleted object. > > I seem to have missed the beginning of this discussion. > I don't see what the p

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-17 Thread Greg Ewing
Martin v. Löwis wrote: > when some caller of PyEval_EvalFrameEx still carries > a pointer to some object that got deleted, and then still some code can > get hold of the then-deleted object. I seem to have missed the beginning of this discussion. I don't see what the problem is here. If a caller n

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-17 Thread Martin v. Löwis
Jeffrey Yasskin wrote: > On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote: >> Should we however intensively search and correct all of them? >> Is there a clever way to prevent these problems globally, for example >> by delaying finalizers "just a little"? > > A simple way to

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-17 Thread Jeffrey Yasskin
On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote: > Should we however intensively search and correct all of them? > Is there a clever way to prevent these problems globally, for example > by delaying finalizers "just a little"? A simple way to do this would be to push object

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-16 Thread Nick Coghlan
Brett Cannon wrote: > On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote: >> Is there a clever way to prevent these problems globally, for example >> by delaying finalizers "just a little"? > > Beats me. Finalizers can be delayed 'just a little' with a macro called Py_CLEAR ;

Re: [Python-Dev] Py_CLEAR to avoid crashes

2008-02-16 Thread Brett Cannon
On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote: > Hello, > > I think I found a prolific vein of potential crashes throughout the > python interpreter. > The idea is inspired from the discussion Issue1020188 and is quite > simple: find a Py_DECREF(xxx->yyy), where xxx repres

[Python-Dev] Py_CLEAR to avoid crashes

2008-02-16 Thread Amaury Forgeot d'Arc
Hello, I think I found a prolific vein of potential crashes throughout the python interpreter. The idea is inspired from the discussion Issue1020188 and is quite simple: find a Py_DECREF(xxx->yyy), where xxx represents any kind of Python object, and craft a __del__ method so that the same function