[Python-Dev] __del__ is not called after creating a new reference
Hello,
I already tried to ask on python-list, see
https://mail.python.org/pipermail/python-list/2017-March/720037.html
but it seems that this list is not for technical questions.
Let me resend my question to python-dev. Please tell me if I should not spam
this list with newbiesh questions, and thanks in advance.
---
I started to learn python a few days ago and I am trying to understand what
__del__() actually does. https://docs.python.org/3/reference/datamodel.html
says:
object.__del__(self)
...
Note that it is possible (though not recommended!) for the __del__()
method to postpone destruction of the instance by creating a new
reference to it. It may then be called at a later time when this new
reference is deleted.
However, this trivial test-case
class C:
def __del__(self):
print("DEL")
global X
X = self
C()
print(X)
X = 0
print(X)
shows that __del__ is called only once, it is not called again after "X = 0":
DEL
<__main__.C object at 0x7f067695f4a8>
0
(Just in case, I verified later that this object actually goes away and its
memory is freed, so the problem is not that it still has a reference).
I've cloned https://github.com/python/cpython.git and everything looks clear
at first glance (but let me repeat that I am very new to python):
PyObject_CallFinalizerFromDealloc() calls PyObject_CallFinalizer()
which finally calls "__del__" method in slot_tp_finalize(), then it
notices that "X = self" creates the new reference and does:
/* tp_finalize resurrected it! Make it look like the original Py_DECREF
* never happened.
*/
refcnt = self->ob_refcnt;
_Py_NewReference(self);
self->ob_refcnt = refcnt;
However, PyObject_CallFinalizer() also does _PyGC_SET_FINALIZED(self, 1)
and that is why __del__ is not called again after "X = 0":
/* tp_finalize should only be called once. */
if (PyType_IS_GC(tp) && _PyGC_FINALIZED(self))
return;
The comment and the code are very explicit, so this does nt look like a
bug in cpython.
Probably the docs should be fixed?
Or this code is actually wrong? The test-case works as documented if I
remove _PyGC_SET_FINALIZED() in PyObject_CallFinalizer() or add another
_PyGC_SET_FINALIZED(self, 0) into PyObject_CallFinalizerFromDealloc()
after _Py_NewReference(self), but yes, yes, I understand that this is
not correct and won't really help.
Oleg.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?
Hello,
Let me first clarify, I do not claim this is a bug, I am trying to learn
python and now I trying to understand yield-from.
This simple test-case
g = (x for x in range(10))
def fg():
for x in g:
yield x
print(next(fg()))
print(next(g))
works as expected and prints:
0
1
However, if I change fg() to use yield-from
g = (x for x in range(10))
def fg():
yield from g
print(next(fg()))
print(next(g))
then next(g) raises StopIteration:
0
Traceback (most recent call last):
File "/tmp/T.py", line 10, in
print(next(g))
StopIteration
because g.close() is called by destructor of the object returned by fg().
To me this looks strange and confusing. I tried to google, and found
https://docs.python.org/3/whatsnew/3.3.html#pep-380 but it doesn't document
this behaviour. I understand that yield-from should propagate .close(), but
why _PyGen_Finalize() should send close() to the gi_yieldfrom object?
I applied the patch below just to verify that I actually understand what's
going on, and with this patch the 2nd test-case works as I'd expect. But
since I am very new to python I'd suspect that the code is fine and I simply
do not understand why it works this way.
So. could someone please explain the rationale behind this behaviour? And
probably update the docs should be updated?
Thanks in advance.
Oleg.
---
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 24a1da6..d5152eb 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -6,6 +6,7 @@
#include "opcode.h"
static PyObject *gen_close(PyGenObject *, PyObject *);
+static PyObject *do_gen_close(PyGenObject *, PyObject *);
static PyObject *async_gen_asend_new(PyAsyncGenObject *, PyObject *);
static PyObject *async_gen_athrow_new(PyAsyncGenObject *, PyObject *);
@@ -71,7 +72,7 @@ _PyGen_Finalize(PyObject *self)
}
}
else {
-res = gen_close(gen, NULL);
+res = do_gen_close(gen, NULL);
}
if (res == NULL) {
@@ -373,10 +374,9 @@ _PyGen_yf(PyGenObject *gen)
}
static PyObject *
-gen_close(PyGenObject *gen, PyObject *args)
+do_gen_close(PyGenObject *gen, PyObject *yf)
{
PyObject *retval;
-PyObject *yf = _PyGen_yf(gen);
int err = 0;
if (yf) {
@@ -407,6 +407,11 @@ gen_close(PyGenObject *gen, PyObject *args)
return NULL;
}
+static PyObject *
+gen_close(PyGenObject *gen, PyObject *args)
+{
+return do_gen_close(gen, _PyGen_yf(gen));
+}
PyDoc_STRVAR(throw_doc,
"throw(typ[,val[,tb]]) -> raise exception in generator,\n\
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __del__ is not called after creating a new reference
On 03/20, Nathaniel Smith wrote:
>
> Modern CPython, and all extant versions of PyPy and Jython, guarantee that
> __del__ is called at most once. MicroPython doesn't support user-defined
> __del__ methods.
>
> It's fine if the text wants to leave that open, but the current phrasing is
> pretty misleading IMO. I also read it as saying that __del__ would be
> called again if the object is collected again (which may or may not
> happen).
Yes, that is why I was confused. Just I could not believe nobody else noticed
this "bug" so I decided to check the sources and yes, the code looks very clear.
> But AFAICT there are actually zero implementations where this is
> true.
Probably this was mostly true until the commit 796564c2 ("Issue #18112: PEP
442 implementation (safe object finalization)."), python2 calls __del__ again.
> Probably worth a small edit :-)
Agreed. And it seems that not only me was confused,
http://doc.pypy.org/en/latest/cpython_differences.html says:
There are a few extra implications from the difference in the GC. Most
notably, if an object has a __del__, the __del__ is never called more
than once in PyPy; but CPython will call the same __del__ several times
if the object is resurrected and dies again.
Thanks to all!
Oleg.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?
On 03/28, Eric Snow wrote: > > On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov wrote: > > Hello, > > > > Let me first clarify, I do not claim this is a bug, I am trying to learn > > python and now I trying to understand yield-from. > > Given that you haven't gotten a response here, and this looks a bit strange. My question is simple, the implementation looks clear and straightforward, I am a bit surprised none of cpython devs bothered to reply. > you may want to take > this over to the core-mentors...@python.org list. Well, I'm afraid to contact this closed and not-for-mortals list, not sure this very basic question should go there ;) perhaps you are already a member, feel free to forward. I downloaded micropython and it doesn't propagate .close() in this case. but it seem to differ very much from cpython, not sure this matters at all. Thanks, Oleg. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __del__ is not called after creating a new reference
On 04/02, Armin Rigo wrote:
>
> Hi all,
>
> On 20 March 2017 at 22:28, Nathaniel Smith wrote:
> > Modern CPython, and all extant versions of PyPy and Jython, guarantee that
> > __del__ is called at most once.
>
> Just a note, if someone actually depends on this: it is not true in
> all cases. For example, in CPython 3.5.3:
>
> >>> class X:
> ... __slots__=()# <= note this!
> ... def __del__(self):
> ... print("DEL")
> ... global resurrect
> ... resurrect = self
> ...
> >>> print(X())
> <__main__.X object at 0x7f5d1ad600d0>
> DEL
> >>> resurrect=None
> DEL
Objects/typeobject.c:type_new()
/* Enable GC unless this class is not adding new instance variables and
the base class did not use GC. */
if ((base->tp_flags & Py_TPFLAGS_HAVE_GC) ||
type->tp_basicsize > base->tp_basicsize)
type->tp_flags |= Py_TPFLAGS_HAVE_GC;
That is. "type->tp_basicsize > base->tp_basicsize" is false if empty __slots__,
so we do not set HAVE_GC.
And PyObject_CallFinalizer() doesn't set FINALIZED if PyType_IS_GC(tp) == F.
Oleg.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?
On 04/03, Stephen J. Turnbull wrote: > > Nathaniel Smith writes: > > > > Well, I'm afraid to contact this closed and not-for-mortals list, > > > not sure this very basic question should go there ;) perhaps you > > > are already a member, feel free to forward. > > > > core-mentorship is intended as a friendly place for folks who are > > starting to study CPython internals. I'm not sure where you got the > > impression that it's not-for-mortals but I suspect the people > > running it would like to know so they can fix it :-). > > I imagine Oleg got that impression because he tried to access the > archives, which are closed to non-members. Yes, so I came to conclusion that I should not bother this list with the very simple questions. > If it's something else, we would *definitely* like to know. No, no, sorry for confusion! And thanks to all. Oleg. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
