Re: [Python-Dev] HAVE_FSTAT?
I've had a quick look with grep -R HAVE_ * | egrep '[.]c:'. Modules/posixmodule.c has HAVE_UTIME_H and it might be standard libc on all posix platforms. Objects/obmalloc.c has HAVE_MMAP… but I guess that's fine given other platforms might not have such facility. Depending on the granularity (on a per platform or per feature) probably yes, there aren't many left. I hope this helps On 17 May 2013, at 16:56, Antoine Pitrou wrote: > On Fri, 17 May 2013 09:15:29 -0500 > Skip Montanaro wrote: >>> Some pieces of code are still guarded by: >>> #ifdef HAVE_FSTAT >>> ... >>> #endif >> >> Are there other guards for similarly common libc functions? > > I don't think so. Someone should take a look though :-) > > Regards > > Antoine. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 442: Safe object finalization
Hello, I would like to submit the following PEP for discussion and evaluation. Regards Antoine. PEP: 442 Title: Safe object finalization Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2013-05-18 Python-Version: 3.4 Post-History: Resolution: TBD Abstract This PEP proposes to deal with the current limitations of object finalization. The goal is to be able to define and run finalizers for any object, regardless of their position in the object graph. This PEP doesn't call for any change in Python code. Objects with existing finalizers will benefit automatically. Definitions === Reference A directional link from an object to another. The target of the reference is kept alive by the reference, as long as the source is itself alive and the reference isn't cleared. Weak reference A directional link from an object to another, which doesn't keep alive its target. This PEP focusses on non-weak references. Reference cycle A cyclic subgraph of directional links between objects, which keeps those objects from being collected in a pure reference-counting scheme. Cyclic isolate (CI) A reference cycle in which no object is referenced from outside the cycle *and* whose objects are still in a usable, non-broken state: they can access each other from their respective finalizers. Cyclic garbage collector (GC) A device able to detect cyclic isolates and turn them into cyclic trash. Objects in cyclic trash are eventually disposed of by the natural effect of the references being cleared and their reference counts dropping to zero. Cyclic trash (CT) A reference cycle, or former reference cycle, in which no object is referenced from outside the cycle *and* whose objects have started being cleared by the GC. Objects in cyclic trash are potential zombies; if they are accessed by Python code, the symptoms can vary from weird AttributeErrors to crashes. Zombie / broken object An object part of cyclic trash. The term stresses that the object is not safe: its outgoing references may have been cleared, or one of the objects it references may be zombie. Therefore, it should not be accessed by arbitrary code (such as finalizers). Finalizer A function or method called when an object is intended to be disposed of. The finalizer can access the object and release any resource held by the object (for example mutexes or file descriptors). An example is a ``__del__`` method. Resurrection The process by which a finalizer creates a new reference to an object in a CI. This can happen as a quirky but supported side-effect of ``__del__`` methods. Impact == While this PEP discusses CPython-specific implementation details, the change in finalization semantics is expected to affect the Python ecosystem as a whole. In particular, this PEP obsoletes the current guideline that "objects with a ``__del__`` method should not be part of a reference cycle". Benefits The primary benefits of this PEP regard objects with finalizers, such as objects with a ``__del__`` method and generators with a ``finally`` block. Those objects can now be reclaimed when they are part of a reference cycle. The PEP also paves the way for further benefits: * The module shutdown procedure may not need to set global variables to None anymore. This could solve a well-known class of irritating issues. The PEP doesn't change the semantics of: * Weak references caught in reference cycles. * C extension types with a custom ``tp_dealloc`` function. Description === Reference-counted disposal -- In normal reference-counted disposal, an object's finalizer is called just before the object is deallocated. If the finalizer resurrects the object, deallocation is aborted. *However*, if the object was already finalized, then the finalizer isn't called. This prevents us from finalizing zombies (see below). Disposal of cyclic isolates --- Cyclic isolates are first detected by the garbage collector, and then disposed of. The detection phase doesn't change and won't be described here. Disposal of a CI traditionally works in the following order: 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. The CI becomes a CT as the GC systematically breaks all known references inside it (using the ``tp_clear`` function). 3. Nothing. All CT objects should have been disposed of in step 2 (as a side-effect of clearing references); this collection is finished. This PEP proposes to turn CI disposal into the following sequence (new steps are in bold): 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. **The fina
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, May 18, 2013 at 6:59 PM, Antoine Pitrou wrote: > Resurrection > The process by which a finalizer creates a new reference to an > object in a CI. This can happen as a quirky but supported > side-effect of ``__del__`` methods. I really like the PEP overall, but could we at least get the option to have cases of object resurrection spit out a warning? And a clear rationale for not turning on such a warning by default? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, 18 May 2013 21:05:48 +1000 Nick Coghlan wrote: > On Sat, May 18, 2013 at 6:59 PM, Antoine Pitrou wrote: > > Resurrection > > The process by which a finalizer creates a new reference to an > > object in a CI. This can happen as a quirky but supported > > side-effect of ``__del__`` methods. > > I really like the PEP overall, but could we at least get the option to > have cases of object resurrection spit out a warning? And a clear > rationale for not turning on such a warning by default? Where would you put the option? As for the rationale, it's simply compatibility: resurrection works without warnings right now :) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, May 18, 2013 at 9:46 PM, Antoine Pitrou wrote: > On Sat, 18 May 2013 21:05:48 +1000 > Nick Coghlan wrote: >> On Sat, May 18, 2013 at 6:59 PM, Antoine Pitrou wrote: >> > Resurrection >> > The process by which a finalizer creates a new reference to an >> > object in a CI. This can happen as a quirky but supported >> > side-effect of ``__del__`` methods. >> >> I really like the PEP overall, but could we at least get the option to >> have cases of object resurrection spit out a warning? And a clear >> rationale for not turning on such a warning by default? > > Where would you put the option? > As for the rationale, it's simply compatibility: resurrection works > without warnings right now :) Command line, probably. However, you're right that's something we can consider later - for the PEP it's enough that it still works, and we just avoid calling the __del__ method a second time. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, 18 May 2013 22:51:35 +1000 Nick Coghlan wrote: > On Sat, May 18, 2013 at 9:46 PM, Antoine Pitrou wrote: > > On Sat, 18 May 2013 21:05:48 +1000 > > Nick Coghlan wrote: > >> On Sat, May 18, 2013 at 6:59 PM, Antoine Pitrou > >> wrote: > >> > Resurrection > >> > The process by which a finalizer creates a new reference to an > >> > object in a CI. This can happen as a quirky but supported > >> > side-effect of ``__del__`` methods. > >> > >> I really like the PEP overall, but could we at least get the option to > >> have cases of object resurrection spit out a warning? And a clear > >> rationale for not turning on such a warning by default? > > > > Where would you put the option? > > As for the rationale, it's simply compatibility: resurrection works > > without warnings right now :) > > Command line, probably. However, you're right that's something we can > consider later - for the PEP it's enough that it still works, and we > just avoid calling the __del__ method a second time. Actually, the __del__ method is called again on the next destruction attempt - as mentioned in the PEP: « Following this scheme, an object's finalizer is always called exactly once. The only exception is if an object is resurrected: the finalizer will be called again later. » I could change it to only call __del__ ever once, it just sounded more logical to call it each time destruction is attempted. (this is in contrast to weakrefs, though, which are cleared once and for all) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
Hi Antoine, On Sat, May 18, 2013 at 10:59 AM, Antoine Pitrou wrote: > Cyclic isolate (CI) > A reference cycle in which no object is referenced from outside the > cycle *and* whose objects are still in a usable, non-broken state: > they can access each other from their respective finalizers. Does this definition include more complicated cases? For example: A -> B -> Aand A -> C -> A Neither cycle is isolated. If there is no reference from outside, then the set of all three objects is isolated, but isn't strictly a cycle. I think the term is "strongly connected component". > 1. Weakrefs to CI objects are cleared, and their callbacks called. At >this point, the objects are still safe to use. > > 2. **The finalizers of all CI objects are called.** You need to be very careful about what each call to a finalizer can do to the object graph. It may already be what you're doing, but the most careful solution is to collect in "1." the complete list of objects with finalizers that are in cycles; then incref them all; then call the finalizer of each of them; then decref them all. Such a solution gives new cases to think about, which are slightly unexpected for CPython's model: for example, if you have a cycle A -> B -> A, let's say the GC calls A.__del__ first; it might cause it to store a reference to B somewhere else, e.g. in some global; but then the GC calls B.__del__ anyway. This is probably fine but should be considered. > 3. **The CI is traversed again to determine if it is still isolated. How is this done? I don't see a clear way to determine it by looking only at the objects in the CI, given that arbitrary modifications of the object graph may have occurred. The solution I can think of doesn't seem robust against minor changes done by the finalizer. Take the example "A -> lst -> B -> A", where the reference from A to B is via a list (e.g. there is an attribute "A.attr = [B]"). If A.__del__ does the seemingly innocent change of replacing the list with a copy of itself, e.g. "A.attr = A.attr[:]", then after the finalizers are called, "lst" is gone and we're left with "A -> lst2 -> B -> A". Checking that this cycle is still isolated requires a possibly large number of checks, as far as I can tell. This can lead to O(n**2) behavior if there are n objects in total and O(n) cycles. The solution seems to be to simply wait for the next GC execution. Assuming that a finalizer is only called once, this only delays a bit freeing objects with finalizers in cycles (but your PEP still works to call finalizers and eventually collect the objects). Alternatively, this might be done immediately: in the point "3." above we can forget everything we found so far, and redo the tracking on all objects (this time ignoring finalizers that were already called). In fact, it may be necessary anyway: anything found before might be invalid after the finalizers are called, so forgetting it all and redoing the tracking from scratch seems to be the only way. > Type objects get a new ``tp_finalize`` slot to which ``__del__`` methods > are bound. Generators are also modified to use this slot, rather than > ``tp_del``. At the C level, a ``tp_finalize`` function is a normal > function which will be called with a regular, alive object as its only > argument. It should not attempt to revive or collect the object. Do you mean the opposite in the latest sentence? ``tp_finalize`` can do anything... A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
Great PEP, I would really like to see this happen as it defines much saner semantics for finalization than what we currently have. One small question below: This PEP proposes to turn CI disposal into the following sequence (new > steps are in bold): > > 1. Weakrefs to CI objects are cleared, and their callbacks called. At >this point, the objects are still safe to use. > > 2. **The finalizers of all CI objects are called.** > > 3. **The CI is traversed again to determine if it is still isolated. >If it is determined that at least one object in CI is now reachable >from outside the CI, this collection is aborted and the whole CI >is resurrected. Otherwise, proceed.** > Not sure if my question is the same as Armin's here, but worth a try: by saying "the CI is traversed again" do you mean the original objects from the CI as discovered earlier, or is a new scan being done? What about a new object entering the CI during step (2)? I.e. the original CI was A->B->A but now one of the finalizers created some C such that B->C and C->A adding it to the connected component? Reading your description in (3) strictly it says: in this case the collection is aborted. This CI will be disposed next time collection is run. Is this correct? Eli > > 4. The CI becomes a CT as the GC systematically breaks all >known references inside it (using the ``tp_clear`` function). > > 5. Nothing. All CT objects should have been disposed of in step 4 >(as a side-effect of clearing references); this collection is >finished. > Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
Hi Armin, On Sat, 18 May 2013 15:24:08 +0200 Armin Rigo wrote: > Hi Antoine, > > On Sat, May 18, 2013 at 10:59 AM, Antoine Pitrou wrote: > > Cyclic isolate (CI) > > A reference cycle in which no object is referenced from outside the > > cycle *and* whose objects are still in a usable, non-broken state: > > they can access each other from their respective finalizers. > > Does this definition include more complicated cases? For example: > > A -> B -> Aand A -> C -> A > > Neither cycle is isolated. If there is no reference from outside, > then the set of all three objects is isolated, but isn't strictly a > cycle. I think the term is "strongly connected component". Yes, I should fix this definition to be more exact. > > 1. Weakrefs to CI objects are cleared, and their callbacks called. At > >this point, the objects are still safe to use. > > > > 2. **The finalizers of all CI objects are called.** > > You need to be very careful about what each call to a finalizer can do > to the object graph. It may already be what you're doing, but the > most careful solution is to collect in "1." the complete list of > objects with finalizers that are in cycles; then incref them all; then > call the finalizer of each of them; then decref them all. Such a > solution gives new cases to think about, which are slightly unexpected > for CPython's model: for example, if you have a cycle A -> B -> A, > let's say the GC calls A.__del__ first; it might cause it to store a > reference to B somewhere else, e.g. in some global; but then the GC > calls B.__del__ anyway. This is probably fine but should be > considered. Yes, I know this is possible. My opinion is that it is fine to call B's finalizer anyway. Calling all finalizers regardless of interim changes in the object graph also makes things a bit more deterministic: otherwise, which finalizers are called would depend on the call order, which is undefined. > > 3. **The CI is traversed again to determine if it is still isolated. > > How is this done? I don't see a clear way to determine it by looking > only at the objects in the CI, given that arbitrary modifications of > the object graph may have occurred. The same way a generation is traversed, but restricted to the CI. First the gc_refs field of each CI object is initialized to its ob_refcnt (again). Then, tp_traverse is called on each CI object, and each visited CI object has its gc_refs decremented. This substracts CI-internal references from the gc_refs fields. At the end of the traversal, if all CI objects have their gc_refs equal to 0, then the CI has no external reference to it and can be cleared. If at least one CI object has non-zero gc_refs, the CI cannot be cleared. > Alternatively, > this might be done immediately: in the point "3." above we can forget > everything we found so far, and redo the tracking on all objects (this > time ignoring finalizers that were already called). This would also be more costly, performance-wise. A CI should generally be quite small, but a whole generation is arbitrary big. > > Type objects get a new ``tp_finalize`` slot to which ``__del__`` methods > > are bound. Generators are also modified to use this slot, rather than > > ``tp_del``. At the C level, a ``tp_finalize`` function is a normal > > function which will be called with a regular, alive object as its only > > argument. It should not attempt to revive or collect the object. > > Do you mean the opposite in the latest sentence? ``tp_finalize`` can > do anything... Not exactly, but I worded it poorly. What I meant is that the C code in tp_finalize shouldn't *manually* revive the object, since it is called with an object with a strictly positive refcount. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, 18 May 2013 06:37:54 -0700 Eli Bendersky wrote: > Great PEP, I would really like to see this happen as it defines much saner > semantics for finalization than what we currently have. One small question > below: > > > This PEP proposes to turn CI disposal into the following sequence (new > > steps are in bold): > > > > 1. Weakrefs to CI objects are cleared, and their callbacks called. At > >this point, the objects are still safe to use. > > > > 2. **The finalizers of all CI objects are called.** > > > > 3. **The CI is traversed again to determine if it is still isolated. > >If it is determined that at least one object in CI is now reachable > >from outside the CI, this collection is aborted and the whole CI > >is resurrected. Otherwise, proceed.** > > > > Not sure if my question is the same as Armin's here, but worth a try: by > saying "the CI is traversed again" do you mean the original objects from > the CI as discovered earlier, or is a new scan being done? What about a new > object entering the CI during step (2)? I.e. the original CI was A->B->A > but now one of the finalizers created some C such that B->C and C->A adding > it to the connected component? It is the original CI which is traversed. If a new reference is introduced into the reference chain, the traversal in step 3 will decide to resurrect the CI. This is not necessarily a problem, since the next GC collection will try collecting again. > Reading your description in (3) strictly it says: in this case the > collection is aborted. This CI will be disposed next time collection is > run. Is this correct? Yup. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On 18/05/2013 9:59am, Antoine Pitrou wrote: This PEP proposes to turn CI disposal into the following sequence (new steps are in bold): 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. **The finalizers of all CI objects are called.** How do you know that one of the finalizers will not do something which causes another to fail? Presumably the following would cause an AttributeError to be printed: class Node: def __init__(self): self.next = None def __del__(self): print(self, self.next) del self.next # break Node object a = Node() b = Node() a.next = b b.next = a del a, b gc.collect() Are there are less contrived examples which will cause errors where currently there are none? -- Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, May 18, 2013 at 6:47 AM, Antoine Pitrou wrote: > On Sat, 18 May 2013 06:37:54 -0700 > Eli Bendersky wrote: > > Great PEP, I would really like to see this happen as it defines much > saner > > semantics for finalization than what we currently have. One small > question > > below: > > > > > > This PEP proposes to turn CI disposal into the following sequence (new > > > steps are in bold): > > > > > > 1. Weakrefs to CI objects are cleared, and their callbacks called. At > > >this point, the objects are still safe to use. > > > > > > 2. **The finalizers of all CI objects are called.** > > > > > > 3. **The CI is traversed again to determine if it is still isolated. > > >If it is determined that at least one object in CI is now reachable > > >from outside the CI, this collection is aborted and the whole CI > > >is resurrected. Otherwise, proceed.** > > > > > > > Not sure if my question is the same as Armin's here, but worth a try: by > > saying "the CI is traversed again" do you mean the original objects from > > the CI as discovered earlier, or is a new scan being done? What about a > new > > object entering the CI during step (2)? I.e. the original CI was A->B->A > > but now one of the finalizers created some C such that B->C and C->A > adding > > it to the connected component? > > It is the original CI which is traversed. If a new reference is > introduced into the reference chain, the traversal in step 3 will > decide to resurrect the CI. This is not necessarily a problem, since > the next GC collection will try collecting again. > > > Reading your description in (3) strictly it says: in this case the > > collection is aborted. This CI will be disposed next time collection is > > run. Is this correct? > > Yup. > Thanks, this actually makes a lot of sense. It's strictly better than the current situation where objects with __del__ are never collected. In the proposed scheme, the weird ones will be delayed and some really weird ones may never be collected, but the vast majority of __del__ methods do no resurrection so usually it will just work. This is a great proposal - killer new feature for 3.4 ;-) Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, 18 May 2013 14:56:38 +0100 Richard Oudkerk wrote: > On 18/05/2013 9:59am, Antoine Pitrou wrote: > > This PEP proposes to turn CI disposal into the following sequence (new > > steps are in bold): > > > > 1. Weakrefs to CI objects are cleared, and their callbacks called. At > > this point, the objects are still safe to use. > > > > 2. **The finalizers of all CI objects are called.** > > How do you know that one of the finalizers will not do something which > causes another to fail? > > Presumably the following would cause an AttributeError to be printed: > > class Node: > def __init__(self): > self.next = None > def __del__(self): > print(self, self.next) > del self.next # break Node object > > a = Node() > b = Node() > a.next = b > b.next = a > del a, b > gc.collect() It works fine: $ ./python sbt.py <__main__.Node object at 0x7f3acbf8f400> <__main__.Node object at 0x7f3acbf8f878> <__main__.Node object at 0x7f3acbf8f878> <__main__.Node object at 0x7f3acbf8f400> The reason is that, when you execute "del self.next", this removes the last reference to self.next and destroys it immediately. In essence, you were expecting to see: - enter a.__del__, destroy b - leave a.__del__ - enter b.__del__ oops? But what happens is: - enter a.__del__, destroy b - enter b.__del__ - leave b.__del__ - leave a.__del__ Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
Hi Antoine, On Sat, May 18, 2013 at 3:45 PM, Antoine Pitrou wrote: >> How is this done? I don't see a clear way to determine it by looking >> only at the objects in the CI, given that arbitrary modifications of >> the object graph may have occurred. > > The same way a generation is traversed, but restricted to the CI. > > First the gc_refs field of each CI object is initialized to its > ob_refcnt (again). > > Then, tp_traverse is called on each CI object, and each visited > CI object has its gc_refs decremented. This substracts CI-internal > references from the gc_refs fields. > > At the end of the traversal, if all CI objects have their gc_refs equal > to 0, then the CI has no external reference to it and can be cleared. > If at least one CI object has non-zero gc_refs, the CI cannot be > cleared. Ok, indeed. Then you really should call finalizers only once: in case one of the finalizers in a cycle did a trivial change like I described, the algorithm above will conservatively assume the cycle should be kept alive. At the next GC collection we must not call the finalizer again, because it's likely to just do a similar trivial change. (There are other open questions about calling finalizers multiple times; e.g. an instance of this class has its finalizer called ad infinitum and leaks, even though X() is never part of any cycle: class X(object): def __del__(self): print "tick" lst = [self] lst.append(lst) Try interactively: every gc.collect() prints "tick", even if you make only one instance.) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, 18 May 2013 16:22:55 +0200 Armin Rigo wrote: > Hi Antoine, > > On Sat, May 18, 2013 at 3:45 PM, Antoine Pitrou wrote: > >> How is this done? I don't see a clear way to determine it by looking > >> only at the objects in the CI, given that arbitrary modifications of > >> the object graph may have occurred. > > > > The same way a generation is traversed, but restricted to the CI. > > > > First the gc_refs field of each CI object is initialized to its > > ob_refcnt (again). > > > > Then, tp_traverse is called on each CI object, and each visited > > CI object has its gc_refs decremented. This substracts CI-internal > > references from the gc_refs fields. > > > > At the end of the traversal, if all CI objects have their gc_refs equal > > to 0, then the CI has no external reference to it and can be cleared. > > If at least one CI object has non-zero gc_refs, the CI cannot be > > cleared. > > Ok, indeed. Then you really should call finalizers only once: in case > one of the finalizers in a cycle did a trivial change like I > described, the algorithm above will conservatively assume the cycle > should be kept alive. At the next GC collection we must not call the > finalizer again, because it's likely to just do a similar trivial > change. Well, the finalizer will only be called if the resurrected object is dereferenced again; otherwise the object won't be considered by the GC. So, this will only happen if someone keeps trying to destroy a resurrected object. Calling finalizers only once is fine with me, but it would be a change in behaviour; I don't know if it may break existing code. (for example, say someone is using __del__ to manage a freelist) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On 18/05/2013 3:18pm, Antoine Pitrou wrote: It works fine: $ ./python sbt.py <__main__.Node object at 0x7f3acbf8f400> <__main__.Node object at 0x7f3acbf8f878> <__main__.Node object at 0x7f3acbf8f878> <__main__.Node object at 0x7f3acbf8f400> The reason is that, when you execute "del self.next", this removes the last reference to self.next and destroys it immediately. So even more contrived: class Node: def __init__(self, x): self.x = x self.next = None def __del__(self): print(self.x, self.next.x) del self.x a = Node(1) b = Node(2) a.next = b b.next = a del a, b gc.collect() -- Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On Sat, 18 May 2013 15:52:56 +0100 Richard Oudkerk wrote: > On 18/05/2013 3:18pm, Antoine Pitrou wrote: > > It works fine: > > > > $ ./python sbt.py > > <__main__.Node object at 0x7f3acbf8f400> <__main__.Node object at > > 0x7f3acbf8f878> > > <__main__.Node object at 0x7f3acbf8f878> <__main__.Node object at > > 0x7f3acbf8f400> > > > > The reason is that, when you execute "del self.next", this removes the > > last reference to self.next and destroys it immediately. > > So even more contrived: > > class Node: > def __init__(self, x): > self.x = x > self.next = None > def __del__(self): > print(self.x, self.next.x) > del self.x > > a = Node(1) > b = Node(2) > a.next = b > b.next = a > del a, b > gc.collect() Indeed, there is an exception during destruction (which is ignored as any exception raised from __del__): $ ./python sbt.py 1 2 Exception ignored in: > Traceback (most recent call last): File "sbt.py", line 17, in __del__ print(self.x, self.next.x) AttributeError: 'Node' object has no attribute 'x' The only reason this currently succeeds is that the objects end up in gc.garbage, of course. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 442: Safe object finalization
On 5/18/2013 11:22 AM, Antoine Pitrou wrote: On Sat, 18 May 2013 15:52:56 +0100 Richard Oudkerk wrote: So even more contrived: class Node: def __init__(self, x): self.x = x self.next = None def __del__(self): print(self.x, self.next.x) del self.x An attribute reference that can fail should be wrapped with try-except. a = Node(1) b = Node(2) a.next = b b.next = a del a, b gc.collect() Indeed, there is an exception during destruction (which is ignored as any exception raised from __del__): $ ./python sbt.py 1 2 Exception ignored in: > Traceback (most recent call last): File "sbt.py", line 17, in __del__ print(self.x, self.next.x) AttributeError: 'Node' object has no attribute 'x' Though ignored, the bug is reported, hinting that you should fix it ;-). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Undo the deprecation of _asdict().
18.05.13 10:06, raymond.hettinger написав(ла): http://hg.python.org/cpython/rev/1b760f926846 changeset: 83823:1b760f926846 user:Raymond Hettinger date:Sat May 18 00:05:20 2013 -0700 summary: Undo the deprecation of _asdict(). Why? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Use PY_FORMAT_SIZE_T because Visual Studio does not understand %zd format.
18.05.13 19:37, richard.oudkerk написав(ла): http://hg.python.org/cpython/rev/0648e7fe7a72 changeset: 83829:0648e7fe7a72 user:Richard Oudkerk date:Sat May 18 17:35:19 2013 +0100 summary: Use PY_FORMAT_SIZE_T because Visual Studio does not understand %zd format. See also DEBUG_PRINT_FORMAT_SPEC() in Python/formatter_unicode.c, _PyDebugAllocatorStats() in Objects/obmalloc.c, and kqueue_event_repr() in Modules/selectmodule.c. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Use PY_FORMAT_SIZE_T because Visual Studio does not understand %zd format.
18.05.13 23:00, Serhiy Storchaka написав(ла): 18.05.13 19:37, richard.oudkerk написав(ла): http://hg.python.org/cpython/rev/0648e7fe7a72 changeset: 83829:0648e7fe7a72 user:Richard Oudkerk date:Sat May 18 17:35:19 2013 +0100 summary: Use PY_FORMAT_SIZE_T because Visual Studio does not understand %zd format. See also DEBUG_PRINT_FORMAT_SPEC() in Python/formatter_unicode.c, _PyDebugAllocatorStats() in Objects/obmalloc.c, and kqueue_event_repr() in Modules/selectmodule.c. And _PyUnicode_Dump() in Objects/unicodeobject.c. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More compact dictionaries with faster iteration
On May 15, 2013, at 4:32 AM, Christian Tismer wrote: > What is the current status of this discussion? > I'd like to know whether it is a considered alternative implementation. As far as I can tell, I'm the only one working on it (and a bit slowly at that). My plan is to implement it for frozensets to see how it works out. Frozensets are a nice first experiment for several reasons: * The current implementation is cleaner than dictionaries (which have become more complicated due to key-sharing). * It will be easy to benchmark (by racing sets vs frozen sets) for an apples-to-apples comparison. * There is no need to have a list-like over-allocation scheme since frozensets can't grow after they are created. That will guarantee a significant space savings and it will simplify the coding. * I wrote the code for setobject.c so I know all the ins-and-outs. > > There is also a discussion in python-ideas right now where this > alternative is mentioned, and I think especially for small dicts > as **kwargs, it could be a cheap way to introduce order. The compaction of keys and values into a dense array was intended to save space, improve cache performance, and improve iteration speed. The ordering was just a side-effect and one that is easily disturbed if keys ever get deleted. So a compacted dict might be a cheap way to introduce order for kwargs, but it would need special handling if the user decided to delete keys. BTW, I'm +1 on the idea for ordering keyword-args. It makes it easier to debug if the arguments show-up in the order they were created. AFAICT, no purpose is served by scrambling them (which is exacerbated by the new randomized hashing security feature). Raymond___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Ordering keyword dicts
On Sat, May 18, 2013 at 10:27 PM, Raymond Hettinger wrote: > BTW, I'm +1 on the idea for ordering keyword-args. It makes > it easier to debug if the arguments show-up in the order they > were created. AFAICT, no purpose is served by scrambling them > (which is exacerbated by the new randomized hashing security feature). I'm slow at warming up to the idea. My main concern is speed -- since most code doesn't need it and function calls are already slow (and obviously very common :-) it would be a shame if this slowed down function calls that don't need it noticeably. An observation is that it's only necessary to preserve order if the function definition uses **kwds. AFAIK we currently don't know if this is the case when the call is made though, but perhaps the information could be made available to the call site somehow. There are also many special cases to consider; e.g. using **kwds in the call where kwds is an unordered dict, or calls from C, or calls to C. But maybe someone considers this a challenge and comes up with a patch? The benefits to *some* use cases would be obvious. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Purpose of Doctests [Was: Best practices for Enum]
On May 14, 2013, at 9:39 AM, Gregory P. Smith wrote: > Bad: doctests. I'm hoping that core developers don't get caught-up in the "doctests are bad meme". Instead, we should be clear about their primary purpose which is to test the examples given in docstrings. In many cases, there is a great deal of benefit to docstrings that have worked-out examples (see the docstrings in the decimal module for example). In such cases it is also worthwhile to make sure those examples continue to match reality. Doctests are a vehicle for such assurance. In other words, doctests have a perfectly legitimate use case. We should continue to encourage users to make thorough unit tests and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either. Another meme that I hope dispel is the notion that the core developers are free to break user code (such as doctests) if they believe the users aren't coding in accordance with best practices. Our goal is to improve their lives with our modifications, not to make their lives more difficult. Currently, we face an adoption problem with Python 3. At PyCon, an audience of nearly 2500 people said they had tried Python 3 but weren't planning to convert to it in production code. All of the coredevs are working to make Python 3 more attractive than Python 2, but we also have to be careful to not introduce obstacles to conversion. Breaking tests makes it much harder to convert (especially because people need to rely on their tests to see if the conversion was successful). Raymond P.S. Breaking doctests should also be seen as a "canary in a coal mine." When they break, it also means that printed examples are out of date, that code parsers may break, that diffs start being different, that programs that feed into other programs (perhaps via pipes and filters) may be changing their interface, etc. Occasionally, we make need to break such things but there should be a compelling offsetting benefit (i.e. evaluating thoughtfully whether "I'm trying to help you by making your constant integers have a nicer repr" is worth "Sorry, I broke your tests, made your published examples out of date, and slowed down your code." -- in some modules it will be worth it, but in others we should value stability over micro-improvments). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
