[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset 10e8b97d0fd7 by Antoine Pitrou in branch 'default': Make the reference counting of dictkeys objects participate in refleak hunting http://hg.python.org/cpython/rev/10e8b97d0fd7 -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Changes by Benjamin Peterson : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset c5fd332e5857 by Benjamin Peterson in branch 'default': change insertdict to not steal references (#13903) http://hg.python.org/cpython/rev/c5fd332e5857 -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Change insertdict to follow normal (non-stealing) ref-counting behaviour which fixes possible leakage. Patch attached. -- Added file: http://bugs.python.org/file25422/insertdict.patch ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: This patch integrates the dictkeys' refcounting into the refcount checking framework. Seems to work ok, but it would be better if someone more acquainted with the code could confirm it. -- Added file: http://bugs.python.org/file25385/dkdebug.patch __

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: > New changeset a3beae842f13 by Benjamin Peterson in branch 'default': > decref cached keys on type deallocation (#13903) > http://hg.python.org/cpython/rev/a3beae842f13 Is there any way to detect / avoid leaks on this separate refcounting scheme? --

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset a3beae842f13 by Benjamin Peterson in branch 'default': decref cached keys on type deallocation (#13903) http://hg.python.org/cpython/rev/a3beae842f13 -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Decref cached-keys when type is deallocated. Patch attached. -- Added file: http://bugs.python.org/file25381/cached_keys.patch ___ Python tracker __

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset 5d5b72a71898 by Benjamin Peterson in branch 'default': distiguish between refusing to creating shared keys and error (#13903) http://hg.python.org/cpython/rev/5d5b72a71898 -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset b044e0568be2 by Martin v. Loewis in branch 'default': Account for shared keys in type's __sizeof__ (#13903). http://hg.python.org/cpython/rev/b044e0568be2 -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Changes by Giampaolo Rodola' : -- nosy: -giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Failed to differentiate between failure and error in make_split_table(). Patch attached -- Added file: http://bugs.python.org/file25340/make_split_table_error.patch ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset 507a6703d6a3 by Benjamin Peterson in branch 'default': fix dict gc tracking (#13903) http://hg.python.org/cpython/rev/507a6703d6a3 -- ___ Python tracker _

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Failing to maintain GC tracking in setdefault and copy (for split-tables) Patch attached -- Added file: http://bugs.python.org/file25339/gc_tracking.patch ___ Python tracker __

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset 34b6998efd2c by Benjamin Peterson in branch 'default': fix instance dicts with str subclasses (#13903) http://hg.python.org/cpython/rev/34b6998efd2c -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: I fixed it back then, but didn't add the test. It subsequently regressed. Should know better. Patch (with test this time) attached. -- resolution: fixed -> status: closed -> open Added file: http://bugs.python.org/file25318/str_subclass.patch _

[issue13903] New shared-keys dictionary implementation

Yury Selivanov added the comment: Mark, did you add the test that your patch initially was failing with? http://mail.python.org/pipermail/python-dev/2012-February/116605.html -- ___ Python tracker ___

[issue13903] New shared-keys dictionary implementation

Benjamin Peterson added the comment: Okay. I committed the latest patch. Subtleties like __sizeof__ can be worked out as people start using it. -- ___ Python tracker ___ __

[issue13903] New shared-keys dictionary implementation

Roundup Robot added the comment: New changeset 6e5855854a2e by Benjamin Peterson in branch 'default': Implement PEP 412: Key-sharing dictionaries (closes #13903) http://hg.python.org/cpython/rev/6e5855854a2e -- nosy: +python-dev resolution: -> fixed stage: patch review -> committed/rej

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: I've updated the repository and uploaded a new patch in response to Benjamin's review. (And the contributor form is in the post). One remaining issue is the return value of __sizeof__(). If it is an int, then it cannot accurately reflect the memory use, but retu

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file25313/73423916a242.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Martin v. Löwis added the comment: Mark, can you please submit a contributor form? -- nosy: +loewis ___ Python tracker ___ ___ Python

[issue13903] New shared-keys dictionary implementation

Raymond Hettinger added the comment: Moving the "assigned to" to "nobody". I won't have a chance for a thorough review for another ten days. Hopefully, someone else will have a chance to review it before then. -- assignee: rhettinger -> ___ Pyth

[issue13903] New shared-keys dictionary implementation

Changes by Yury Selivanov : -- nosy: +Yury.Selivanov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: I don't really understand your objection to changing the method-cache size. As I said, I can remove the change, but that will cause the performance regression that Antoine noticed. -- ___ Python tracker

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: How about changing the method-cache size in a separate patch? On my machine, reducing the method-cache size from 2**10 to 2**9 results in a very small improvement in performance (with the original dict). That way we don't get a performance regression with the n

[issue13903] New shared-keys dictionary implementation

Raymond Hettinger added the comment: [Antoine] >I also think that, apart from the dict storage changes, > your patch should strive not to change any other tunables. I agree. Please keep the patch focused on the single task, the shared keys. -- __

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: > > Also, I don't get your remark about it running in a short time. Your > > patch AFAICT doesn't need any warm up period to exhibit any > > improvements. > > What I mean is that the runtime is so short, no one would notice any > change, so who cares? None of

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Antoine Pitrou wrote: > Antoine Pitrou added the comment: > >> I'm not bothered by the regression in "silent_logging", >> as it is a micro benchmark with a very short running time. > > I'm not concerned about the micro-benchmark itself but the fact that it > mi

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: > I'm not bothered by the regression in "silent_logging", > as it is a micro benchmark with a very short running time. I'm not concerned about the micro-benchmark itself but the fact that it might hint at a wider problem. Also, I don't get your remark about it

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: I'm not bothered by the regression in "silent_logging", as it is a micro benchmark with a very short running time. The regression in "mako" is, I think, caused by competition for the data cache between the new dict implementation and the method-cache used by _PyT

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file25096/372d0bca85ae.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: The latest patch has a significant (negative) effect on some benchmarks: ### silent_logging ### Min: 0.057927 -> 0.068228: 1.18x slower Avg: 0.058218 -> 0.068660: 1.18x slower Significant (t=-36.06) ### mako ### Min: 0.118240 -> 0.140451: 1.19x slower Avg: 0.1

[issue13903] New shared-keys dictionary implementation

Jim Jewett added the comment: On Fri, Mar 9, 2012 at 12:13 PM, Jim Jewett > So to get beneath 2/3 without lots of reallocation > probably requires knowing when the key set is likely > to be complete, and that is indeed more complex than > the current changes.  (That said, you have left code > i

[issue13903] New shared-keys dictionary implementation

Jim Jewett added the comment: >> As of Feb 28, 2012, the PEP mentions an additional >> optimization of storing the values in an array indexed >> by (effectively) key insertion order, rather than key >> position. ("Alternative Implementation") >> It states that this would reduce memory usage for

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24766/257e16e71654.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Jim Jewett wrote: > Jim Jewett added the comment: > > As of Feb 28, 2012, the PEP mentions an additional optimization of storing > the values in an array indexed by (effectively) key insertion order, rather > than key position. ("Alternative Implementation") >

[issue13903] New shared-keys dictionary implementation

Jim Jewett added the comment: As of Feb 28, 2012, the PEP mentions an additional optimization of storing the values in an array indexed by (effectively) key insertion order, rather than key position. ("Alternative Implementation") It states that this would reduce memory usage for the values a

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24670/49b7e7e4a27c.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Removed file: http://bugs.python.org/file24465/e50db1b7ad7b.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24510/691ce331f955.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Removed file: http://bugs.python.org/file24456/20702d1acf17.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24465/e50db1b7ad7b.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24456/20702d1acf17.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Removed file: http://bugs.python.org/file24455/bc286099ce9a.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24455/bc286099ce9a.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Removed file: http://bugs.python.org/file24454/1f703b2607af.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Antoine Pitrou wrote: > Antoine Pitrou added the comment: > > Looking at your latest patch, I worry about "any deletion > +(including pop & popitem) causes a split table to become a combined table". > Why wouldn't you use a dummy pointer (such as ((PyObject *)

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: Looking at your latest patch, I worry about "any deletion +(including pop & popitem) causes a split table to become a combined table". Why wouldn't you use a dummy pointer (such as ((PyObject *) 1)) to signal deleted slots? -- ___

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24454/1f703b2607af.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Removed file: http://bugs.python.org/file24453/a9138aba7896.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Removed file: http://bugs.python.org/file24360/6a21f3b35e20.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24453/a9138aba7896.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : -- hgrepos: +112 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : -- hgrepos: -109 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue13903] New shared-keys dictionary implementation

Gregory P. Smith added the comment: FYI - I strongly support this type of work to reduce memory use of the Python interpreter! :) Also, yes, constant changing should be separate from this change but are worth occasionally re-measuring and justifying as common computer architectures have cha

[issue13903] New shared-keys dictionary implementation

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue13903] New shared-keys dictionary implementation

Antoine Pitrou added the comment: As Victor I think the tunables could be changed separately, unless they truely get in the way of the shared-keys optimization. By the way, there will need a "protection" for the case where instances are used as bags of key/value pairs (dict-alikes with attrib

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Raymond Hettinger wrote: > Raymond Hettinger added the comment: > > Changing dictionaries is a big deal. You're changing many pieces at once > (not a good idea) including changing tunable parameters that are well-studied > (I spent a month testing whether 5/8

[issue13903] New shared-keys dictionary implementation

Raymond Hettinger added the comment: Changing dictionaries is a big deal. You're changing many pieces at once (not a good idea) including changing tunable parameters that are well-studied (I spent a month testing whether 5/8 was better idea that 2/3 for resizing or when the ideal small dict

[issue13903] New shared-keys dictionary implementation

Changes by Raymond Hettinger : -- assignee: -> rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue13903] New shared-keys dictionary implementation

Mark Shannon added the comment: Does this really need a PEP? There is no new language feature and no change to any API. It is just saving some memory. The only possible issue is changing dict repr() ordering, but issue 13703 will do that anyway. -- ___

[issue13903] New shared-keys dictionary implementation

Raymond Hettinger added the comment: This would likely require a PEP before having a chance of being considered for inclusion. -- nosy: +rhettinger ___ Python tracker ___ _

[issue13903] New shared-keys dictionary implementation

Changes by John O'Connor : -- nosy: +jcon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue13903] New shared-keys dictionary implementation

STINNER Victor added the comment: I see two unrelated parts in your patch: - change dictionary structure in memory - change many constants linked to optimization: PyDICT_MAXFREELIST: 80->40, 2/3->5/8, etc. You may open a new issue for the second part, except if I am wrong and you need to ch

[issue13903] New shared-keys dictionary implementation

Changes by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue13903] New shared-keys dictionary implementation

Changes by Philip Jenvey : -- nosy: +pjenvey ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue13903] New shared-keys dictionary implementation

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue13903] New shared-keys dictionary implementation

Terry J. Reedy added the comment: In the initial comment, 'Dummy' to 'Deleted' here but only here: - Holds an active (key, value) pair. Active can transition to Dummy + Holds an active (key, value) pair. Active can transition to Deleted Im Lib/test/test_pprint.py def test_set_reprs

[issue13903] New shared-keys dictionary implementation

Changes by Antoine Pitrou : -- nosy: +benjamin.peterson, pitrou stage: -> patch review versions: +Python 3.3 -Python 3.4 ___ Python tracker ___ _

[issue13903] New shared-keys dictionary implementation

Changes by Georg Brandl : Removed file: http://bugs.python.org/file24357/061f8573af54.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13903] New shared-keys dictionary implementation

Changes by Mark Shannon : Added file: http://bugs.python.org/file24360/6a21f3b35e20.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13903] New shared-keys dictionary implementation

Changes by Georg Brandl : -- keywords: +patch Added file: http://bugs.python.org/file24357/061f8573af54.diff ___ Python tracker ___ __

[issue13903] New shared-keys dictionary implementation

New submission from Mark Shannon : The proposed dictionary implementation allows sharing of keys & hashes between dictionaries. This leads to substantial memory savings for object-oriented programs. For non-OO programs the impact is negligible. -- components: Interpreter Core hgrepos: