Re: [Python-Dev] PEP 509: Add a private version to dict (version 3)

Hi, Guido van Rossum and Jim J. Jewett suggested me to *not require* to always increase the dict version if a dict method does not modify its content. I modified the Changes section to only require that the version is increased when the dictionary content is modified. I also explained the nice si

Re: [Python-Dev] PEP 509: Add a private version to dict (version 3)

Hi, > Backwards Compatibility > === > > Since the ``PyDictObject`` structure is not part of the stable ABI and > the new dictionary version not exposed at the Python scope, changes are > backward compatible. My current implementation inserts the new ma_version_tag field in the

[Python-Dev] PEP 509: Add a private version to dict (version 3)

d is incremented when the dictionary is modified. PySizer --- `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python, Google Summer of Code 2005 project by Nick Smallbone. This project has a patch for CPython 2.4 which adds ``key_time`` and ``value_time`` fields to dictionary e

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-21 3:34 AM, Victor Stinner wrote: [..] But if a global counter doesn't make the slow more complex and opens new kinds of optimization, I agree to change my PEP 509. Please. That would allow us to use ma_version to implement caches in CPython itself. Yury

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 6:08 GMT+01:00 Yury Selivanov : > Yeah, I think that's what we agreed on: > https://mail.python.org/pipermail/python-dev/2016-January/142837.html > > The only advantage of ma_extra pointer is that it allows to add more stuff > to dicts in the future. I don't agree on ma_extra since I do

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 2:54 GMT+01:00 Andrew Barnert : > This idea worries me. I'm not sure why, but I think because of threading. > After all, it's pretty rare for two threads to both want to work on the same > dict, but very, very common for two threads to both want to work on _any_ > dict. So, imagine so

Re: [Python-Dev] PEP 509: Add a private version to dict

Andrew Barnert via Python-Dev wrote: imagine someone manages to remove the GIL from CPython by using STM: now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, If this becomes a problem, the tag could be split into two parts of m and n bit

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 5:01 PM, Greg Ewing wrote: Yury Selivanov wrote: What I propose is to add a pointer "ma_extra" (same 64bits), which will be set to NULL for most dict instances (instead of ma_version). "ma_extra" can then point to a struct that has a globally unique dict ID (uint64), and a versi

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 8:54 PM, Andrew Barnert via Python-Dev wrote: >I think Glenn was assuming we had a single, global version # that all dicts shared without having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for all dictionaries bu

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016, 17:54 Andrew Barnert wrote: > On Wednesday, January 20, 2016 4:10 PM, Brett Cannon > wrote: > > > >I think Glenn was assuming we had a single, global version # that all > dicts shared without having a per-dict version ID. The key thing here is > that we have a global counter

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wednesday, January 20, 2016 4:10 PM, Brett Cannon wrote: >I think Glenn was assuming we had a single, global version # that all dicts >shared without having a per-dict version ID. The key thing here is that we >have a global counter that tracks the number of mutations for all dictionaries

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 1:08 GMT+01:00 Brett Cannon : > On Wed, 20 Jan 2016 at 15:46 Victor Stinner >> The worst case is when a value different than the watched value is >> modified between each guard check. In this case, we always need a dict >> lookup. An heuristic can be chosen to decide to give up after N

Re: [Python-Dev] PEP 509: Add a private version to dict

On 1/20/2016 4:08 PM, Brett Cannon wrote: On Wed, 20 Jan 2016 at 15:46 Victor Stinner > wrote: Hi, 2016-01-20 22:18 GMT+01:00 Glenn Linderman mailto:v%2bpyt...@g.nevcal.com>>: > On 1/20/2016 12:50 PM, Brett Cannon wrote: >> >> A global (sh

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 15:46 Victor Stinner wrote: > Hi, > > 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > > On 1/20/2016 12:50 PM, Brett Cannon wrote: > >> > >> A global (shared between all dicts) unit64 ma_version is actually quite > >> reliable -- if a program does 1,000,000 dict modification

Re: [Python-Dev] PEP 509: Add a private version to dict

Hi, 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > On 1/20/2016 12:50 PM, Brett Cannon wrote: >> >> A global (shared between all dicts) unit64 ma_version is actually quite >> reliable -- if a program does 1,000,000 dict modifications per second, >> it would take it 600,000 years till wrap-around.

Re: [Python-Dev] PEP 509: Add a private version to dict

Yury Selivanov wrote: What I propose is to add a pointer "ma_extra" (same 64bits), which will be set to NULL for most dict instances (instead of ma_version). "ma_extra" can then point to a struct that has a globally unique dict ID (uint64), and a version tag (unit64). Why not just use a single

Re: [Python-Dev] PEP 509: Add a private version to dict

On 1/20/2016 12:50 PM, Brett Cannon wrote: A global (shared between all dicts) unit64 ma_version is actually quite reliable -- if a program does 1,000,000 dict modifications per second, it would take it 600,000 years till wrap-around. But would invalidate everything, instead o

Re: [Python-Dev] PEP 509: Add a private version to dict

On 11/01/16 16:49, Victor Stinner wrote: Hi, After a first round on python-ideas, here is the second version of my PEP. The main changes since the first version are that the dictionary version is no more exposed at the Python level and the field type now also has a size of 64-bit on 32-bit pla

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 12:27 Yury Selivanov wrote: > > > On 2016-01-20 2:45 PM, Glenn Linderman wrote: > > For the reuse case, can't you simply keep the ma_version "live" in > > dict items on the free list, rather than starting over at (presumably) > > 0 ? Then if the dict is reused, it bumps th

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 2:45 PM, Glenn Linderman wrote: For the reuse case, can't you simply keep the ma_version "live" in dict items on the free list, rather than starting over at (presumably) 0 ? Then if the dict is reused, it bumps the ma_version, and the fallback code goes on with (presumably) rel

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 10:46 Yury Selivanov wrote: > Brett, > > On 2016-01-20 1:22 PM, Brett Cannon wrote: > > > > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov > > wrote: > > > > On 2016-01-18 5:43 PM, Victor Stinner wrote: > > > Is someone opposed to t

Re: [Python-Dev] PEP 509: Add a private version to dict

On 1/20/2016 10:36 AM, Maciej Fijalkowski wrote: Why can't you simply use the id of the dict object as the globally unique >dict ID? It's already globally unique amongst all Python objects which makes >it inherently unique amongst dicts. > >___ >Python-

Re: [Python-Dev] PEP 509: Add a private version to dict

there is also the problem that you don't want it on all dicts. So having two extra words is more to pay than having extra objects (also, comparison is cheaper for guards) On Wed, Jan 20, 2016 at 8:23 PM, Yury Selivanov wrote: > > > On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote: >>> >>> > >> >>

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote: > You don't free a version tag that's stored in the guard. You store the object and not id Ah, got it. Although for my current cache design it would be more memory efficient to use the dict itself to store its own unique id and tag, hence my

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, Jan 20, 2016 at 8:08 PM, Yury Selivanov wrote: > > On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote: >> >> On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov >> wrote: >> > [..] Brett, you need two things - the ID of the dict and the version tag. What we do in pypy is we have a

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote: On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov wrote: [..] Brett, you need two things - the ID of the dict and the version tag. What we do in pypy is we have a small object (called, surprisingly, VersionTag()) and we use the ID of that. That

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov wrote: > > > On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote: >> >> On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: >>> >>> >>> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov >>> wrote: > > [..] "ma_extra" would also make it easier for

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote: On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: [..] "ma_extra" would also make it easier for us to extend dicts in the future. Why can't you simply use the id of the dict object as

Re: [Python-Dev] PEP 509: Add a private version to dict

Brett, On 2016-01-20 1:22 PM, Brett Cannon wrote: On Wed, 20 Jan 2016 at 10:11 Yury Selivanov > wrote: On 2016-01-18 5:43 PM, Victor Stinner wrote: > Is someone opposed to this PEP 509? > > The main complain was the change on the public Python

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 1:15 PM, Maciej Fijalkowski wrote: [..] Anyway, I would strongly suggest getting some benchmarks done and showing performance benefits first, because you don't want PEPs to be final when you don't exactly know the details. I agree 100%. Yury

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: >> >> On 2016-01-18 5:43 PM, Victor Stinner wrote: >> > Is someone opposed to this PEP 509? >> > >> > The main complain was the change on the public Python API, but the PEP >> > doesn't c

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: > On 2016-01-18 5:43 PM, Victor Stinner wrote: > > Is someone opposed to this PEP 509? > > > > The main complain was the change on the public Python API, but the PEP > > doesn't change the Python API anymore. > > > > I'm not aware of any remainin

Re: [Python-Dev] PEP 509: Add a private version to dict

The easiest version is to have global numbering (as opposed to local). Anyway, I would strongly suggest getting some benchmarks done and showing performance benefits first, because you don't want PEPs to be final when you don't exactly know the details. On Wed, Jan 20, 2016 at 7:02 PM, Yury Seliv

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-18 5:43 PM, Victor Stinner wrote: Is someone opposed to this PEP 509? The main complain was the change on the public Python API, but the PEP doesn't change the Python API anymore. I'm not aware of any remaining issue on this PEP. Victor, I've been experimenting with the PEP to imp

Re: [Python-Dev] PEP 509: Add a private version to dict

On Jan 18, 2016, at 11:43 PM, Victor Stinner wrote: >Is someone opposed to this PEP 509? > >The main complain was the change on the public Python API, but the PEP >doesn't change the Python API anymore. > >I'm not aware of any remaining issue on this PEP. Have you done any performance analysis fo

Re: [Python-Dev] PEP 509: Add a private version to dict

Is someone opposed to this PEP 509? The main complain was the change on the public Python API, but the PEP doesn't change the Python API anymore. I'm not aware of any remaining issue on this PEP. Victor 2016-01-11 17:49 GMT+01:00 Victor Stinner : > Hi, > > After a first round on python-ideas, h

Re: [Python-Dev] PEP 509

Terry Reedy writes: > While I understand the rationale against __version__, it strikes me as a > better description of what it is, and easier on the brain than > __cache_token__. Maybe there is something even better, such as > __seqnum__. Or __generation__, as in "generational garbage col

Re: [Python-Dev] PEP 509

On 01/12/2016 03:24 PM, Victor Stinner wrote: > 2016-01-12 23:24 GMT+01:00 Ethan Furman wrote: >> Even if not exposed at the Python layer, it's still exposed when >> working at the C layer. Is __version__ any less confusing there? >> (I only work in C when working on Python, and only occasionall

Re: [Python-Dev] PEP 509

2016-01-12 23:24 GMT+01:00 Ethan Furman : > Even if not exposed at the Python layer, it's still exposed when working at > the C layer. Is __version__ any less confusing there? (I only work in C > when working on Python, and only occasionally, so my question is real.) Fields of the PyDictObject m

Re: [Python-Dev] PEP 509

On 1/12/2016 5:24 PM, Ethan Furman wrote: On 01/12/2016 01:34 PM, Victor Stinner wrote: 2016-01-12 19:52 GMT+01:00 Ethan Furman : [1] We're not going to call it __version__ are we? Seems like __cache_token__ is a much better name. While I understand the rationale against __version__, it stri

Re: [Python-Dev] PEP 509

On 01/12/2016 01:34 PM, Victor Stinner wrote: 2016-01-12 19:52 GMT+01:00 Ethan Furman : [1] We're not going to call it __version__ are we? Seems like __cache_token__ is a much better name. See the online version to the most recent version of the PEP: https://www.python.org/dev/peps/pep-0509/

Re: [Python-Dev] PEP 509

2016-01-12 19:34 GMT+01:00 Jim J. Jewett : > (1) Please make it clear within the abstract what counts as a change. I don't think that an abstract must give the long list of cases when the version is modified or not. It's explained in detail at: https://www.python.org/dev/peps/pep-0509/#changes >

Re: [Python-Dev] PEP 509

2016-01-12 19:52 GMT+01:00 Ethan Furman : > [1] We're not going to call it __version__ are we? Seems like > __cache_token__ is a much better name. See the online version to the most recent version of the PEP: https://www.python.org/dev/peps/pep-0509/ In the first version I proposed to expose the

Re: [Python-Dev] PEP 509

On 01/12/2016 10:34 AM, Jim J. Jewett wrote: (1c) Section "Guard against changing dict during iteration" says "Sadly, the dictionary version proposed in this PEP doesn't help to detect dictionary mutation." Why not? Wouldn't that mutation involve replacing a value, which ought to trigger a ve

[Python-Dev] PEP 509

(1) Please make it clear within the abstract what counts as a change. (1a) E.g., a second paragraph such as "Adding or removing a key, or replacing a value, counts as a change. Modifying an object in place, or replacing it with itself may not be picked up." (1b) Is there a way to force a vers

Re: [Python-Dev] PEP 509: Add a private version to dict

Well, it was just a remark. 2016-01-12 0:35 GMT+01:00 Andrew Barnert : > Are you saying that d[key] = d[key] may or may not increment the version, so > any optimizer can't rely on the fact that it doesn't? Optimizers don't have to rely on this exactly behaviour. Not incrementing the version on s

Re: [Python-Dev] PEP 509: Add a private version to dict

On Jan 11, 2016, at 15:24, Victor Stinner wrote: > > 2016-01-12 0:07 GMT+01:00 Gregory P. Smith : >>> Changes >>> === >>> >>> (...) >> >> Please be more explicit about what tests you are performing on the values. >> setitem's "if the value is different" really should mean "if value is not >

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-12 0:07 GMT+01:00 Gregory P. Smith : >> Changes >> === >> >> (...) > > Please be more explicit about what tests you are performing on the values. > setitem's "if the value is different" really should mean "if value is not > dict['key']". similarly for update, there should never be equa

Re: [Python-Dev] PEP 509: Add a private version to dict

On Mon, Jan 11, 2016 at 8:50 AM Victor Stinner wrote: > Hi, > > After a first round on python-ideas, here is the second version of my > PEP. The main changes since the first version are that the dictionary > version is no more exposed at the Python level and the field type now > also has a size o

Re: [Python-Dev] PEP 509: Add a private version to dict

On Mon, Jan 11, 2016 at 9:56 PM, Victor Stinner wrote: > Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski" a écrit : >> Hi Victor. >> >> You know that pypy does this stuff without changing and exposing >> python semantics right? We have a version dict that does not leak >> abstractions to the user.

Re: [Python-Dev] PEP 509: Add a private version to dict

Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski" a écrit : > Hi Victor. > > You know that pypy does this stuff without changing and exposing > python semantics right? We have a version dict that does not leak > abstractions to the user. The PEP adds a field to the C structure PyDictObject. Are you

Re: [Python-Dev] PEP 509: Add a private version to dict

Hi Victor. You know that pypy does this stuff without changing and exposing python semantics right? We have a version dict that does not leak abstractions to the user. In general, doing stuff like that where there is a public API that leaks details of certain optimizations makes it harder and har

[Python-Dev] PEP 509: Add a private version to dict

Hi, After a first round on python-ideas, here is the second version of my PEP. The main changes since the first version are that the dictionary version is no more exposed at the Python level and the field type now also has a size of 64-bit on 32-bit platforms. The PEP is part of a serie of 3 PEP