Re: [Python-Dev] Compact ordered set

2019-03-07 Thread Inada Naoki
ordered set 2 Hi, Raymond. Thank you for your detailed response and I'm sorry about the late reply. All of your points make sense to me. My implementation has not been battle-tested yet. As I wrote in a previous mail, there is only one benchmark in pyperformance was affected significantly. (M

Re: [Python-Dev] Compact ordered set

2019-02-28 Thread Greg Ewing
Antoine Pitrou wrote: On a more abstract level, set and dict are both content-addressed collections parametered on hash and equality functions. Indeed. It's been said that a set is like "half a dict", and this is why sets were implemented using dicts in the old days. It's kind of an obvious th

Re: [Python-Dev] Compact ordered set

2019-02-28 Thread Chris Angelico
On Thu, Feb 28, 2019 at 10:58 PM Antoine Pitrou wrote: > Some of them may be coming from C++, where the respective > characteristics of set and map (or unordered_set and > unordered_multimap) are closely related. I'm sure other languages > show similar analogies. > > On a more abstract level, set

Re: [Python-Dev] Compact ordered set

2019-02-28 Thread Xavier Morel
> On 2019-02-28, at 12:56 , Antoine Pitrou wrote: > > On Thu, 28 Feb 2019 22:43:04 +1100 > Steven D'Aprano wrote: >> On Wed, Feb 27, 2019 at 02:15:53PM -0800, Barry Warsaw wrote: >> >>> I’m just relaying a data point. Some Python folks I’ve worked with do >>> make the connection between dict

Re: [Python-Dev] Compact ordered set

2019-02-28 Thread Antoine Pitrou
On Thu, 28 Feb 2019 22:43:04 +1100 Steven D'Aprano wrote: > On Wed, Feb 27, 2019 at 02:15:53PM -0800, Barry Warsaw wrote: > > > I’m just relaying a data point. Some Python folks I’ve worked with do > > make the connection between dicts and sets, and have questions about > > the ordering guaran

Re: [Python-Dev] Compact ordered set

2019-02-28 Thread Steven D'Aprano
On Wed, Feb 27, 2019 at 02:15:53PM -0800, Barry Warsaw wrote: > I’m just relaying a data point. Some Python folks I’ve worked with do > make the connection between dicts and sets, and have questions about > the ordering guarantees of then (and how they relate). Sets and dicts are not related b

Re: [Python-Dev] Compact ordered set

2019-02-28 Thread INADA Naoki
On Thu, Feb 28, 2019 at 7:23 AM Henry Chen wrote: > If sets were ordered, then what ought pop() return - first, last, or > nevertheless an arbitrary element? I lean toward arbitrary because in > existing code, set.pop often implies that which particular element is > immaterial. > > dict.popitem()

Re: [Python-Dev] Compact ordered set

2019-02-27 Thread Henry Chen
If sets were ordered, then what ought pop() return - first, last, or nevertheless an arbitrary element? I lean toward arbitrary because in existing code, set.pop often implies that which particular element is immaterial. On Wed, Feb 27, 2019 at 2:18 PM Barry Warsaw wrote: > On Feb 27, 2019, at 1

Re: [Python-Dev] Compact ordered set

2019-02-27 Thread Barry Warsaw
On Feb 27, 2019, at 13:54, Chris Barker via Python-Dev wrote: > > A mapping and a set type really don't have much to do with each other other > than implementation -- anyone that isn't familiar with python C code, or hash > tables in general, wouldn't likely have any expectation of them having

Re: [Python-Dev] Compact ordered set

2019-02-27 Thread Chris Barker via Python-Dev
On Tue, Feb 26, 2019 at 3:43 PM Barry Warsaw wrote: > The behavior differences between dicts and sets is already surprising to > many users, so we should be careful not to make the situation worse. > It's a nice to have, but other than the fact that we all used to use a dict when we really want

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Barry Warsaw
On Feb 26, 2019, at 13:02, Raymond Hettinger wrote: > * I gave up on ordering right away. If we care about performance, keys can > be stored in the order added; but no effort should be expended to maintain > order if subsequent deletions occur. Likewise, to keep set-to-set operations > effi

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 17:33, INADA Naoki a écrit : > My company gives me dedicated Linux machine with Core(TM) i7-6700. > So I think it's not issue of my machine. Oh great :-) > perf shows this line caused many page fault. > https://github.com/python/cpython/blob/c606a9cbd48f69d3f4a09204c781dd

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Raymond Hettinger
Quick summary of what I found when I last ran experiments with this idea: * To get the same lookup performance, the density of index table would need to go down to around 25%. Otherwise, there's no way to make up for the extra indirection and the loss of cache locality. * There was a small win

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Raymond Hettinger
> On Feb 26, 2019, at 3:30 AM, INADA Naoki wrote: > > I'm working on compact and ordered set implementation. > It has internal data structure similar to new dict from Python 3.6. > > On Feb 26, 2019, at 3:30 AM, INADA Naoki wrote: > > I'm working on compact and ordered set implementation.

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread INADA Naoki
On Wed, Feb 27, 2019 at 12:37 AM Victor Stinner wrote: > > Le mar. 26 févr. 2019 à 12:33, INADA Naoki a écrit : > > - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower > > (+52%)> ... > > ... > > unpickle and unpickle_list shows massive slowdown. I suspect this slowdown > > i

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 12:33, INADA Naoki a écrit : > - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower > (+52%)> ... > ... > unpickle and unpickle_list shows massive slowdown. I suspect this slowdown > is not caused from set change. Linux perf shows many pagefault is happ

[Python-Dev] Compact ordered set

2019-02-26 Thread INADA Naoki
Hello, folks. I'm working on compact and ordered set implementation. It has internal data structure similar to new dict from Python 3.6. It is still work in progress. Comments, tests, and documents should be updated. But it passes existing tests excluding test_sys and test_gdb (both tests check