Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Greg Ewing canterbury.ac.nz> writes: > > This was a deliberate decision -- in fact I argued for it myself. > The buffer interface is meant to be a minimal-overhead way for > C code to get at the underlying data. Requiring allocation of > a PyObject would be too expensive. Tuples are used everywh

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: - it uses something (Py_buffer) which is not a PyObject and has totally different allocation/lifetime semantics This was a deliberate decision -- in fact I argued for it myself. The buffer interface is meant to be a minimal-overhead way for C code to get at the underlying

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan wrote: The multi-dimensional cases get pretty tricky though, since they will almost always end up dealing with non-contiguous data. The PEP 3118 protocol is up to handling the task, but the implementation of the index mapping to handle these multi-dimensional cases is highly non-tri

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Travis Oliphant wrote: When a slice view is made, a new memoryview object is created with a Py_buffer structure that needs to allocate it's own shape and strides (or something that will allow correct shape and strides to be reported to any consumer). In this way, there are two Py_buffer stru

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Greg Ewing wrote: Nick Coghlan wrote: Maintaining a PyDict instance to map from view pointers to shapes and strides info doesn't strike me as a "complex scheme" though. I don't see why a given buffer provider should ever need more than one set of shape/strides arrays at a time. It can allocate

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: Nick Coghlan gmail.com> writes: I don't see anything wrong with the PEP 3118 protocol. Apart from the fact that: - it uses something (Py_buffer) which is not a PyObject and has totally different allocation/lifetime semantics (which makes it non-trivial to adapt to for an

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan wrote: Antoine Pitrou wrote: In all honesty, I admit I am annoyed by all the problems with the buffer API / memoryview object, many of which are caused by its utterly bizarre design (and the fact that the design team went missing in action after imposing such a bizarre and complex d

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Greg Ewing wrote: Antoine Pitrou wrote: Why should it need two? Why couldn't the embedded Py_buffer fullfill all the needs of the memoryview object? Two things here: 1) The memoryview should *not* be holding onto a Py_buffer in between calls to its getitem and setitem methods. It

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: Nick Coghlan gmail.com> writes: For the slicing problem in particular, memoryview is currently trying to get away with only one Py_buffer object when it needs TWO. Why should it need two? Why couldn't the embedded Py_buffer fullfill all the needs of the memoryview object

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: Alexander Belopolsky gmail.com> writes: I did not follow numpy development for the last year or more, so I won't qualify as "the numpy folks," but my understanding is that numpy does exactly what Nick recommended: the viewed object owns shape and strides just as it owns th

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Alexander Belopolsky wrote: On Mon, Dec 8, 2008 at 6:25 PM, Antoine Pitrou <[EMAIL PROTECTED]> wrote: .. Alexander's suggestion of going and looking at what the numpy folks have done in this area is probably a good idea too. Well, I'm open to others doing this, but I won't do it myself. My inte

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: Hello, The Py_buffer struct has two pointers named `shape` and `strides`. Each points to an array of Py_ssize_t values whose length is equal to the number of dimensions of the buffer object. Unfortunately, the buffer protocol spec doesn't explain how allocation of these arr

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan gmail.com> writes: > > I don't see anything wrong with the PEP 3118 protocol. Apart from the fact that: - it uses something (Py_buffer) which is not a PyObject and has totally different allocation/lifetime semantics (which makes it non-trivial to adapt to for anyone used to the rest

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Greg Ewing wrote: > In any case, I think it should be possible to implement > either version without the memoryview having to own > more than one Py_buffer and one set of shape/strides > at a time. Slicing the memoryview creates another > memoryview with its own Py_buffer and shape/strides. The im

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > In all honesty, I admit I am annoyed by all the problems with the buffer API / > memoryview object, many of which are caused by its utterly bizarre design (and > the fact that the design team went missing in action after imposing such a > bizarre and complex design on us), a

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: If the memoryview wasn't holding onto a Py_buffer, one couldn't rely on its length or anything else because the underlying object could be mutated at any moment Hmm, it seems there are two different approaches that could be taken here to the design of a memoryview object.

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Greg Ewing canterbury.ac.nz> writes: > >1) The memoryview should *not* be holding onto a Py_buffer > in between calls to its getitem and setitem methods. It > should request one from the underlying object when needed > and release it again as soon as possible. If the memory

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: Why should it need two? Why couldn't the embedded Py_buffer fullfill all the needs of the memoryview object? Two things here: 1) The memoryview should *not* be holding onto a Py_buffer in between calls to its getitem and setitem methods. It should request on

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan gmail.com> writes: > > For the slicing problem in particular, memoryview is currently trying to > get away with only one Py_buffer object when it needs TWO. Why should it need two? Why couldn't the embedded Py_buffer fullfill all the needs of the memoryview object? If the memoryview

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan wrote: [from the PEP] "If the exporter wants to be able to change an object's shape, strides, and/or suboffsets before releasebuffer is called then it should allocate those arrays when getbuffer is called (pointing to them in the buffer-info structure provided) and free them when re

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: That doesn't work if e.g. you take a slice of a memoryview object, since the shape changes in the process. See http://bugs.python.org/issue4580 I haven't looked in detail at how memoryview is currently implemented, but it seems to me that the way it should work is that wh

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan wrote: Maintaining a PyDict instance to map from view pointers to shapes and strides info doesn't strike me as a "complex scheme" though. I don't see why a given buffer provider should ever need more than one set of shape/strides arrays at a time. It can allocate them on creation,

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > Le mardi 09 décembre 2008 à 22:33 +1000, Nick Coghlan a écrit : >> memoryview also currently gets the shape wrong on slices: > > I know, that's what I'm trying to fix... Yes, I was slightly misled by your use of slice assignment to demonstrate the problem. It also turns ou

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou pitrou.net> writes: > > > The first is that memoryview is treating the len field in the > > Py_buffer struct as the number of objects in the view in a few places > > instead of as the total number of bytes being exposed (it is actually > > the latter, as defined in PEP 3118). > >

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Le mardi 09 décembre 2008 à 22:33 +1000, Nick Coghlan a écrit : > I have zero problem whatsoever if slice assignment TO a memoryview > object is permitted only if the shape stays the same (i.e. I think that > issue should be closed as "not a bug"). I'm not even talking about slice /assignment/ her

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > Alexander Belopolsky gmail.com> writes: >> I did not follow numpy development for the last year or more, so I >> won't qualify as "the numpy folks," but my understanding is that numpy >> does exactly what Nick recommended: the viewed object owns shape and >> strides just as

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > Alexander Belopolsky gmail.com> writes: >> I did not follow numpy development for the last year or more, so I >> won't qualify as "the numpy folks," but my understanding is that numpy >> does exactly what Nick recommended: the viewed object owns shape and >> strides just as

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Alexander Belopolsky gmail.com> writes: > > I did not follow numpy development for the last year or more, so I > won't qualify as "the numpy folks," but my understanding is that numpy > does exactly what Nick recommended: the viewed object owns shape and > strides just as it owns the data. The v

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> No, you misunderstand what I meant. Py_buffer doesn't need to be changed >> at all. The *issuing type* would define a new structure with the >> additional fields, such as: > > With to the current buffer API, this is not possible. It's the

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: (of course complex schemes can be devised where the callee maintains its own separate storage for shape and strides, but I don't think we want to go there) But that's exactly where you're supposed to be going. If the object providing the buffer has variable-sized shape an

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

On Mon, Dec 8, 2008 at 6:25 PM, Antoine Pitrou <[EMAIL PROTECTED]> wrote: .. >> Alexander's suggestion of going and looking at what the numpy folks have >> done in this area is probably a good idea too. > > Well, I'm open to others doing this, but I won't do it myself. My interest is > in > fixing

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan gmail.com> writes: > > No, you misunderstand what I meant. Py_buffer doesn't need to be changed > at all. The *issuing type* would define a new structure with the > additional fields, such as: With to the current buffer API, this is not possible. It's the caller who allocates the Py

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> Actually, I think your suggested scheme for the one-dimensional case >> shows the way forward: ownership of the shape and strides memory belongs >> to the object issuing the Py_buffer struct, and that object needs to >> deal with it when t

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Nick Coghlan gmail.com> writes: > > Actually, I think your suggested scheme for the one-dimensional case > shows the way forward: ownership of the shape and strides memory belongs > to the object issuing the Py_buffer struct, and that object needs to > deal with it when the buffer is released. De

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

I don't have much to add to Nick's reply other than to point you to numpy, , as a reference implementation. You may also get better responses on the numpy list, < [EMAIL PROTECTED]>. On Mon, Dec 8, 2008 at 3:46 PM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > A

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

Antoine Pitrou wrote: > For the one-dimensional case, I had in mind a simple scheme where the > Py_buffer > struct has an additional two-member Py_ssize_t array. Then `shape` and > `strides` > can point to the first and second member of this array, respectively. This > wouldn't solve the multi-di

[Python-Dev] Allocation of shape and strides fields in Py_buffer

Hello, The Py_buffer struct has two pointers named `shape` and `strides`. Each points to an array of Py_ssize_t values whose length is equal to the number of dimensions of the buffer object. Unfortunately, the buffer protocol spec doesn't explain how allocation of these arrays should be handled.