Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-07-18 Thread Matti Picus
The NEP [0] and the corresponding PR [1] have gone through another round of editing. I would like to restart the discussion here if anyone has more to add. Things that have changed since the last round: - The functions now accept a context argument - The code has been cleaned up for consistenc

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-15 Thread Eric Wieser
Note that PEP-445 which introduced `PyMemAllocatorEx` specifically rejected omitting the `ctx` argument here: https://www.python.org/dev/peps/pep-0445/#id23, which is another argument in favor of having it. I'll try to give a more thorough justification for the pyobject / capsule suggestion in ano

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-13 Thread eliaskoromilas
Eric Wieser wrote >> Yes, sorry, had been a while since I had looked it up: >> >> https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx > > That `PyMemAllocatorEx` looks almost exactly like one of the two variants > I > was proposing. Is there a reason for wanting to define our own struct

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-12 Thread leofang
Eric Wieser wrote >> Yes, sorry, had been a while since I had looked it up: >> >> https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx > > That `PyMemAllocatorEx` looks almost exactly like one of the two variants > I was proposing. Is there a reason for wanting to define our own structur

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-11 Thread Sebastian Berg
On Tue, 2021-05-11 at 09:54 +0100, Eric Wieser wrote: > > Yes, sorry, had been a while since I had looked it up: > > > > https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx > > That `PyMemAllocatorEx` looks almost exactly like one of the two > variants I > was proposing. Is there a rea

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-11 Thread Eric Wieser
> Yes, sorry, had been a while since I had looked it up: > > https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx That `PyMemAllocatorEx` looks almost exactly like one of the two variants I was proposing. Is there a reason for wanting to define our own structure vs just using that one? I

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-10 Thread Matti Picus
On 10/5/21 8:43 pm, Sebastian Berg wrote: But right now the proposal says this is static, and I honestly don't see much reason for it to be freeable? I think this is the crux of the issue. The current design is for a singly-allocated struct to be passed around since it is just an aggregate

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-10 Thread Sebastian Berg
On Mon, 2021-05-10 at 10:01 +0100, Eric Wieser wrote: > > The Python version of this does have a `void *ctx`, but I am not > > sure if > the use for this is actually valuable for the NumPy use-cases. > > Do you mean "the CPython version"? If so, can you link a reference? Yes, sorry, had been a wh

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-10 Thread Eric Wieser
> The Python version of this does have a `void *ctx`, but I am not sure if the use for this is actually valuable for the NumPy use-cases. Do you mean "the CPython version"? If so, can you link a reference? > While I like the `PyObject *` idea, I am also not sure that it helps much. If we want a

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-09 Thread Sebastian Berg
On Thu, 2021-05-06 at 13:06 +0100, Eric Wieser wrote: > Another argument for supporting stateful allocators would be > compatibility > with the stateful C++11 allocator API, such as > https://en.cppreference.com/w/cpp/memory/allocator_traits/allocate. The Python version of this does have a `void

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Eric Wieser
Another argument for supporting stateful allocators would be compatibility with the stateful C++11 allocator API, such as https://en.cppreference.com/w/cpp/memory/allocator_traits/allocate. Adding support for stateful allocators at a later date would almost certainly create an ABI breakage or lots

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Matti Picus
On 6/5/21 2:07 pm, Eric Wieser wrote: The NEP looks good, but I worry the API isn't flexible enough. My two main concerns are: ### Stateful allocators Consider an allocator that aligns to `N` bytes, where `N` is configurable from a python call in someone else's extension module. ... ### Th

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Eric Wieser
The NEP looks good, but I worry the API isn't flexible enough. My two main concerns are: ### Stateful allocators Consider an allocator that aligns to `N` bytes, where `N` is configurable from a python call in someone else's extension module. Where do they store `N`? They can hide it in `PyDataMe

[Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Matti Picus
Here is the current rendering of the NEP:https://numpy.org/neps/nep-0049.html The mailing list discussion, started on April 20 did not bring up any objections to the proposal, nor were there objections in the discussion around the text of the NEP. There were questions around details of the