[Numpy-discussion] Re: [EXTERNAL] Copy on __setitem__

[Mateusz Sokol]

It looks like general assignment function `array_assign_subscript` calls
`PyArray_CopyObject(view, op)` link

which in turn always calls `PyArray_DiscoverDTypeAndShape` with `copy=1`
and this is where `copy=True` takes place.

We could add a `copy` parameter to `PyArray_CopyObject` or set `copy=-1`
(to `None`) for the `PyArray_DiscoverDTypeAndShape` call.

Mateusz

On Thu, Dec 26, 2024 at 10:19 AM Sebastian Berg 
wrote:

> That seems like a bug but not sure why it would happen.  It needs to
> call `__array__`, but indeed of course not with `copy=True`.
>
> Would you open an issue on github?
>
> - Sebastian
>
>
> On Thu, 2024-12-26 at 03:46 +, Israel, Daniel M via NumPy-
> Discussion wrote:
> > Sure.  I didn’t originally, because I thought it would require an
> > entire custom array container, but the following trivial example
> > actually shows the behavior in question:
> >
> > import numpy
> >
> > class MyThing(object):
> > def __array__(self, dtype=None, copy=None):
> > print(f"MyThing.__array__(dtype={dtype}, copy={copy})")
> > return numpy.ones((5, 5))
> >
> > u = numpy.zeros((5, 5))
> > v = MyThing()
> >
> > u[...] = v
> >
> > If you run this code, as part of the final assignment statement, the
> > __array__ method is called for ‘v’ with copy=True.  Why?
> >
> > —
> > Daniel Israel
> > XCP-4: Continuum Models and Numerical Algorithms
> > d...@lanl.gov
> >
> > On Dec 25, 2024, at 3:23 PM, Steven Ellis
> >  wrote:
> >
> > Hi David,
> >
> > New to the listserv, but, maybe you can provide a reproducible
> > example?
> >
> > Steven
> >
> > On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion
> > mailto:numpy-discussion@python.org>>
> > wrote:
> > I was updating some code that uses a custom array container built
> > with the mixin library.  Specifically, I was trying to eliminate some
> > warnings due to the change to the __array__ interface to add a copy
> > argument.  In doing so, I discovered that, for two objects u, v in my
> > container class, the code:
> >
> > u[…] = v
> >
> > performs a copy on v.  Specifically, it calls __array__() with
> > copy=True.  This seems unnecessary and wasteful of memory.  Can
> > someone explain to me what is happening here?
> >
> > Thanks.
> >
> > —
> > Daniel Israel
> > XCP-4: Continuum Models and Numerical Algorithms
> > d...@lanl.gov
> >
> > ___
> > NumPy-Discussion mailing list --
> > numpy-discussion@python.org
> > To unsubscribe send an email to
> > numpy-discussion-le...@python.org > numpy-discussion-le...@python.org>
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/<
> >
> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-discussion.python.org/__;!!Bt8fGhp8LhKGRg!A_N5Cns4800odNjjStN3QAH94TwUNBEkvr7lqW4-PnnUiGS_kckVyXKfS4QTR4YADAXXGOTbKZ-6RDYTC7knRCMj$
> > >
> > Member address:
> > stevenalonzoel...@gmail.com
> >
> > ___
> > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > To unsubscribe send an email to numpy-discussion-le...@python.org
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > Member address: sebast...@sipsolutions.net
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: mso...@quansight.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Copy on __setitem__

[Steven Ellis via NumPy-Discussion]

Hi David,

New to the listserv, but, maybe you can provide a reproducible example?

Steven

On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> I was updating some code that uses a custom array container built with the
> mixin library.  Specifically, I was trying to eliminate some warnings due
> to the change to the __array__ interface to add a copy argument.  In doing
> so, I discovered that, for two objects u, v in my container class, the code:
>
> u[…] = v
>
> performs a copy on v.  Specifically, it calls __array__() with copy=True.
> This seems unnecessary and wasteful of memory.  Can someone explain to me
> what is happening here?
>
> Thanks.
>
> —
> Daniel Israel
> XCP-4: Continuum Models and Numerical Algorithms
> d...@lanl.gov
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: stevenalonzoel...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: [EXTERNAL] Copy on __setitem__

[Sebastian Berg]

That seems like a bug but not sure why it would happen.  It needs to
call `__array__`, but indeed of course not with `copy=True`.

Would you open an issue on github?

- Sebastian


On Thu, 2024-12-26 at 03:46 +, Israel, Daniel M via NumPy-
Discussion wrote:
> Sure.  I didn’t originally, because I thought it would require an
> entire custom array container, but the following trivial example
> actually shows the behavior in question:
> 
> import numpy
> 
> class MyThing(object):
>     def __array__(self, dtype=None, copy=None):
>     print(f"MyThing.__array__(dtype={dtype}, copy={copy})")
>     return numpy.ones((5, 5))
> 
> u = numpy.zeros((5, 5))
> v = MyThing()
> 
> u[...] = v
> 
> If you run this code, as part of the final assignment statement, the
> __array__ method is called for ‘v’ with copy=True.  Why?
> 
> —
> Daniel Israel
> XCP-4: Continuum Models and Numerical Algorithms
> d...@lanl.gov
> 
> On Dec 25, 2024, at 3:23 PM, Steven Ellis
>  wrote:
> 
> Hi David,
> 
> New to the listserv, but, maybe you can provide a reproducible
> example?
> 
> Steven
> 
> On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion
> mailto:numpy-discussion@python.org>>
> wrote:
> I was updating some code that uses a custom array container built
> with the mixin library.  Specifically, I was trying to eliminate some
> warnings due to the change to the __array__ interface to add a copy
> argument.  In doing so, I discovered that, for two objects u, v in my
> container class, the code:
> 
> u[…] = v
> 
> performs a copy on v.  Specifically, it calls __array__() with
> copy=True.  This seems unnecessary and wasteful of memory.  Can
> someone explain to me what is happening here?
> 
> Thanks.
> 
> —
> Daniel Israel
> XCP-4: Continuum Models and Numerical Algorithms
> d...@lanl.gov
> 
> ___
> NumPy-Discussion mailing list --
> numpy-discussion@python.org
> To unsubscribe send an email to
> numpy-discussion-le...@python.org numpy-discussion-le...@python.org>
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/<
> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-discussion.python.org/__;!!Bt8fGhp8LhKGRg!A_N5Cns4800odNjjStN3QAH94TwUNBEkvr7lqW4-PnnUiGS_kckVyXKfS4QTR4YADAXXGOTbKZ-6RDYTC7knRCMj$
> >
> Member address:
> stevenalonzoel...@gmail.com
> 
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: [EXTERNAL] Copy on __setitem__

[Robert Kern via NumPy-Discussion]

On Thu, Dec 26, 2024 at 5:48 PM Benjamin Root via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> Seems to make sense to me? Or is the following a bug?
>
> >>> import numpy as np
> >>> u = np.zeros(5)
> >>> v = np.ones(5)
> >>> u
> array([0., 0., 0., 0., 0.])
> >>> u[...] = v
> >>> u
> array([1., 1., 1., 1., 1.])
> >>> v[4] = 5
> >>> v
> array([1., 1., 1., 1., 5.])
> >>> u
> array([1., 1., 1., 1., 1.])
>
> If you don't do a copy, then it is a view, right? And so, should modifying
> v[4] change u[4]? Relatedly, if these were object arrays, of mutable
> objects, should mutating u[4] point to the exact same instance as v[4] and
> mutating it in one array means that the other array "sees" the same changes?
>

No one is suggesting that behavior should change. The issue is whether or
not an unnecessary, _additional_ copy gets made when converting a
not-exactly-an-`ndarray` object to an `ndarray` object before doing the
assignment (which always copies values over to the destination array).

-- 
Robert Kern
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: [EXTERNAL] Copy on __setitem__

[Benjamin Root via NumPy-Discussion]

Seems to make sense to me? Or is the following a bug?

>>> import numpy as np
>>> u = np.zeros(5)
>>> v = np.ones(5)
>>> u
array([0., 0., 0., 0., 0.])
>>> u[...] = v
>>> u
array([1., 1., 1., 1., 1.])
>>> v[4] = 5
>>> v
array([1., 1., 1., 1., 5.])
>>> u
array([1., 1., 1., 1., 1.])

If you don't do a copy, then it is a view, right? And so, should modifying
v[4] change u[4]? Relatedly, if these were object arrays, of mutable
objects, should mutating u[4] point to the exact same instance as v[4] and
mutating it in one array means that the other array "sees" the same changes?

Sorry if I'm missing the point, just want to make sure we don't
inadvertently change important implicit behavior.

Ben Root

On Thu, Dec 26, 2024 at 7:07 AM Mateusz Sokol  wrote:

> It looks like general assignment function `array_assign_subscript` calls
> `PyArray_CopyObject(view, op)` link
> 
> which in turn always calls `PyArray_DiscoverDTypeAndShape` with `copy=1`
> and this is where `copy=True` takes place.
>
> We could add a `copy` parameter to `PyArray_CopyObject` or set `copy=-1`
> (to `None`) for the `PyArray_DiscoverDTypeAndShape` call.
>
> Mateusz
>
> On Thu, Dec 26, 2024 at 10:19 AM Sebastian Berg <
> sebast...@sipsolutions.net> wrote:
>
>> That seems like a bug but not sure why it would happen.  It needs to
>> call `__array__`, but indeed of course not with `copy=True`.
>>
>> Would you open an issue on github?
>>
>> - Sebastian
>>
>>
>> On Thu, 2024-12-26 at 03:46 +, Israel, Daniel M via NumPy-
>> Discussion wrote:
>> > Sure.  I didn’t originally, because I thought it would require an
>> > entire custom array container, but the following trivial example
>> > actually shows the behavior in question:
>> >
>> > import numpy
>> >
>> > class MyThing(object):
>> > def __array__(self, dtype=None, copy=None):
>> > print(f"MyThing.__array__(dtype={dtype}, copy={copy})")
>> > return numpy.ones((5, 5))
>> >
>> > u = numpy.zeros((5, 5))
>> > v = MyThing()
>> >
>> > u[...] = v
>> >
>> > If you run this code, as part of the final assignment statement, the
>> > __array__ method is called for ‘v’ with copy=True.  Why?
>> >
>> > —
>> > Daniel Israel
>> > XCP-4: Continuum Models and Numerical Algorithms
>> > d...@lanl.gov
>> >
>> > On Dec 25, 2024, at 3:23 PM, Steven Ellis
>> >  wrote:
>> >
>> > Hi David,
>> >
>> > New to the listserv, but, maybe you can provide a reproducible
>> > example?
>> >
>> > Steven
>> >
>> > On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion
>> > mailto:numpy-discussion@python.org>>
>> > wrote:
>> > I was updating some code that uses a custom array container built
>> > with the mixin library.  Specifically, I was trying to eliminate some
>> > warnings due to the change to the __array__ interface to add a copy
>> > argument.  In doing so, I discovered that, for two objects u, v in my
>> > container class, the code:
>> >
>> > u[…] = v
>> >
>> > performs a copy on v.  Specifically, it calls __array__() with
>> > copy=True.  This seems unnecessary and wasteful of memory.  Can
>> > someone explain to me what is happening here?
>> >
>> > Thanks.
>> >
>> > —
>> > Daniel Israel
>> > XCP-4: Continuum Models and Numerical Algorithms
>> > d...@lanl.gov
>> >
>> > ___
>> > NumPy-Discussion mailing list --
>> > numpy-discussion@python.org
>> > To unsubscribe send an email to
>> > numpy-discussion-le...@python.org> > numpy-discussion-le...@python.org>
>> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/<
>> >
>> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-discussion.python.org/__;!!Bt8fGhp8LhKGRg!A_N5Cns4800odNjjStN3QAH94TwUNBEkvr7lqW4-PnnUiGS_kckVyXKfS4QTR4YADAXXGOTbKZ-6RDYTC7knRCMj$
>> > >
>> > Member address:
>> > stevenalonzoel...@gmail.com
>> >
>> > ___
>> > NumPy-Discussion mailing list -- numpy-discussion@python.org
>> > To unsubscribe send an email to numpy-discussion-le...@python.org
>> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> > Member address: sebast...@sipsolutions.net
>>
>> ___
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> Member address: mso...@quansight.com
>>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ben.v.r...@gmail.com
>
___
NumPy-D