[Numpy-discussion] [NEP 52] Introduction of `array_utils` public namespace in `np.lib` submodule

[Mateusz Sokol]

Hi all!

The two most important goals of NEP 52 are cleanups of the top `np` namespace 
and `np.lib` namespace.

A set of functions from `np` and `np.lib.*` submodules has been identified that 
could establish a new public namespace within the `np.lib` submodule.

The issue https://github.com/numpy/numpy/issues/24166 proposes a 
`np.lib.array_utils` name for the new namespace that could host functions such 
as `byte_bounds` (originally in `np`), `normalize_axis_tuple` (originally in 
`np.lib.stride_tricks`) or `normalize_axis_index` (originally in 
`np.lib.shape_base`).

If you have any thoughts on naming/content of the new namespace, please share 
your feedback!

Best regards,
Mateusz
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Change in numpy.percentile

[Mateusz Sokol]

Just to mention for visibility: Introducing a "nan" option and deprecating
nan* functions was considered for 2.0 main namespace refactor but it
was deemed large enough to be (hopefully) tackled in a separate
story/project.

https://github.com/numpy/numpy/issues/24306#issuecomment-1660073584 (first
bullet point on the list)

On Mon, Oct 9, 2023 at 12:48 PM Andrew Nelson  wrote:

> On Mon, 9 Oct 2023 at 20:34,  wrote:
>
>> Surely you can do this for all functions of eg.nan*. Why separate them is
>> the only thing that distinguishes them.  Setting the parameter seems to be
>> more handy and user-friendly.  Well for me it's seems better to do it right
>> away in NumPy 2.0
>>
>
> I think I prefer the clearer intent of having nan* functions.
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: mso...@quansight.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Adding NumpyUnpickler to Numpy 1.26 and future Numpy 2.0

[Mateusz Sokol]

Hi! Thank you for all your feedback this week!

We have made a decision to take a less disruptive option that we considered
and that came up in this discussion.

We back out of the `NumpyUnpickler` class solution for reading pickles
across major NumPy versions.

Instead, we will retain `numpy.core` stubs in NumPy 2.0 to allow loading
NumPy 1.x pickles.
Additionally, `numpy._core` stubs will be backported to 1.26 to ensure
compatibility the other way around - loading NumPy 2.0 pickles with NumPy
1.26 installed.

Both major versions will continue to create pickles with their own contents
(NumPy 1.26 with `numpy.core` paths and NumPy 2.0 with `numpy._core` paths).

This way any pickle will be loadable by both major versions.


On Tue, Oct 10, 2023 at 3:33 PM Nathan  wrote:

>
>
> On Tue, Oct 10, 2023 at 7:03 AM Ronald van Elburg <
> r.a.j.van.elb...@hetnet.nl> wrote:
>
>> I have one more useCase to consider from our ecosystem.
>>
>> We dump numpy arrays into a MongoDB using GridFS for subsequent
>> visualization, some snippets:
>>
>> '''Python
>> with BytesIO() as BIO:
>> np.save(BIO, numpy_array)
>> serialized_A = BIO.getvalue()
>> filehandle_id = self.representations_files.put(serialized_A)
>> '''
>>
>> and then restore them in the other application:
>>
>> '''Python
>> numpy_array = np.load(BytesIO(serializedA))
>> '''
>> For us this is for development work only and I am less concerned about
>> having mixed versions in my database, but in principle that is a scenario.
>> But it seems to me that for this to work the reading application needs to
>> be migrated to version 2 and temporarily extended with the NumpyUnpickler
>> before the writing application is migrated. Or they need to be migrated at
>> the same time. Is that correct?
>
>
> np.save and np.load will use NumpyUnpickler under the hood so you won’t
> have any issues, you would only have issues if you saved or loaded pickles
> using the pickle module directly.
>
>
>
>> ___
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> Member address: nathan12...@gmail.com
>>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: mso...@quansight.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] New Ruff rule for migrating to NumPy 2.0

[Mateusz Sokol]

Hi all!

Some time ago we added a new rule to Ruff linter, "NPY201", which updates the 
codebase to a NumPy 2.0 compatible version.

You can read about it in the migration guide: 
https://numpy.org/devdocs/numpy_2_0_migration_guide.html#ruff-plugin
And on the Ruff docs website: 
https://docs.astral.sh/ruff/rules/numpy2-deprecation/ 
(it's still in a "preview" mode but available since 0.1.4 release).

Best regards,
Mateusz
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: [EXTERNAL] Copy on __setitem__

[Mateusz Sokol]

It looks like general assignment function `array_assign_subscript` calls
`PyArray_CopyObject(view, op)` link

which in turn always calls `PyArray_DiscoverDTypeAndShape` with `copy=1`
and this is where `copy=True` takes place.

We could add a `copy` parameter to `PyArray_CopyObject` or set `copy=-1`
(to `None`) for the `PyArray_DiscoverDTypeAndShape` call.

Mateusz

On Thu, Dec 26, 2024 at 10:19 AM Sebastian Berg 
wrote:

> That seems like a bug but not sure why it would happen.  It needs to
> call `__array__`, but indeed of course not with `copy=True`.
>
> Would you open an issue on github?
>
> - Sebastian
>
>
> On Thu, 2024-12-26 at 03:46 +, Israel, Daniel M via NumPy-
> Discussion wrote:
> > Sure.  I didn’t originally, because I thought it would require an
> > entire custom array container, but the following trivial example
> > actually shows the behavior in question:
> >
> > import numpy
> >
> > class MyThing(object):
> > def __array__(self, dtype=None, copy=None):
> > print(f"MyThing.__array__(dtype={dtype}, copy={copy})")
> > return numpy.ones((5, 5))
> >
> > u = numpy.zeros((5, 5))
> > v = MyThing()
> >
> > u[...] = v
> >
> > If you run this code, as part of the final assignment statement, the
> > __array__ method is called for ‘v’ with copy=True.  Why?
> >
> > —
> > Daniel Israel
> > XCP-4: Continuum Models and Numerical Algorithms
> > d...@lanl.gov
> >
> > On Dec 25, 2024, at 3:23 PM, Steven Ellis
> >  wrote:
> >
> > Hi David,
> >
> > New to the listserv, but, maybe you can provide a reproducible
> > example?
> >
> > Steven
> >
> > On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion
> > mailto:numpy-discussion@python.org>>
> > wrote:
> > I was updating some code that uses a custom array container built
> > with the mixin library.  Specifically, I was trying to eliminate some
> > warnings due to the change to the __array__ interface to add a copy
> > argument.  In doing so, I discovered that, for two objects u, v in my
> > container class, the code:
> >
> > u[…] = v
> >
> > performs a copy on v.  Specifically, it calls __array__() with
> > copy=True.  This seems unnecessary and wasteful of memory.  Can
> > someone explain to me what is happening here?
> >
> > Thanks.
> >
> > —
> > Daniel Israel
> > XCP-4: Continuum Models and Numerical Algorithms
> > d...@lanl.gov
> >
> > ___
> > NumPy-Discussion mailing list --
> > numpy-discussion@python.org
> > To unsubscribe send an email to
> > numpy-discussion-le...@python.org > numpy-discussion-le...@python.org>
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/<
> >
> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-discussion.python.org/__;!!Bt8fGhp8LhKGRg!A_N5Cns4800odNjjStN3QAH94TwUNBEkvr7lqW4-PnnUiGS_kckVyXKfS4QTR4YADAXXGOTbKZ-6RDYTC7knRCMj$
> > >
> > Member address:
> > stevenalonzoel...@gmail.com
> >
> > ___
> > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > To unsubscribe send an email to numpy-discussion-le...@python.org
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > Member address: sebast...@sipsolutions.net
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: mso...@quansight.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] StringDType scalar type

[Mateusz Sokol]

Hi all!

There's an ongoing discussion about introducing a dedicated scalar type for 
`StringDType`.
One of the remarks was that the scope of the change is large enough so it 
deserves a
separate paragraph in the NEP 55.
It's also vital to collect more opinions from the community before the actual 
work, therefore
here's a list of active discussion threads:

- An official issue for StringDType scalar: 
https://github.com/numpy/numpy/issues/28165
- PR which updates NEP 55 to describe StringDType scalar: 
https://github.com/numpy/numpy/pull/28842
- Draft PR meant to ship the feature (goes after NEP 55 update): 
https://github.com/numpy/numpy/pull/28196

Please share your thoughts! 

Best regards,
Mateusz
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Making `T` property Array API compatible

[Mateusz Sokol]

Hi all!

The Array API standard states that `T` property should only be applied to 
2-dimensional arrays, in all other cases it should raise an error:
https://data-apis.org/array-api/latest/API_specification/generated/array_api.array.T

To ensure that NumPy also follows this rule, I opened a PR that raises a 
warning for `arr.T` for non-2-dimensional arrays and scalars: 
https://github.com/numpy/numpy/pull/28678

For non-2-dimensional arrays, the replacement for `arr.T` can be either: Array 
API compatible, namely `np.permute_dims(arr, range(arr.ndim)[::-1])`, or 
shorter, NumPy specific: `arr.transpose()`.

The new discrepancy between `arr.T` and `arr.transpose()` is justified, as `T` 
is defined by the Array API, where `transpose` isn't and should retain the 
existing behavior.

Please share your thoughts! 

Best regards,
Mateusz
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com