[Numpy-discussion] preparing downstream code for NumPy 2.0 & API deprecations/removals

[Ralf Gommers]

Hi all,

This email is about two 2.0 release related topics:
1. Advice/guidance for downstream library authors and end users
2. Strategy for development work around public API changes that will be
breaking backwards compatibility.

I also just created https://github.com/numpy/numpy/issues/24300 as a
tracking issue that we can post announcements on and anyone can subscribe
to. I imagine many folks will want some way to follow along and be notified
of important changes around the release, but not subscribe to this mailing
list.

Some of the content of that issue overlaps with this email. In case of
questions/comments about the content of that issue, let's discuss it here.
Please keep that tracking issue for announcements, not for technical
discussion.


**Advice or downstream package authors and end users**

1. If you rely on the NumPy C API (e.g. via direct use in C/C++, or via
Cython code that uses NumPy), please add a `numpy<2.0` requirement in your
package's dependency metadata. Rationale: the NumPy C ABI will change in
2.0, so any compiled extension modules that rely on NumPy are likely to
break, they need to be recompiled.

2. If you rely on a large API surface from NumPy's Python API, also
consider adding the same ` numpy<2.0` requirement to your metadata.
Rationale: we will do a significant cleanup (see NEP 52), so unless you
only use modern/recommended functions and objects, your code is likely to
require at least some adjustments.

3. Consider cleaning up your code. E.g. remove `from numpy import *`, or
importing any private modules like `numpy.core`. See
https://github.com/numpy/numpy/blob/main/numpy/tests/test_public_api.py#L114-L126
for what we consider public/private. If it's not in the NumPy docs or in
the list of public modules there, don't use it!

4. Plan to do a release of your own packages which depend on `numpy`
shortly after the first NumPy 2.0 release candidate is released (probably
in Dec 2023). Rationale: at that point, you can release packages that will
work with both 2.0 and 1.X, and hence your own end users will not be seeing
much/any disruption (you want `pip install mypacackage` to continue working
on the day NumPy 2.0 is released).

5. Consider testing against NumPy nightlies in your own CI. We publish
those at https://anaconda.org/scientific-python-nightly-wheels/numpy, and
have documented that as a stable location at
https://numpy.org/devdocs/dev/depending_on_numpy.html. Rationale: this will
detect potential issues in your code so you can fix them well ahead of the
NumPy 2.0 release.



**Strategy for public API changes for 2.0**

Based on experience over the past weeks with adding deprecations and making
breaking changes, I think it'd be good to articulate a strategy for
Python/C API changes. We are not yet collectively used to the change of
pace that the run-up to a major release is. I think we want to use and
balance these two principles:

1. Make the API and behavior changes that we want to see for 2.0, in a way
that doesn't incur unreasonable amounts of effort or get completely blocked
by backwards compatibility constraints which we'd apply for a regular minor
release.
2. Mitigate the inevitable disturbances for downstream projects and end
users as best as we can.

To start with (2), issuing good guidance (like in the section above and the
tracking issue gh-24300) is one way to help. Another one is to, when you
are in the process of making a change, use code search tools and either
code that will break downstream proactively or at least notify downstream
project authors. The former can be done by sending PRs to at least the
largest projects (SciPy, Pandas, scikit-learn, scikit-image, Matplotlib)
when there are easy changes to make. And otherwise by filing issues on the
issue tracker of other projects. Yet another way, in case of mechanical
changes like removal of aliases, would be to provide a sed script that
others can run to automatically update their code as much as possible.

For (1), I think it's important to understand that 2.0 is a one-off change
of pace where our regular backwards compatibility policy does not apply. If
particular functions are desirable to touch but widely used, it may be wise
to leave them in or deprecate them of course - this is a case by case
decision. However, it doesn't have to be done like that. Every single
object in the NumPy API is used somewhere, and removing it is going to
affect some users/packages. This is inevitable, and we can't achieve our
2.0 goals if every little niche API change is going to require following
our regular backwards compatibility strategy. The only thing that would do
is to make 2.2 the breaking release.

Many downstream libraries have CI setups that turn deprecation warnings
into hard errors, hence often it doesn't matter whether we deprecate
something in `main`, or remove it straight away. Apply good judgement here
I'd say (how widely is something used, is the replacement trivial or does
it require some thought

[Numpy-discussion] NumPy 1.25.2 released

[Charles R Harris]

Hi All,

On behalf of the NumPy team, I'm pleased to announce the release of NumPy
1.25.2. NumPy 1.25.2 is a maintenance release that fixes bugs and
regressions discovered after the 1.25.1 release. This is the last planned
release in the 1.25.x series, the next final release will be 1.26.0, which
will use the meson build system and support Python 3.12.

The Python versions supported by this release are 3.9-3.11 Note that 32 bit
wheels are only provided for Windows, all other wheels are 64 bits on
account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit
support. Wheels can be downloaded from PyPI
; source archives, release notes,
and wheel hashes are available on Github
.


*Contributors*

A total of 13 people contributed to this release.  People with a "+" by
their names contributed a patch for the first time.

   - Aaron Meurer
   - Andrew Nelson
   - Charles Harris
   - Kevin Sheppard
   - Matti Picus
   - Nathan Goldbaum
   - Peter Hawkins
   - Ralf Gommers
   - Randy Eckenrode +
   - Sam James +
   - Sebastian Berg
   - Tyler Reddy
   - dependabot[bot]



*Pull requests merged*
A total of 19 pull requests were merged for this release.

   - #24148: MAINT: prepare 1.25.x for further development
   - #24174: ENH: Improve clang-cl compliance
   - #24179: MAINT: Upgrade various build dependencies.
   - #24182: BLD: use ``-ftrapping-math`` with Clang on macOS
   - #24183: BUG: properly handle negative indexes in ufunc_at fast path
   - #24184: BUG: PyObject_IsTrue and PyObject_Not error handling in
   setflags
   - #24185: BUG: histogram small range robust
   - #24186: MAINT: Update meson.build files from main branch
   - #24234: MAINT: exclude min, max and round from ``np.__all__``
   - #24241: MAINT: Dependabot updates
   - #24242: BUG: Fix the signature for np.array_api.take
   - #24243: BLD: update OpenBLAS to an intermediate commit
   - #24244: BUG: Fix reference count leak in str(scalar).
   - #24245: BUG: fix invalid function pointer conversion error
   - #24255: BUG: Factor out slow ``getenv`` call used for memory policy
   warning
   - #24292: CI: correct URL in cirrus.star [skip cirrus]
   - #24293: BUG: Fix C types in scalartypes
   - #24294: BUG: do not modify the input to ufunc_at
   - #24295: BUG: Further fixes to indexing loop and added tests


Cheers,

Charles Harris
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] The 1.26.x maintenance branch has been created.

[Charles R Harris]

Hi All,

The 1.26.x maintenance branch has been created. The 1.26.x branch is a
continuation of the 1.25.x branch and serves to mark the change from our
distutils based builds to the meson builds needed to support the upcoming
Python 3.12 release.

Chuck
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: The 1.26.x maintenance branch has been created.

[Ralf Gommers]

On Mon, Jul 31, 2023 at 6:53 PM Charles R Harris 
wrote:

> Hi All,
>
> The 1.26.x maintenance branch has been created. The 1.26.x branch is a
> continuation of the 1.25.x branch and serves to mark the change from our
> distutils based builds to the meson builds needed to support the upcoming
> Python 3.12 release.
>

Awesome, thanks Chuck! Just checking: are you preparing one or a couple of
backport PRs for the merged PRs with the Backport-Candidate label? If you
want me to do the build system related ones, please let me know.

We're pretty close to the finish line - most of the TODO items at
https://github.com/numpy/numpy/issues/23981 were done. The main one left is
SIMD support, and Sayed got quite far already in making that work - I'm
prioritizing reviewing it now. We should be able to do a beta release
without it though, as soon as Python 3.12.0rc1 shows up (possibly it will
appear later today).

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: The 1.26.x maintenance branch has been created.

[Charles R Harris]

On Mon, Jul 31, 2023 at 12:16 PM Ralf Gommers 
wrote:

>
>
> On Mon, Jul 31, 2023 at 6:53 PM Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> The 1.26.x maintenance branch has been created. The 1.26.x branch is a
>> continuation of the 1.25.x branch and serves to mark the change from our
>> distutils based builds to the meson builds needed to support the upcoming
>> Python 3.12 release.
>>
>
> Awesome, thanks Chuck! Just checking: are you preparing one or a couple of
> backport PRs for the merged PRs with the Backport-Candidate label? If you
> want me to do the build system related ones, please let me know.
>
> We're pretty close to the finish line - most of the TODO items at
> https://github.com/numpy/numpy/issues/23981 were done. The main one left
> is SIMD support, and Sayed got quite far already in making that work - I'm
> prioritizing reviewing it now. We should be able to do a beta release
> without it though, as soon as Python 3.12.0rc1 shows up (possibly it will
> appear later today).
>
>
Hi Ralf,

There are a bunch of backport candidates with a 1.26.0 milestone, so I'll
put those in first, then we can get started on the CI updates and the meson
builds. I'd appreciate any help you can offer with those. I'll try to get
started with the backports tonight, but there will likely be some problems
along the way.

Chuck

> Cheers,
> Ralf
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: charlesr.har...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: The 1.26.x maintenance branch has been created.

[Charles R Harris]

On Mon, Jul 31, 2023 at 1:08 PM Charles R Harris 
wrote:

>
>
> On Mon, Jul 31, 2023 at 12:16 PM Ralf Gommers 
> wrote:
>
>>
>>
>> On Mon, Jul 31, 2023 at 6:53 PM Charles R Harris <
>> charlesr.har...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> The 1.26.x maintenance branch has been created. The 1.26.x branch is a
>>> continuation of the 1.25.x branch and serves to mark the change from our
>>> distutils based builds to the meson builds needed to support the upcoming
>>> Python 3.12 release.
>>>
>>
>> Awesome, thanks Chuck! Just checking: are you preparing one or a couple
>> of backport PRs for the merged PRs with the Backport-Candidate label? If
>> you want me to do the build system related ones, please let me know.
>>
>> We're pretty close to the finish line - most of the TODO items at
>> https://github.com/numpy/numpy/issues/23981 were done. The main one left
>> is SIMD support, and Sayed got quite far already in making that work - I'm
>> prioritizing reviewing it now. We should be able to do a beta release
>> without it though, as soon as Python 3.12.0rc1 shows up (possibly it will
>> appear later today).
>>
>>
> Hi Ralf,
>
> There are a bunch of backport candidates with a 1.26.0 milestone, so I'll
> put those in first, then we can get started on the CI updates and the meson
> builds. I'd appreciate any help you can offer with those. I'll try to get
> started with the backports tonight, but there will likely be some problems
> along the way.
>
> Chuck
>


 *as soon as Python 3.12.0rc1 shows up*

It will be a few days before cibuildwheel catches up with the rc1 release,
I expect we can get a beta (or rc1) out in a week or two, but not much
sooner.

Chuck
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: The 1.26.x maintenance branch has been created.

[Charles R Harris]

On Mon, Jul 31, 2023 at 1:13 PM Charles R Harris 
wrote:

>
>
> On Mon, Jul 31, 2023 at 1:08 PM Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>>
>>
>> On Mon, Jul 31, 2023 at 12:16 PM Ralf Gommers 
>> wrote:
>>
>>>
>>>
>>> On Mon, Jul 31, 2023 at 6:53 PM Charles R Harris <
>>> charlesr.har...@gmail.com> wrote:
>>>
 Hi All,

 The 1.26.x maintenance branch has been created. The 1.26.x branch is a
 continuation of the 1.25.x branch and serves to mark the change from our
 distutils based builds to the meson builds needed to support the upcoming
 Python 3.12 release.

>>>
>>> Awesome, thanks Chuck! Just checking: are you preparing one or a couple
>>> of backport PRs for the merged PRs with the Backport-Candidate label? If
>>> you want me to do the build system related ones, please let me know.
>>>
>>> We're pretty close to the finish line - most of the TODO items at
>>> https://github.com/numpy/numpy/issues/23981 were done. The main one
>>> left is SIMD support, and Sayed got quite far already in making that work -
>>> I'm prioritizing reviewing it now. We should be able to do a beta release
>>> without it though, as soon as Python 3.12.0rc1 shows up (possibly it will
>>> appear later today).
>>>
>>>
>> Hi Ralf,
>>
>> There are a bunch of backport candidates with a 1.26.0 milestone, so I'll
>> put those in first, then we can get started on the CI updates and the meson
>> builds. I'd appreciate any help you can offer with those. I'll try to get
>> started with the backports tonight, but there will likely be some problems
>> along the way.
>>
>> Chuck
>>
>
>
>  *as soon as Python 3.12.0rc1 shows up*
>
> It will be a few days before cibuildwheel catches up with the rc1 release,
> I expect we can get a beta (or rc1) out in a week or two, but not much
> sooner.
>
>
See https://github.com/numpy/numpy/pull/24308.

Chuck
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com