[Numpy-discussion] Importance of setting up SVML during build

[Sandro Tosi]

Hello,
i'm preparing 1.22.1 for inclusion in Debian, and noticed how the new
numpy has a git submodule to include numpy/SVML in its source tree.

We are basing the debian package on the github tarball, and that does
not include any submodule, so i'm wondering how important it is to
have the SVML files around when building the packages.

Given it's not immediately possible to include an external set of
files, i'm just trying to balance the effort to inject them and their
usefulness.

At the same time, i'm wondering if
https://github.com/numpy/numpy/pull/20695 (the PR that introduced
NPY_DISABLE_SVML, also probably
https://github.com/numpy/numpy/pull/20741 for the doc) should be
backported to the 1.22.x branch, so that we can start using it in the
release that introduced that feature.

Thanks,
-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Importance of setting up SVML during build

[Ralf Gommers]

On Sun, Jan 16, 2022 at 9:16 AM Sandro Tosi  wrote:

> Hello,
> i'm preparing 1.22.1 for inclusion in Debian, and noticed how the new
> numpy has a git submodule to include numpy/SVML in its source tree.
>
> We are basing the debian package on the github tarball, and that does
> not include any submodule, so i'm wondering how important it is to
> have the SVML files around when building the packages.
>

Not using SVML means you're throwing away performance for your users, and
using a non untested config on 64-bit Linux. The latter isn't going to be a
problem in practice, since SVML was recently introduced. But I'd prefer for
Debian to include it.

You are using the wrong tarball I believe. I just checked
numpy-1.22.1.tar.gz

from our GitHub Releases page, and it does include SVML (at
numpy/core/src/umath/svml). If you instead grabbed the
autogenerated-by-github "Source code" tarball, that is the wrong one. We'd
delete it if we could, but unfortunately that is not possible.



> Given it's not immediately possible to include an external set of
> files, i'm just trying to balance the effort to inject them and their
> usefulness.
>
> At the same time, i'm wondering if
> https://github.com/numpy/numpy/pull/20695 (the PR that introduced
> NPY_DISABLE_SVML, also probably
> https://github.com/numpy/numpy/pull/20741 for the doc) should be
> backported to the 1.22.x branch, so that we can start using it in the
> release that introduced that feature.
>

I followed up on this on the PR. Thanks for pointing out this issue.

Cheers,
Ralf


>
> Thanks,
> --
> Sandro "morph" Tosi
> My website: http://sandrotosi.me/
> Me at Debian: http://wiki.debian.org/SandroTosi
> Twitter: https://twitter.com/sandrotosi
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ralf.gomm...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] deprecating numpy.distutils

[Ralf Gommers]

Hi all,

Now that 1.22.0 is out the door, I think it's time to deprecate
`numpy.distutils` completely.

A brief summary for context:
- `distutils` was deprecated in Python 3.10, and will be removed in Python
3.12 (Oct 2023 release date)
- `setuptools` has re-enabled its vendored copy of `distutils` in its 60.0
release. It's incompatible with `numpy.distutils` and many more breaking
changes are to be expected.
- Rebasing `numpy.distutils` on top of `setuptools` will be a moving
target. More importantly, `numpy.distutils` still has serious and hard to
fix bugs and design issues. For example, we cannot build SciPy in parallel
due to race conditions in the Fortran support code, and there's a very
elusive bug where building an extension module which links in a static
Fortran-only library is underlinked and breaks at runtime in dev
environments.

My initial plan was to contribute Fortran support and other
`numpy.distutils` features to `setuptools`, but I've personally changed my
mind on that and don't plan to do that work. I think `setuptools` is moving
too slow for this to be a productive exercise, and the design of distutils
is fundamentally unfixable. So it's a much better investment of time to
help projects migrate away from `numpy.distutils` (and hence from
`setuptools`).

The tracking issue for this is https://github.com/numpy/numpy/issues/18588.
Please see the comments and linked issue/thread for more details.

My proposal is:
1. deprecate `numpy.distutils` now in main.
2. keep it around for another 2 years after the 1.23.0 release. On >=3.12
it will raise an informative error when someone tries to import it.
3. Only setuptools < 60.0 is supported. This is the status as of today -
unless there's an issue with Python 3.11 support, I don't see a point in
spending time on keeping up with new setuptools releases, a pin is just
fine.
4. Write a docs page with migration info on how to deal with the
deprecation, and update it before the 1.23.0 release.

Note that the setuptools maintainers have said that they are happy to
consider new features if people want to integrate them. The most
popular/useful features for `numpy.distutils` are (1) Fortran support, and
(2) nested setup.py support. The former is probably not a good idea to
integrate, the latter may be. There aren't many packages who need
BLAS/LAPACK support anymore, scikit-learn/statsmodels/etc. switched to the
Cython/C API exposed by SciPy, rather than linking to BLAS/LAPACK directly.

For NumPy itself, we should move to Meson after the SciPy 1.9.0 release has
gone out and ironed out all the remaining issues there. We can revisit that
in 6 months or so.

Thoughts?

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Importance of setting up SVML during build

[Sandro Tosi]

> You are using the wrong tarball I believe. I just checked numpy-1.22.1.tar.gz 
> from our GitHub Releases page, and it does include SVML (at 
> numpy/core/src/umath/svml). If you instead grabbed the 
> autogenerated-by-github "Source code" tarball, that is the wrong one. We'd 
> delete it if we could, but unfortunately that is not possible.

argh, you're absolutely right: i am currently using the tags tarball
instead of the release one; i'll switch to that when a new release is
cut.


>> At the same time, i'm wondering if
>> https://github.com/numpy/numpy/pull/20695 (the PR that introduced
>> NPY_DISABLE_SVML, also probably
>> https://github.com/numpy/numpy/pull/20741 for the doc) should be
>> backported to the 1.22.x branch, so that we can start using it in the
>> release that introduced that feature.
>
>
> I followed up on this on the PR. Thanks for pointing out this issue.

Thanks!
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: deprecating numpy.distutils

[Sandro Tosi]

> 4. Write a docs page with migration info on how to deal with the deprecation, 
> and update it before the 1.23.0 release.

i think this will be extremely important. Just to give you the current
POV of debian projects using numpy.distutils:

- 
https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1
(there are 74 of them)
- https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0
(only one "from numpy import distutils")

hope this could help you come up with a smooth deprecation process :)

thanks,
-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Documentation Team meeting - Monday January 17

[Melissa Mendonça]

Hi all!

Our first Documentation Team meeting of 2022 snuck up on me, sorry for not
announcing sooner! We'll meet on *Monday, January 17* at ***4PM UTC***.

All are welcome - you don't need to already be a contributor to join. If
you have questions or are curious about what we're doing, we'll be happy to
meet you!

If you wish to join on Zoom, use this link:

https://zoom.us/j/96219574921?pwd=VTRNeGwwOUlrYVNYSENpVVBRRjlkZz09

Here's the permanent hackmd document with the meeting notes (still being
updated in the next few days!):

https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg


Hope to see you around!

** You can click this link to get the correct time at your timezone:
https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentation+Team+Meeting&iso=20220117T16&p1=1440&ah=1


*** You can add the NumPy community calendar to your google calendar by
clicking this link: https://calendar.google.com/calendar
/r?cid=YmVya2VsZXkuZWR1X2lla2dwaWdtMjMyamJobGRzZmIyYzJqODFjQGdyb3VwLmNhbGVuZGFyLmdvb2dsZS5jb20

- Melissa
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com