[Numpy-discussion] Re: Meson - C extension - Finding numpy includes in virtual env
On Sun, Nov 26, 2023 at 9:06 PM Nathan wrote: > I want to caution about using `pip install -e .` to get a development > install of numpy. This will work fine working on numpy itself, but won’t be > useful if you need to use the development version of numpy to build another > library. This doesn’t work because in-place installs don’t install the > numpy headers (arguably it was a bug that the old setuptools install did) > into the git repo, so the include paths `np.get_include()` reports won’t be > correct. > This sounds about right - editable installs are only useful for working on `numpy` itself, and perhaps basic pure Python packages on top. But editable installs are fundamentally not complete installs, and should not be used for developing a set of packages together that depend on each other. I'll note that there's an active discussion (PEP 735 draft) which touched on a "workspaces" concept for this. Also, please do not ever use editable installs without `--no-build-isolation`, that may lead to weird issues. Cheers, Ralf > > See this meson-python issue: > https://github.com/mesonbuild/meson-python/issues/429 > > For my work I tend to use a persistent build directory with build > isolation disabled as discussed in the meson-python docs. This gives me > fast rebuilds without using an in-place build. It does mean there’s a build > and install step when you edit python code in numpy that would otherwise be > unnecessary and sometimes the cache can go stale for reasons that aren’t > totally obvious. > > In principle numpy could fix this by ensuring the headers get generated in > the git repo in the place they’re supposed to be installed. I have no idea > how hard it would be beyond that it would definitely require messing with > the codegen scripts. > > On Sun, Nov 26, 2023 at 10:53 AM Stefan van der Walt via NumPy-Discussion < > numpy-discussion@python.org> wrote: > >> Hi Doug, >> >> On Sun, Nov 26, 2023, at 06:29, Doug Turnbull wrote: >> >> To debug, I ran `pip install . --no-build-isolation` it worked (using >> venv's numpy) >> >> >> When developing NumPy, we typically build in the existing environment. >> This is done either via `pip install -e .` (which installs hooks to trigger >> a re-compile upon import), or via the spin tool ( >> https://github.com/scientific-python/spin), which have meson commands >> pre-bundled: >> >> pip install spin >> spin # lists commands available >> >> Best regards, >> Stéfan >> >> ___ >> NumPy-Discussion mailing list -- numpy-discussion@python.org >> To unsubscribe send an email to numpy-discussion-le...@python.org >> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >> Member address: nathan12...@gmail.com >> > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: ralf.gomm...@gmail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Hi, I'm involved in packaging NumPy for http://pkgsrc.org/. We install a set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the same prefix. This includes multiple variants of OpenBLAS with regard to multithreading (and indexing). For this purpose, we point software to use the build-time chosen BLAS implementation via BLAS_LIBS and similar variables, or, as seems to be appropriate for the new meson build of NumPy, via .pc files. A package depends on the generic BLAS library family and central user configuration chooses which one the packages should use during build. What would be the correct way to force the NumPy build to just use our BLAS choice, avoiding any automatisms that might surprise us? How agnostic is NumPy regarding the offered BLAS? Does it have to know that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd imagine just setting OMP_NUM_THREADS handles paralellism) or MKL, Netlib, … ? It doesn't scale to have to tell it "openblas" or "netlib", as there is no universal vocabulary to name the variants, and NumPy doesn't even know openblas_openmp from serial openblas or openblas_pthread (right?). Basically, I want to do meson setup -Dcblas_pc=$CBLAS_PC with CBLAS_PC being the module name of one of $prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/openblas_pthread.pc $prefix/lib/pkgconfig/openblas_openmp.pc $prefix/lib/pkgconfig/openblas64_openmp.pc so that pkg-config does its thing without the NumPy build guessing around. Is that feasible already? Is it easily supportable with some changes to the build? I dream of a world where package build scripts don't have to add dozens of idiosyncratic lines to detect these libs. I'd like things to work like for CMake's FindBLAS with -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see https://cmake.org/cmake/help/latest/module/FindBLAS.html since version 3.25). Can we have that? And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? These are differing libraries, possibly coming in differing binaries, even if your OpenBLAS builds also combine them. So I guess it should be -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of the reference Netlib implementation provides four distinct libraries and .pc files: $prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/blas.pc $prefix/lib/pkgconfig/lapacke.pc $prefix/lib/pkgconfig/lapack.pc We do support installing openblas64 and friends alongside the others and I imagine just setting an ILP64 option and repective symbol suffix (none as of yet, as it's not a settled thing upstream) for the NumPy build if a 64 variant is chosen by the user. I wonder a bit if there are possible pitfalls combining other libraries with Python and indirectly some incompatible BLAS variant via NumPy … but one point of our user choice is that they could ensure that all packages really use the same BLAS. Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
On Mon, Nov 27, 2023 at 2:10 PM Dr. Thomas Orgis < thomas.or...@uni-hamburg.de> wrote: > Hi, > > I'm involved in packaging NumPy for http://pkgsrc.org/. We install a > set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the > same prefix. This includes multiple variants of OpenBLAS with regard to > multithreading (and indexing). For this purpose, we point software to > use the build-time chosen BLAS implementation via BLAS_LIBS and similar > variables, or, as seems to be appropriate for the new meson build of > NumPy, via .pc files. > > A package depends on the generic BLAS library family and central user > configuration chooses which one the packages should use during build. > > What would be the correct way to force the NumPy build to just use our > BLAS choice, avoiding any automatisms that might surprise us? > > How agnostic is NumPy regarding the offered BLAS? Does it have to know > that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd > imagine just setting OMP_NUM_THREADS handles paralellism) The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality - whether those execute under the hood in parallel or not, or with OpenMP or pthreads, is unknown. When a user or downstream library wants to control that parallelism, they can use an environment variable or https://github.com/joblib/threadpoolctl. or MKL, Netlib, … ? By default, the NumPy build will try all of those, in the order given by the `blas-order` and `lapack-order` build options (see `meson_options.txt in the root of the repo). It doesn't scale to have to tell it "openblas" or "netlib", > as there is no universal vocabulary to name the variants, and NumPy > doesn't even know openblas_openmp from serial openblas or > openblas_pthread (right?). > > Basically, I want to do > > meson setup -Dcblas_pc=$CBLAS_PC > > with CBLAS_PC being the module name of one of > > $prefix/lib/pkgconfig/cblas.pc > $prefix/lib/pkgconfig/openblas_pthread.pc > $prefix/lib/pkgconfig/openblas_openmp.pc > $prefix/lib/pkgconfig/openblas64_openmp.pc > > so that pkg-config does its thing without the NumPy build guessing > around. Is that feasible already? Is it easily supportable with some > changes to the build? I dream of a world where package build scripts > don't have to add dozens of idiosyncratic lines to detect these libs. > Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas= -C-Dlapack=`. See http://scipy.github.io/devdocs/building/blas_lapack.html for more guidance. > I'd like things to work like for CMake's FindBLAS with > -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see > > https://cmake.org/cmake/help/latest/module/FindBLAS.html > > since version 3.25). > > Can we have that? > Yes, that is implemented since NumPy 1.26.2 and in the main branch. > > And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? > No need for LAPACKE. > These are differing libraries, possibly coming in differing binaries, > even if your OpenBLAS builds also combine them. So I guess it should be > -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of > the reference Netlib implementation provides four distinct libraries > and .pc files: > > $prefix/lib/pkgconfig/cblas.pc > $prefix/lib/pkgconfig/blas.pc > $prefix/lib/pkgconfig/lapacke.pc > $prefix/lib/pkgconfig/lapack.pc > > We do support installing openblas64 and friends alongside the others > and I imagine just setting an ILP64 option and repective symbol suffix > (none as of yet, as it's not a settled thing upstream) for the NumPy > build if a 64 variant is chosen by the user. I wonder a bit if there > are possible pitfalls combining other libraries with Python and > indirectly some incompatible BLAS variant via NumPy … but one point of > our user choice is that they could ensure that all packages really use > the same BLAS. > You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS. Cheers, Ralf > > > Alrighty then, > > Thomas > > -- > Dr. Thomas Orgis > HPC @ Universität Hamburg > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: ralf.gomm...@googlemail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: savetxt() has fmt='%.18e' as default, but fmt='%.16e' is always sufficient
Thanks for the explanation, Robert Kern. It seems then that indeed, these days the optimal default would be `fmt='%.16e'`. The test on StackOverflow was just a quick demonstration. See also the description for `DBL_DECIMAL_DIG` here: https://en.cppreference.com/w/cpp/header/cfloat > But if your true concern is that 9% of disk space, > you probably don't want to be using `savetxt()` in any case. Well, I do in fact want to save as text, but I would rather not do so in an unnecessarily wasteful fashion. So i just set `fmt='%.16e'` myself, no biggie. The point, however, is that most users will not set this, and so `savetxt()` is a cause of wasted disk space generally, which I think should be fixed (if indeed we can be 100% certain that `fmt='%.16e'` is in fact always enough). ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Am Mon, 27 Nov 2023 14:58:45 +0100 schrieb Ralf Gommers : > The NumPy build does not know anything about this. It will just build, and > it will simply call the OpenBLAS functionality Great! > Yes, that is possible. You should be building with a build frontend (pip or > pypa/build) and then the invocation will include `-C-Dblas= > -C-Dlapack=`. I'm confused about these frontends, I must say. I imagined that if you're using meson, one could just call meson setup/build? That being said: I am not sure now myself how the pkgsrc build actually works right now. There's common machinery to 'build python stuff' and the part about meson-based packages is rather fresh and not documented yet. The build output starts with * Building wheel... + /data/pkg/bin/python3.11 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/vendored-meson/meson/meson.py setup /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/.mesonpy-_lv … so some wrapped call to a vendored copy of meson that NumPy ships. Adding -Dblas=$CBLAS_PC to that command should do the trick, no? (however that is effected) > > And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? > > > > No need for LAPACKE. Good, if also somewhat weird;-) I'm curious, though: When you need the CBLAS API, why is the dependency called blas and not cblas? In practice, most accelerated libraries offer all APIs in one binary and -Dlapack is already redundant, but when we use the netlib reference, blas, cblas, lapack, and lapacke are distinct entities. Calling cblas just blas where lapack _does_ mean the Fortran one, is rather confusing. > You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to > craft a blas.pc which points at a 64-bit BLAS. So -Dblas=openblas64 -Dlapack=openblas64 -Duse-ilp64 would do it, right? Alrighty then, Thomas PS: You might want to fix that one: ../../numpy/meson.build:124: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'blas' custom lookup. -- Dr. Thomas Orgis HPC @ Universität Hamburg ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
On Mon, Nov 27, 2023 at 6:51 PM Dr. Thomas Orgis <
thomas.or...@uni-hamburg.de> wrote:
> Am Mon, 27 Nov 2023 14:58:45 +0100
> schrieb Ralf Gommers :
>
> > The NumPy build does not know anything about this. It will just build,
> and
> > it will simply call the OpenBLAS functionality
>
>
> Great!
>
> > Yes, that is possible. You should be building with a build frontend (pip
> or
> > pypa/build) and then the invocation will include `-C-Dblas= name>
> > -C-Dlapack=`.
>
> I'm confused about these frontends, I must say. I imagined that if
> you're using meson, one could just call meson setup/build? That being
> said: I am not sure now myself how the pkgsrc build actually works
> right now. There's common machinery to 'build python stuff' and the
> part about meson-based packages is rather fresh and not documented yet.
>
You have to go through a "build frontend" to produce a wheel, which then
gets installed/repackaged for your distro. If you call meson/ninja
directly, you will not get the Python package metadata that meson-python
produces. And you do need that, others there are some things that will
break (e.g., using `importlib` APIs for introspecting Python packages). So
what your machinery should be doing is building with `pip install .
--no-build-isolation` or `python -m build --no-isolation`.
>
> The build output starts with
>
> * Building wheel...
> + /data/pkg/bin/python3.11
> /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/vendored-meson/meson/meson.py
> setup /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2
> /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/.mesonpy-_lv
>
> … so some wrapped call to a vendored copy of meson that NumPy ships.
>
Yes, we need that for extra BLAS/LAPACK and SIMD related functionality that
is still in the process of being upstreamed into Meson.
> Adding -Dblas=$CBLAS_PC to that command should do the trick, no?
> (however that is effected)
>
Sounds like it, assuming CBLAS_PC is the name of a library.
>
> > > And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK?
> > >
> >
> > No need for LAPACKE.
>
> Good, if also somewhat weird;-) I'm curious, though: When you need the
> CBLAS API, why is the dependency called blas and not cblas? In
> practice, most accelerated libraries offer all APIs in one binary and
> -Dlapack is already redundant, but when we use the netlib reference,
> blas, cblas, lapack, and lapacke are distinct entities. Calling cblas
> just blas where lapack _does_ mean the Fortran one, is rather confusing.
>
Partly a matter of history since we always did it like this, but I think
there's more to it. The two libraries are called BLAS and LAPACK, those
offer distinct functionality. CBLAS and LAPACKE are basically much less
important implementation details, and typically shipped in the same library
because they're interfaces to the exact same functionality. We're not
"calling CBLAS just BLAS" here, but rather: BLAS is the main name and has
the functionality you want. CBLAS is an optional interface to that, and if
you want it you have to ask for it with (in Meson):
dependency('blas', modules: ['cblas'])
It doesn't make much sense for us to expose CBLAS (or LAPACKE) as a
separate thing in our own build interface.
> > You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work
> to
> > craft a blas.pc which points at a 64-bit BLAS.
>
> So -Dblas=openblas64 -Dlapack=openblas64 -Duse-ilp64 would do it, right?
>
Exactly.
>
>
> Alrighty then,
>
> Thomas
>
> PS: You might want to fix that one:
>
> ../../numpy/meson.build:124: WARNING: Project targets '>=1.2.99' but uses
> feature introduced in '1.3.0': dep 'blas' custom lookup.
>
Yeah, that'll go away when we update the vendored copy, will be done in the
next few days.
Cheers,
Ralf
> --
> Dr. Thomas Orgis
> HPC @ Universität Hamburg
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ralf.gomm...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: savetxt() has fmt='%.18e' as default, but fmt='%.16e' is always sufficient
On Mon, 27 Nov 2023 at 17:19, Jeppe Dakin wrote: > > But if your true concern is that 9% of disk space, > > you probably don't want to be using `savetxt()` in any case. > Well, I do in fact want to save as text, but I would rather not do so in an > unnecessarily wasteful fashion. `fmt='%s'` might be a good option if you want to save space: 1. it will output the nearest decimal value that round-trips 2. it handles the case of 32bit floats needing half as many digits 3. it does the right thing for integer values Sam ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
