[Numpy-discussion] Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Hi, I'm involved in packaging NumPy for http://pkgsrc.org/. We install a set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the same prefix. This includes multiple variants of OpenBLAS with regard to multithreading (and indexing). For this purpose, we point software to use the build-time chosen BLAS implementation via BLAS_LIBS and similar variables, or, as seems to be appropriate for the new meson build of NumPy, via .pc files. A package depends on the generic BLAS library family and central user configuration chooses which one the packages should use during build. What would be the correct way to force the NumPy build to just use our BLAS choice, avoiding any automatisms that might surprise us? How agnostic is NumPy regarding the offered BLAS? Does it have to know that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd imagine just setting OMP_NUM_THREADS handles paralellism) or MKL, Netlib, … ? It doesn't scale to have to tell it "openblas" or "netlib", as there is no universal vocabulary to name the variants, and NumPy doesn't even know openblas_openmp from serial openblas or openblas_pthread (right?). Basically, I want to do meson setup -Dcblas_pc=$CBLAS_PC with CBLAS_PC being the module name of one of $prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/openblas_pthread.pc $prefix/lib/pkgconfig/openblas_openmp.pc $prefix/lib/pkgconfig/openblas64_openmp.pc so that pkg-config does its thing without the NumPy build guessing around. Is that feasible already? Is it easily supportable with some changes to the build? I dream of a world where package build scripts don't have to add dozens of idiosyncratic lines to detect these libs. I'd like things to work like for CMake's FindBLAS with -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see https://cmake.org/cmake/help/latest/module/FindBLAS.html since version 3.25). Can we have that? And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? These are differing libraries, possibly coming in differing binaries, even if your OpenBLAS builds also combine them. So I guess it should be -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of the reference Netlib implementation provides four distinct libraries and .pc files: $prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/blas.pc $prefix/lib/pkgconfig/lapacke.pc $prefix/lib/pkgconfig/lapack.pc We do support installing openblas64 and friends alongside the others and I imagine just setting an ILP64 option and repective symbol suffix (none as of yet, as it's not a settled thing upstream) for the NumPy build if a 64 variant is chosen by the user. I wonder a bit if there are possible pitfalls combining other libraries with Python and indirectly some incompatible BLAS variant via NumPy … but one point of our user choice is that they could ensure that all packages really use the same BLAS. Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Am Mon, 27 Nov 2023 14:58:45 +0100 schrieb Ralf Gommers : > The NumPy build does not know anything about this. It will just build, and > it will simply call the OpenBLAS functionality Great! > Yes, that is possible. You should be building with a build frontend (pip or > pypa/build) and then the invocation will include `-C-Dblas= > -C-Dlapack=`. I'm confused about these frontends, I must say. I imagined that if you're using meson, one could just call meson setup/build? That being said: I am not sure now myself how the pkgsrc build actually works right now. There's common machinery to 'build python stuff' and the part about meson-based packages is rather fresh and not documented yet. The build output starts with * Building wheel... + /data/pkg/bin/python3.11 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/vendored-meson/meson/meson.py setup /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/.mesonpy-_lv … so some wrapped call to a vendored copy of meson that NumPy ships. Adding -Dblas=$CBLAS_PC to that command should do the trick, no? (however that is effected) > > And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? > > > > No need for LAPACKE. Good, if also somewhat weird;-) I'm curious, though: When you need the CBLAS API, why is the dependency called blas and not cblas? In practice, most accelerated libraries offer all APIs in one binary and -Dlapack is already redundant, but when we use the netlib reference, blas, cblas, lapack, and lapacke are distinct entities. Calling cblas just blas where lapack _does_ mean the Fortran one, is rather confusing. > You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to > craft a blas.pc which points at a 64-bit BLAS. So -Dblas=openblas64 -Dlapack=openblas64 -Duse-ilp64 would do it, right? Alrighty then, Thomas PS: You might want to fix that one: ../../numpy/meson.build:124: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'blas' custom lookup. -- Dr. Thomas Orgis HPC @ Universität Hamburg ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Thanks a lot for the patience and explanations.
Am Mon, 27 Nov 2023 19:59:13 +0100
schrieb Ralf Gommers :
> You have to go through a "build frontend" to produce a wheel, which then
> gets installed/repackaged for your distro.
This is obviously happening in pkgsrc.
> > The build output starts with
> >
> > * Building wheel...
I only need to add those -C-Dblas= parameters at the correct place.
> > … so some wrapped call to a vendored copy of meson that NumPy ships.
> >
>
> Yes, we need that for extra BLAS/LAPACK and SIMD related functionality that
> is still in the process of being upstreamed into Meson.
… and I dream of the world where BLAS is 'just a library' without any
special support in build systems.
> > Adding -Dblas=$CBLAS_PC to that command should do the trick, no?
> > (however that is effected)
> >
>
> Sounds like it, assuming CBLAS_PC is the name of a library.
It's the name to give to pkg-config to identify the module for CBLAS.
> offer distinct functionality. CBLAS and LAPACKE are basically much less
> important implementation details, and typically shipped in the same library
> because they're interfaces to the exact same functionality.
Yes and no. The Netlib reference implementation comes (or at least can
be built that way) in separate pieces, the code of which is what the
optimized BLAS vendors like to use to build those combined libraries.
So in my case, CBLAS_PC can be just cblas, with the effect
$ pkg-config --libs cblas
-L/data/pkg/lib -lcblas
and
$ pkg-config --libs --static cblas
-L/data/pkg/lib -lcblas -lblas
The reference libcblas is a separate library which could be used to
wrap around any plain BLAS library. Would the meson machinery be
confused by that?
In practice, for optimized builds, you'll get all 4 pieces in one
package (openblas at least), but I'm always also dealing with the
Netlib case where CBLAS is something different from BLAS.
> dependency('blas', modules: ['cblas'])
So there's this elaborate machinery that special-cases BLAS as a
library that could (not) offer certain sets of API? Well, as long as
the dynamic libcblas is used, the base BLAS API should be transparently
avaible and those meson tests will work … ?
I'll do some testing tomorrow, at least.
Alrighty then,
Thomas
--
Dr. Thomas Orgis
HPC @ Universität Hamburg
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Am Sun, 3 Dec 2023 19:54:10 +0100
schrieb "Dr. Thomas Orgis" :
> > You have to go through a "build frontend" to produce a wheel, which then
> > gets installed/repackaged for your distro.
>
> This is obviously happening in pkgsrc.
>
> I'll do some testing tomorrow, at least.
Well, now is another day. Pkgsrc uses python -m build and I added
-Csetup-args=-Dblas=${CBLAS_PC} -Csetup-args=-Dlapack=${LAPACK_PC}
which seems to work out fine using cblas.pc and lapack.pc in the case
of the netlib install. In fact, most linking is done only to libblas.so
instead of libcblas.so, as the linker is smart enough to throw away the
unused lib.
Alrighty then,
Thomas
--
Dr. Thomas Orgis
HPC @ Universität Hamburg
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] incomplete BLAS/CBLAS linking (Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module)
Hapy holidays … but I have an issue still that hopefully can be
addressed with the meson blas detection you are upstreaming(?).
Am Wed, 6 Dec 2023 18:06:01 +0100
schrieb Ralf Gommers :
> > Well, now is another day. Pkgsrc uses python -m build and I added
> >
> > -Csetup-args=-Dblas=${CBLAS_PC} -Csetup-args=-Dlapack=${LAPACK_PC}
> >
> > which seems to work out fine using cblas.pc and lapack.pc in the case
> > of the netlib install. In fact, most linking is done only to libblas.so
> > instead of libcblas.so, as the linker is smart enough to throw away the
> > unused lib.
> >
>
> Great, thanks for confirming!
This works for numpy and also installs scipy nicely, but this produces
a broken scipy install when using netlib reference libraries from
pkgsrc. These come as
libblas.so
liblapack.so (NEEDing libblas.so)
libcblas.so (NEEDing libblas.so)
libpapacke.so (NEEDing liblapack.so, hence libblas.so)
and their respective .pc files. This is the natural order that occus to
me when building from netlib upstream. This also means that one could
just replace BLAS and put stock LAPACK on top, what optimized BLAS libs
usually start out with. Only that they tend to pack all symbols into
one common library, which then project builds like numpy rely on.
Telling the meson build that BLAS is libcblas works as long as actually
CBLAS symbols are used. If not — I presume now, as I didn't yet see the
actual build lines that are triggered via the python -m build and meson
indirections — the linker might discard the -lcblas and leave symbols
unresolved (--as-needed but no --no-undefined).
This happens with scipy:
$ LANG=C readelf -d
/data/pkg/lib/python3.11/site-packages/scipy/sparse/linalg/_dsolve/_superlu.so
|grep NEEDED
0x0001 (NEEDED) Shared library: [libm.so.6]
0x0001 (NEEDED) Shared library: [libc.so.6]
It would link against libopenblas_openmp.so if that had been the CBLAS
(and LAPACK) choice and all would be fine, but here, it should link
with libcblas.so, or directly to libblas.so, just like our regular
install of superlu:
$ LANG=C readelf -d /data/pkg/lib/libsuperlu.so|grep NEEDED
0x0001 (NEEDED) Shared library: [libblas.so.3]
0x0001 (NEEDED) Shared library: [libm.so.6]
0x0001 (NEEDED) Shared library: [libc.so.6]
Of course, just not vendoring superlu would be one solution for scipy,
but I think the deeper issue with the meson BLAS support should be
solved: The 4 parts of the BLAS canon (not talking about SCALAPACK etc.
yet) need to be handled explicitly.
It is confusing, though, as meson prints this:
Run-time dependency blas found: YES 3.11.0
Run-time dependency cblas found: YES 3.11.0
Run-time dependency lapack found: YES 3.11.0
It suggests that it looked for and found 3 libraries, but actually, it
only cared for -llapack and -lcblas. It needs to find -lblas directly,
too (or the cblas package separately, for that matter, not as component
of blas).
Is that easily fixable from your side? (I'm assuming numpy, scipy and
the future stock BLAS support of meson are handled together.) Is this
just an oversight on the scipy side and they could link the vendored
superly with -lblas without changing the meson machinery?
Alrighty then,
Thomas
--
Dr. Thomas Orgis
HPC @ Universität Hamburg
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: incomplete BLAS/CBLAS linking (Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module)
if I enable/disable threading (or in funny ways). You need to change that, or hide all names behind some alternatives system. > Yes indeed. At least, I cannot think of any package/project that > doesn't provide CBLAS with BLAS. While BLAS and LAPACK clearly are > separate from each other, and there are projects that provide only > one (e.g., BLIS is BLAS-only). Thanks for pointing that out about BLIS, that pkgsrc doesn't carry yet. So in our scheme, we'd need to do an extra build of netlib LAPACK that links to libblis. So that would be blis.pc (BLIS with BLAS+CBLAS) lapack-blis.pc (wrapping the above) lapacke-blis.pc (wrapping the above) and for NumPy build -Dblas=blis (or -Dblas=blis-for-cluster-partition) and -Dlapack=lapack-blis (or …). In pkgsrc, we introduced your view of the overall package, actually. User sets PKGSRC_BLAS_TYPES="openblas-pthread" that chooses one provider for all 4 APIs. But we're not in a world where that simply is exposed as a single entity — except for all our openblas builds that include all parts … > For OpenBLAS we > know it's the default to include LAPACK but it can be built without, > so we need to check symbols, and if they're missing we look for > another library. Another library entirely or only one providing LAPACK on top (a possible disaster, I presume)? > Yes, the search order and which packages are searched for is fully > customizable. But you should be good with the default `-Dblas=auto`. > If not, I'd like to understand why. As I said, we for sure do _not_ want to use -Dblas=auto in package builds (unless we installed AutoBLAS?;-). We want to be specific and fail early if the configured choice is not available in the build environment. Pkgsrc is especially trying hard to avoid locating libraries from the host system (possibly a full GNU/Linux distro) if not explicitly told to. Also, my HPC users also should pick the correct BLAS from a list available in a common prefix for their builds (we're avoiding too many environment modules for libraries that can be installed into a common prefix with pkgsrc). Auto-detection ist not always desired. Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
