Hi Timo,
On Mon, Sep 09, 2024 at 05:46:41PM +0200, Timo Röhling wrote:
> numpy-config is actually in python3-numpy, not in python3-numpy-dev, so it
> is not MA: same. Without $PKG_CONFIG, it will simply return the
> configuration for python3-numpy's own architecture, which is arguably not
> surprising behavior. But let's examine the alternatives:
In principle, this simplifies things. Any package that actually uses
numpy-config at build time must hence "Build-Depends: python3-numpy". In
particular, "Build-Depends: python3-numpy:any" nor "Build-Depends:
python3-numpy:native" cut it and that way it'll be always correct (and
cross building will be broken).
I'm not sure this is the worst of options. Cross builders will see the
unsatisfiable dependency and in updating the "python3-numpy" dependency
to something "better", they are obliged to make the package stop using
numpy-config.
> > Things we did for other packages:
> > * Delete foo-config and patch all downstreams to use pkgconf.
> Given that I'm not a big fan of foo-config scripts in the first place, I'm
> very tempted. The NumPy approach is really contrived, as they want you to
> run
>
> export PKG_CONFIG_PATH=$(numpy-config --pkgconfigdir)
> NUMPY_INCLUDE_DIR=$(pkg-config numpy --cflags)
If we were still using pkg-config (rather than pkgconf), this would be
really bad for Debian as setting PKG_CONFIG_PATH made $triple-pkg-config
stop considering the multiarch location and thus setting PKG_CONFIG_PATH
would break pkg-config for all other packages. I'm not sure whether it
still is that bad, but we still have the numpy-config here.
> This is probably a workaround for the limitations of setuptools; you can
> create globally visible script entrypoints, but you cannot install .pc files
> to a globally visible location.
Indeed. They'd like to use pkg-config, but in shoehorning it into
Python, they break it for everyone else. I was considering updating it
to this:
: "${PKG_CONFIG:=pkg-config}"
if ! $PKG_CONFIG --exists numpy; then
export PKG_CONFIG_PATH=$(numpy-config --pkgconfigdir)
fi
NUMPY_INCLUDE_DIR=$($PKG_CONFIG numpy --cflags)
I'm not sure upstream would want to change their advice to this. In
particular, we cannot just prepend $(numpy-config --pkgconfigdir) to the
search path, because in the Debian layout numpy-config will always be
the build architecture one and thus give us a directory that happens to
have a numpy.pc for the build architecture, which is not what we wanted.
> On the other hand, I'd *really* like to avoid introducing yet another
> Debianism to such a popular package in the Python ecosystem, and the
> numpy-config approach is prominently featured in the upstream docs, so we
> can assume it will be adopted and be a potential cause for quite a few bug
> reports.
I suspect that only way of making the upstream-recommended way just work
is introducing another binary package.
> > * Enhance foo-config to parse its $0 and extract the triplet from that.
> > This approach makes AC_PATH_TOOL and most pre-meson/cmake users likely
> > are using autoconf.
> That would work, but would still require lots of patches in downstream
> packages, which are likely to hardcode numpy-config in their build scripts.
Not sure how that works, but at least autotools stuff should work like
AC_PATH_TOOL([NUMPY_CONFIG],[numpy-config]) and will be occasionally
mistaken for AC_PATH_PROG([NUMPY_CONFIG],[numpy-config]) and all one has
to do here is s/PROG/TOOL/. More barebones Makefile-based build systems
tend to use $CROSS_COMPILE and there we can update bare numpy-config
invocations with "${CROSS_COMPILE}numpy-config".
Still yeah, lots of patches.
> And it will not solve cross-building for packages which interrogate
> numpy.get_include() directly, as
> SciPy does in its Meson build.
But this is not a matter of numpy-config then. Would you mind detailing
how numpy.get_include() works? Generally speaking, this sounds like a
bad idea in any case.
> > * Further split the package into two -dev packages.
> Hard pass. I'm not going to atomize the package further (and given that
> numpy-config is not in python3-numpy-dev, it is probably not needed anyway).
Don't rule it out so quickly. How bout this layout:
python3-numpy-dev as it is proposed here would be kept as is. We'd move
numpy-config to a new package python3-numpy-dev-config without
python3-numpy depending on it. This will make some packages FTBFS, but
we can fix each of them by adding the new package to Build-Depends,
which is a relatively simple change not touching upstream stuff. As we
touch them, we can also annotate python3-numpy with :native. Now as we
cross build those packages, we'll install
python3-numpy-dev-config:$DEB_HOST_ARCH and numpy-config --pkgconfigdir
will simply produce
/usr/lib/${DEB_HOST_MULTIARCH}/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig
as it knows that it is outputting for $DEB_HOST_ARCH. That way, what
upstream recommends will likely just work as it is simply returning the
default pkgconf search path. Stuff will break though if a package
happens to require both ${DEB_BUILD_GNU_TYPE}-pkg-config and
${DEB_HOST_GNU_TYPE}-pkg-config and properly differentiates between them
as the setting of PKG_CONFIG_PATH will override the prefix. I still
expect a fair number of packages to just work this way.
> Counter-proposal: If $PKG_CONFIG still seems too iffy for your taste, would
> you feel better if we defined a NumPy-specific environment variable such as
> NUMPY_EXTENSION_ARCHITECTURE?
I think that's actually worse than setting $PKG_CONFIG. The $PKG_CONFIG
approach is subtle, but not without merit. CMake also opted for picking
up this variable and Perl's MakeMaker also tends to use it. Hence it
does have some prior art. What does not have much prior art is
influencing a foo-config script via $PKG_CONFIG as the common way has
been to just get rid of foo-config instead and that doesn't work for
numpy for the reasons you gave above.
> Or is there yet another way how we could determine the desired architecture
> in a cross-build setting?
Really there are only two widely adopted ways:
* The triplet-prefix determines the architecture of a foo-config.
* The package architecture determines the architecture of a foo-config.
I'm not sure whether fixing it at this level is a good way. Touching
PKG_CONFIG_PATH in a build system is a bad idea to start with as it is
a tool meant to be used by the builder, not by the software package. Is
there any way to change the upstream recommendation in a way of not
exporting PKG_CONFIG_PATH when it is not needed? Failing that, the
PKG_CONFIG approach might be least bad, but that's really worth a
README.multiarch or something then.
Helmut
