I would like to propose this build profile for addition to
<https://wiki.debian.org/BuildProfileSpec>. I'll add it to the wiki if
there seems to be rough consensus.
Name: noinsttests
Changes content of binary packages: No ("C: N" on the wiki)
Changes set of binary packages: Yes ("S: Y" on the wiki)
Description:
Binary packages consisting entirely of automated tests, manual tests,
example/demo programs and test tools should not be generated.
Typical use:
# dbus requires GLib and Python if we will build either build-time
# or as-installed tests
Source: dbus
Build-Depends:
...,
libglib2.0-dev <!nocheck> <!noinsttests>,
python3 <!nocheck> <!noinsttests>,
...
Package: dbus-tests
Build-Profiles: <!noinsttests>
...
I hope I'm getting the Build-Profiles syntax right here - I think
it's <!nocheck> <!noinsttests> rather than <!nocheck !noinsttests> but
the difference between the two always confuses me when combined with
double-negatives. The intended pseudocode is:
if check || insttests:
include libglib2.0-dev and python3 in B-D
else: # (!check) && (!insttests)
exclude them
Longer explanation/rationale:
Existence of this profile
-------------------------
Some upstream packages optionally produce "as-installed" tests, for
example those participating in the GNOME installed-tests initiative
(which is a specification for installing automated test programs and data,
together with metadata describing how to run those tests, conceptually
similar to Debian's autopkgtest). In Debian we would typically use
those tests as the basis for an autopkgtest: dbus-tests from src:dbus
and libglib2.0-tests from src:glib2.0 make good examples of this.
These tests sometimes require extra dependencies at configure time, for
example extra libraries or a Python interpreter. When bootstrapping, it
might be useful to disable these dependencies to break cycles, but when a
new architecture has reached the point where most packages are built, it
would be useful to rebuild packages with their installed-tests so that the
autopkgtests can be run.
"nocheck" is unsuitable for this purpose because the specification on
the wiki says it does not change the contents of binary packages or the
set of binary packages. For packages that reuse their build-time tests
as installed-tests (like dbus and a lot of GNOME packages), it is
necessary to compile the tests in order to install them (perhaps requiring
extra B-D), even if we do not plan to run them at build-time.
Controls generation of entire packages
--------------------------------------
Best-practice for build-profiles seems to be that they may enable or
disable the generation of entire packages, but do not alter the contents
of any individual package: so your build might or might not produce
dbus-tests, but if it did produce dbus-tests, you can be sure that it
is a fully-working version of dbus-tests (and is not missing half its
usual functionality or test coverage because you were cutting down
dependencies). This makes "Depends: dbus-tests" meaningful.
This makes the proposed profile "safe" in the sense used in
https://lists.debian.org/debian-devel/2016/04/msg00381.html, or "C: N"
in the table on the wiki.
Not solely for as-installed tests
---------------------------------
Some source packages bundle their "as-installed" tests into the same
binary package as other non-essential, test-oriented executables to
reduce Packages file bloat, for example:
- dbus-tests mostly contains automated as-installed tests, but also has
manual tests, the dbus-test-tool utility, and a special build of libdbus
that can be loaded via LD_LIBRARY_PATH for extra debug/logging/assertions
- gtk-3-examples originally contained example executables (or manual tests
if you prefer to think of them like that), but now contains the
as-installed tests too
I think this should be allowed, and should not prevent use of the
noinsttests profile. We shouldn't fill up the Packages file and the NEW
queue with separate installed-tests, manual tests and examples binary
packages for no good reason, and if you want to install one of those
categories, it seems to me that being forced to install the others is
not onerous.
Some non-GUI libraries do not depend on a graphical toolkit like GTK
or Qt themselves, but build demo/example programs that do depend on a
graphical toolkit (for example librsvg used to build a very simple GTK
SVG viewer, although we've removed that in bullseye, in anticipation of
its deletion by upstream). For these libraries it would make sense to be
able to disable that "heavy" dependency. I think these demos/examples
are more similar to installed-tests than they are different - they
are intended to be used to prove that the library works correctly, and
are not normally (build-)depended on, or directly useful to end-users
(except perhaps when looking for a minimal reproducer for a bug).
The name of the profile
-----------------------
The "noinsttests" name is inspired by previous discussion (on a bug
asking for more <!nocheck> in some package, if I remember correctly?),
and by ginsttest-runner in the gnome-desktop-testing package[1], which
is a test-runner for GNOME's installed-tests initiative.
If people strongly prefer build profiles to be singular, then "noinsttest"
would also be fine.
The conventional names for the build-time options that enable/disable
GNOME installed-tests are --{en,dis}able-installed-tests (Autotools) and
-Dinstalled_tests={true,false} (Meson), but I think "no-installed-tests"
or "no-as-installed-tests", or a version of one of those names with
words run together, would be inconveniently long for a build-profile.
"notests" would be easily confused with "nocheck", which controls whether
build-time tests are to be run (and presumably takes its name from
`make check` in the GNU coding standards).
I couldn't think of any other short term that would encompass all the
automated and manual tests, examples and demos that I want the scope
of this profile to include. The key facts are that they're test code
intended to be used in testing and debugging (as opposed to e.g. dbus or
libglib2.0-bin, which are intended to be used directly "in production"
by users and/or other programs), and that they're installed (as opposed
to traditional build-time `make check` tests that don't get packaged).
Thoughts? Objections? Requested Pantone shades for the bike shed?
smcv
[1] This name is historical accident: it is not really GNOME-specific,
and works equally well for non-GNOME software that installs metadata
that follows the same simple specification for its as-installed tests,
like dbus does. If other desktop environments want to install their
tests, I'm sure they'd be more than welcome to reuse the same spec
and runner.
The only thing about GNOME installed-tests that is *desktop*-specific
is that the spec says the tests are allowed to assume that they are
run as non-root in an active desktop session (deliberately fairly
vague, but assumed to have at least X11 and D-Bus). Tests that do
not actually require or benefit from this guarantee, like the ones
in dbus, are free to ignore it.
