On Wed, Jun 09, 2021 at 12:32:02AM +0000, John Scott wrote: >... > With respect to this particular issue, I'd like to share my perspective > wrangling with a package that poses a similar dilemma: GCC (I'm working > on packaging gcc-sh-elf). Like the status quo with SageMath in Debian, > GCC has a test suite where failures are normal, and in general it takes > an individual to watch out for what number of failures counts as "too > many." Rather than hardcode an arbitrary threshold for what number of > failing tests is acceptable, it seems that it's much better, and in the > interest of Debian ports and alternative build environments, to just > let the tests run for informative purposes.
Note that sagemath already seems to have 2 different classes of failures: https://buildd.debian.org/status/package.php?p=sagemath On armhf, mips64el and ppc64el the build fails reliably with Error: critical test failures (e.g. timeout, segfault, etc.) This seems to be a reasonable build-time smoke test. The only special case is s390x: Checking number of failed tests to determine whether to rerun tests in series... No: 504 tests failed, up to 400 failures are tolerated for rerun It is not a surprise that more tests a failing on big endian, but there are actually no "critical test failures". >... > I believe it's in the best interest of Debian users that this bug be > downgraded for Bullseye so Sage can be used in the mostly-wholesome > shape it's in, but since I lack expertise in maintaining it I too will > leave this to someone else. Flaky builds are a pain, also for bullseye. IMHO the best action would be an upload with the following changes: - your superficial autopkgtest - raising the limit from 200 to something that makes builds non-flaky - optionally force build failure on s390x I could sponsor or NMU such an upload. cu Adrian
