Package: autopkgtest Version: 5.22 Severity: normal Tags: help The SchrootRunner.test_transient_apt_failure test-case in autopkgtest seems to have regressed at some point since it was added. Because it's only run when using schroot, it is not run routinely by the autopkgtest developers or by CI, which means we don't know when this regressed.
I didn't write this code, but the intention of the test_transient_apt_failure and test_persistent_apt_failure test-cases added in 3893ede "Make --apt-upgrade consider a "404" error as test failure" (2016-10-28) seems to be that autopkgtest should report apt misconfigurations as an "erroneous package" (exit status 12, counts as a test failure for the package under test) whereas transient DNS issues should be reported as a "testbed failure" (exit status 16, considered to be a problem with the CI infrastructure rather than a problem with the package under test). The test_transient_apt_failure test-case tries to simulate a DNS issue by setting `deb http://nonexist.ing/ nonexisting main` as the only apt source. However, this gets reported as exit status 12 "erroneous package". This might be a behaviour change in apt, or a regression in autopkgtest itself? If I'm reading the test output correctly, I think apt is exiting 0 when invoked in the setup command, which means autopkgtest continues to trying to run (its own!) tests, but fails to satisfy their dependencies. The test_persistent_apt_failure test-case fails for an unrelated reason which I know how to fix, so please disregard that one for the purposes of this bug report. Reproducer for test failure: * have a minimal qemu virtual machine with Debian 11 * sudo apt-get --no-install-recommends build-dep autopkgtest * sudo apt-get --no-install-recommends install \ autodep8 debootstrap sbuild schroot * sudo mkdir /srv/chroot * sudo sbuild-createchroot --arch=amd64 bullseye \ /srv/chroot/bullseye-amd64-sbuild * as a non-root member of the sbuild group: AUTOPKGTEST_TEST_SCHROOT=bullseye-amd64-sbuild ./tests/autopkgtest * minimized reproducer: AUTOPKGTEST_TEST_SCHROOT=bullseye-amd64-sbuild \ ./tests/autopkgtest \ SchrootRunner.test_transient_apt_failure which failed for me with: ====================================================================== FAIL: test_transient_apt_failure (__main__.SchrootRunner) transient apt errors are treated as testbed failure ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/user/autopkgtest/./tests/autopkgtest", line 3782, in test_transient_apt_failure self.assertEqual(code, 16, err) AssertionError: 12 != 16 : autopkgtest [09:12:21]: starting date: 2022-05-18 autopkgtest [09:12:21]: git checkout: 7ddc91e tests: Update for perl autodep8 behaviour changes autopkgtest [09:12:21]: host host; command line: /home/user/autopkgtest/runner/autopkgtest --setup-commands 'echo "deb http://nonexist.ing/ nonexisting main" > /etc/apt/sources.list' --apt-upgrade -- schroot bullseye-amd64-sbuild autopkgtest [09:12:21]: @@@@@@@@@@@@@@@@@@@@ test bed setup Err:1 http://nonexist.ing nonexisting InRelease Could not resolve 'nonexist.ing' Reading package lists... W: Failed to fetch http://nonexist.ing/dists/nonexisting/InRelease Could not resolve 'nonexist.ing' W: Some index files failed to download. They have been ignored, or old ones used instead. autopkgtest [09:12:22]: testbed dpkg architecture: amd64 autopkgtest [09:12:22]: testbed running kernel: Linux 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) autopkgtest [09:12:22]: @@@@@@@@@@@@@@@@@@@@ unbuilt-tree . autopkgtest [09:12:22]: testing package autopkgtest version 5.22 autopkgtest [09:12:22]: build needed for binaries perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = "C.UTF-8", LANG = "en_GB.utf8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). autopkgtest: WARNING: Test dependencies are unsatisfiable - calling apt install on test deps directly for further data about failing dependencies in test logs autopkgtest [09:12:22]: ERROR: erroneous package: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.
