@laney Thank you for looking into the problem. I will mark systemd-fsckd as flaky and look into the root cause later. It used to be flaky before as well.
I have checked the last 10 failed runs from http://autopkgtest.ubuntu.com/packages/systemd/hirsute/amd64 , with the last one being from 2021-02-24 09:15:57 UTC: $ cat logs https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_091557_d5731@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_044536_8d27f@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_021229_4c845@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_013702_2056d@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_232058_6bbff@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_212809_c332e@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_145406_d6565@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_140557_825b0@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_130532_340ec@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_115952_4304b@/log.gz $ wget -q -i logs $ for i in log.gz*; do zcat $i | tail -n 40 | grep -A 16 "@@@ summary" | grep -v PASS ; done autopkgtest [09:15:42]: @@@@@@@@@@@@@@@@@@@@ summary upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [04:45:21]: @@@@@@@@@@@@@@@@@@@@ summary boot-and-services FAIL non-zero exit status 1 systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [02:00:12]: @@@@@@@@@@@@@@@@@@@@ summary upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [01:23:55]: @@@@@@@@@@@@@@@@@@@@ summary boot-and-services FAIL non-zero exit status 1 upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [23:20:34]: @@@@@@@@@@@@@@@@@@@@ summary boot-and-services FAIL non-zero exit status 1 upstream FAIL non-zero exit status 1 systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [21:19:41]: @@@@@@@@@@@@@@@@@@@@ summary boot-and-services FAIL non-zero exit status 1 upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [14:53:35]: @@@@@@@@@@@@@@@@@@@@ summary boot-and-services FAIL non-zero exit status 1 upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [13:53:19]: @@@@@@@@@@@@@@@@@@@@ summary upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [13:03:15]: @@@@@@@@@@@@@@@@@@@@ summary upstream FAIL timed out systemd-fsckd SKIP exit status 77 and marked as skippable autopkgtest [11:59:24]: @@@@@@@@@@@@@@@@@@@@ summary boot-and-services FAIL non-zero exit status 1 upstream FAIL non-zero exit status 1 systemd-fsckd SKIP exit status 77 and marked as skippable As you can see systemd-fsckd has been skipped in 10 out of 10 runs, but "upstream" test timed out in 7 out of the 10 runs. If you would like to help improving the failure rate caused by the infrastructure please run the systemd autopkgtests on bigger VMs. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1915126 Title: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally Status in Auto Package Testing: Incomplete Status in systemd package in Ubuntu: Invalid Bug description: Hi, I've asked yesterday on IRC but so far got no answer. I think it is right to file a bug about the current state of systemd autopkgtest to unite the efforts in regard to it. I was looking at the systemd tests for a no-change rebuild that really had no reason to now make it fail. While checking I found that as of this month (first bad test on 1st of February) most of the systemd test runs on amd64 will not pass/fail but instead in most cases time out. I've seen various people retry the case as it shows the typical symptoms of a "not the fault of the package, let us retry this" case. But it seems that won't help as the test history is rather clear. * Watch this in monospace to make more sense of it * $ check-autopkgtest-stats.sh -c 50 -p systemd -r "hirsute" -a "amd64" Check last 50 test results for src:systemd on releases 'hirsute' on architectures 'amd64' Of the 50 last tests, we had these subtest failing per release/arch: hirsute amd64 tests-in-lxd (F 2% f 0% S 0% B 12% => P 52%/) .BTTTBTT.BTTTTTBTTTBBTTT..F...................... hostnamed (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... build-login (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... unit-config (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... networkd-testpy (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... localed-locale (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... boot-and-services (F 4% f 0% S 0% B 12% => P 50%/) .BTTTBTT.BTTTTTBTTTBBTTT...............F.....F... timedated (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... boot-smoke (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... logind (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... storage (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... upstream (F 12% f 0% S 0% B 12% => P 42%/) .BTTTBTT.BTTTTTBTTTBBTTTF.....F.F..F........F..F. udev (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... systemd-fsckd (F 8% f 0% S 0% B 12% => P 46%/) FBTTTBTTFBTTTTTBTTTBBTTT.............F......F.... root-unittests (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... localed-x11-keymap (F 0% f 0% S 0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT......................... We see that formerly we had the known to be somewhat flaky, but otherwise working test (time goes from right to left). So we had the usual suspects of tests like upstream/systemd-fsck that failed a few times, but also working runs in between. But since February 22/24 runs failed very bad. 6 (=B) of those cases are aborting mid execution autopkgtest [22:08:58]: ERROR: testbed failure: testbed auxverb failed with exit code 255 And 16 of them timed out <VirtSubproc>: failure: Timed out on waiting for ssh connection Sadly the 2/24 that didn't hard fail by that where broken by the known flaky systemd-fsckd test. Something drives this test crazy that we have to find and resolve, at the current rate nothing depending on it is likely to migrate. To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1915126/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
