So I can reproduce the problem even locally, using a simple: `autopkgtest systemd --test-name=boot-and-services -- lxd autopkgtest/ubuntu/impish/amd64`
When logging into the (stalled) system, while it is waiting for its timeout I can see that eth0 gets configured, but then put in DOWN state. Therefore, systemd-networkd-wait-online.service (and thus network-wait- online.target) never get ready and the test times out waiting on "network-wait-online.target". A simple "networkctl up eth0" (inside the LXD container) during that 60s "timeout-wait" makes the test runner connect an finish the test successfully. The culprit for bringing down the interface seems to be "/run/systemd/system/network-device-down.service", but I have no idea where this files comes from or what it is used for. It's not part of any package. That service has a "Before=NetworkManager.service" dependency, that's why it only breaks tests making use of network-manager (boot-and- services & boot-smoke). The LXD/LXC stack seems to be somehow involved in creating that file: https://github.com/lxc/lxc-ci/pull/266 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1931088 Title: boot-and-services tests fails in impish on armhf (248.3) Status in Auto Package Testing: Incomplete Status in dnsmasq package in Ubuntu: Incomplete Status in gdm3 package in Ubuntu: Incomplete Status in glibc package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Incomplete Status in systemd package in Ubuntu: Confirmed Status in dnsmasq source package in Impish: Incomplete Status in gdm3 source package in Impish: Incomplete Status in glibc source package in Impish: Incomplete Status in isc-dhcp source package in Impish: Incomplete Status in systemd source package in Impish: Confirmed Bug description: Systemd 248.3-1ubuntu1 is rather new, but had 5 successful tests on armhf before now slipping into a bad mode. Now it seems all tests failed in boot-and-services by hanging until killed by VirtSubproc.Timeout of autokgtest. The last [1] test log has a bit more, it shows a python stack overflow ``` VirtSubproc.Timeout Fatal Python error: Cannot recover from stack overflow. Python runtime state: initialized Current thread 0x00007f108e840740 (most recent call first): File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 36 in log File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 86 in debug File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 472 in send File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 521 in command ``` I have seen a bunch of packages including even gdm3 and glibc being blocked by that so I wanted to at least track down the issue until we can put it on someones task list to resolve. No one replied to my pings yet, but maybe that means someone is already debugging this and had enabled some debugging? By running the same in armhf container on arm64 VM on canonistack I've seen no issues. The test worked fine and had no hang/issues. root@systemd-test-fail:~/systemd-248.3# ./debian/tests/boot-and-services lxc 1 test_profile (__main__.AppArmorTest) AppArmor confined unit ... skipped 'fails on armhf testbeds, see LP: #1842352' test_help (__main__.CLITest) --help works and succeeds ... ok test_invalid_option (__main__.CLITest) Calling with invalid option fails ... ok test_version (__main__.CLITest) --version works and succeeds ... ok test_cpushares (__main__.CgroupsTest) service with CPUShares ... ok test_simple (__main__.CgroupsTest) simple service ... ok test_bash_crash (__main__.CoredumpTest) ... skipped 'systemd-coredump does not work in containers' test_log_for_service (__main__.JournalTest) ... ok test_no_options (__main__.JournalTest) ... ok test_boot (__main__.NspawnTest) ... skipped 'nspawn does not work in most containers' test_service (__main__.NspawnTest) ... skipped 'nspawn does not work in most containers' test_failing (__main__.SeccompTest) ... ok test_0_init (__main__.ServicesTest) Verify that init is systemd ... ok test_cron (__main__.ServicesTest) ... ok test_dbus (__main__.ServicesTest) ... ok test_gdm3 (__main__.ServicesTest) ... skipped 'gdm3 not found' test_logind (__main__.ServicesTest) ... ok test_network_manager (__main__.ServicesTest) ... ok test_no_failed (__main__.ServicesTest) No failed units ... ok test_rsyslog (__main__.ServicesTest) ... ok test_tmp_cleanup (__main__.ServicesTest) ... ok test_tmp_mount (__main__.ServicesTest) ... ok test_udev (__main__.ServicesTest) ... skipped 'udev does not work in containers' ---------------------------------------------------------------------- Ran 23 tests in 5.589s OK (skipped=6) So I can not reproduce this on canonistack, but it blocks autopkgtests of various packages pretty reproducibly :-/ [1]: https://autopkgtest.ubuntu.com/results/autopkgtest- impish/impish/armhf/s/systemd/20210604_081326_d4319@/log.gz To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1931088/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
