Package: cloud-init Version: 20.4.1-2+deb11u1 Severity: important X-Debbugs-Cc: xi...@australdx.fr
Dear Maintainer, firewalld and cloud-init have ordering cycles between their systemd unit files, leading to more or less broken boot results when both are installed and active, because at each boot systemd decides to skip a non-deterministically choosen service (not necessarily cloud-init or firewalld) to break the cycle. I'm not sure if any of firewalld or cloud-init is more at fault (maybe in not respecting some systemd rules?) so I'm also opening a duplicate of this bug for the other package. This can have various but potentially serious consequences, depending on what should be, but is not, started. Examples of boot traces of this issue happening: * example 1: sysinit.target: Found ordering cycle on cloud-init.service/start sysinit.target: Found dependency on networking.service/start sysinit.target: Found dependency on network-pre.target/start sysinit.target: Found dependency on firewalld.service/start sysinit.target: Found dependency on basic.target/start sysinit.target: Found dependency on sockets.target/start sysinit.target: Found dependency on uuidd.socket/start sysinit.target: Found dependency on sysinit.target/start sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start * example 2: sysinit.target: Found ordering cycle on cloud-init.service/start sysinit.target: Found dependency on networking.service/start sysinit.target: Found dependency on network-pre.target/start sysinit.target: Found dependency on firewalld.service/start sysinit.target: Found dependency on dbus.service/start sysinit.target: Found dependency on sysinit.target/start sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start * example 3: firewalld.service: Found ordering cycle on dbus.socket/start firewalld.service: Found dependency on sysinit.target/start firewalld.service: Found dependency on cloud-init.service/start firewalld.service: Found dependency on networking.service/start firewalld.service: Found dependency on network-pre.target/start firewalld.service: Found dependency on firewalld.service/start firewalld.service: Job dbus.socket/start deleted to break ordering cycle starting with firewalld.service/start * example 4: firewalld.service: Found ordering cycle on dbus.service/start firewalld.service: Found dependency on sysinit.target/start firewalld.service: Found dependency on cloud-init.service/start firewalld.service: Found dependency on networking.service/start firewalld.service: Found dependency on network-pre.target/start firewalld.service: Found dependency on firewalld.service/start firewalld.service: Job dbus.service/start deleted to break ordering cycle starting with firewalld.service/start basic.target: Found ordering cycle on sysinit.target/start basic.target: Found dependency on cloud-init.service/start basic.target: Found dependency on networking.service/start basic.target: Found dependency on network-pre.target/start basic.target: Found dependency on firewalld.service/start basic.target: Found dependency on basic.target/start basic.target: Job cloud-init.service/start deleted to break ordering cycle starting with basic.target/start * example 5: basic.target: Found ordering cycle on sockets.target/start basic.target: Found dependency on uuidd.socket/start basic.target: Found dependency on sysinit.target/start basic.target: Found dependency on cloud-init.service/start basic.target: Found dependency on networking.service/start basic.target: Found dependency on network-pre.target/start basic.target: Found dependency on firewalld.service/start basic.target: Found dependency on dbus.service/start basic.target: Found dependency on basic.target/start basic.target: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start firewalld.service: Found ordering cycle on dbus.socket/start firewalld.service: Found dependency on sysinit.target/start firewalld.service: Found dependency on cloud-init.service/start firewalld.service: Found dependency on networking.service/start firewalld.service: Found dependency on network-pre.target/start firewalld.service: Found dependency on firewalld.service/start firewalld.service: Job dbus.socket/start deleted to break ordering cycle starting with firewalld.service/start * example 6: networking.service: Found ordering cycle on network-pre.target/start networking.service: Found dependency on firewalld.service/start networking.service: Found dependency on dbus.service/start networking.service: Found dependency on basic.target/start networking.service: Found dependency on sockets.target/start networking.service: Found dependency on uuidd.socket/start networking.service: Found dependency on sysinit.target/start networking.service: Found dependency on cloud-init.service/start networking.service: Found dependency on networking.service/start networking.service: Job network-pre.target/start deleted to break ordering cycle starting with networking.service/start At first I experienced the issue on a Debian Stable (Bullseye), then I was able to reproduce the problem on an up-to-date Bookworm. Note that "systemd-analyze verify" seems to be unable to find the issue, however, the following repro (tried on Bookworm) shows a way to detect the cycles statically (plus, during a reboot you can observe it directly): $ sudo apt install firewalld cloud-init $ echo "datasource_list: [ Fallback ]" | sudo tee /etc/cloud/cloud.cfg.d/99_fallback.cfg $ sudo reboot $ wget https://raw.githubusercontent.com/jantman/misc-scripts/4560db773f463101273539e625c9b48e9f53f87f/dot_find_cycles.py $ # ^ I found that script by reading https://github.com/systemd/systemd/issues/3829 $ sudo apt install 2to3 python3-pygraphviz python3-pydotplus python3-pydot python3-graphviz python3-networkx $ 2to3 -w dot_find_cycles.py $ chmod +x dot_find_cycles.py $ sudo systemd-analyze dot --no-pager --order 2>/dev/null | python3 ./dot_find_cycles.py - | tee order-cycle_cloud-init-firewalld.txt networking.service -> network-pre.target -> firewalld.service -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.service -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> dbus.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> polkit.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service networking.service -> network-pre.target -> firewalld.service -> polkit.service -> sysinit.target -> cloud-init.service -> networking.service systemd-networkd.service -> network-pre.target -> firewalld.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service Best regards, Guillaume Knispel -- System Information: Debian Release: 11.5 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 5.10.0-19-amd64 (SMP w/64 CPU threads) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages cloud-init depends on: ii fdisk 2.36.1-8+deb11u1 ii gdisk 1.0.6-1.1 ii ifupdown 0.8.36 ii locales 2.31-13+deb11u5 ii lsb-base 11.1.0 ii lsb-release 11.1.0 ii net-tools 1.60+git20181103.0eebece-1 ii procps 2:3.3.17-5 ii python3 3.9.2-3 ii python3-configobj 5.0.6-4 ii python3-jinja2 2.11.3-1 ii python3-jsonpatch 1.25-3 ii python3-jsonschema 3.2.0-3 ii python3-oauthlib 3.1.0-2 ii python3-requests 2.25.1+dfsg-2 ii python3-yaml 5.3.1-5 ii util-linux 2.36.1-8+deb11u1 Versions of packages cloud-init recommends: ii cloud-guest-utils 0.31-2 ii eatmydata 105-9 ii sudo 1.9.5p2-3 Versions of packages cloud-init suggests: pn btrfs-progs <none> ii e2fsprogs 1.46.2-2 pn xfsprogs <none> -- no debconf information
