** Description changed: [ Impact ] On systems where systemd-networkd manages interfaces with nested traffic control state, networkd can crash during teardown of qdisc/tclass objects. The failure happens when networkd tears down tc state while rebooting or restarting the link. Systemd version 255 recursively deletes qdiscs and tclasses while iterating the live link sets. With sufficiently nested tc state, deleting one object can mutate the same set that is still being walked, causing re-entrant deletion and eventually stack overflow or a segfault-like crash. For affected users this can make node reboot or network reconfiguration unreliable, because systemd-networkd may crash while processing link cleanup. Release: Ubuntu Noble 24.04, systemd 255.4-1ubuntu8.17 Fixed in systemd 256 and later - + [ Test Plan ] Launch a Noble VM with lxd: lxc launch ubuntu:24.04 tc-crash-test --vm -c limits.cpu=4 -c limits.memory=4GiB Configure default interface with 4 queues: lxc exec tc-crash-test -- ethtool -L enp5s0 combined 4 - Download reproducer script for bug #2164889 (attached in comments) + Download reproducer script for bug #2164889: + https://launchpadlibrarian.net/878024644/repro-systemd-networkd-tc- + stack-overflow.sh Upload repro script to VM: lxc file push repro-systemd-networkd-tc- stack-overflow.sh tc-crash-test/root/ Shell into VM: lxc shell tc-crash-test Enable execute permissions: chmod +x repro-systemd-networkd-tc-stack- overflow.sh Run with ALLOW_DEFAULT_ROUTE: sudo ALLOW_DEFAULT_ROUTE=1 ./repro- systemd-networkd-tc-stack-overflow.sh enp5s0 Output: REPRODUCED=yes ITERATION=1 CRASH_SIGNATURE=journal-segv [ Where problems could occur ] The change affects systemd-networkd's traffic control teardown path, specifically qdisc and tclass removal in src/network/tc/qdisc.c and src/network/tc/tclass.c. If the marking or sweep logic were wrong, networkd could either fail to remove some qdiscs/tclasses or remove more tc state than intended. - A regression could show up as stale qdisc or class state remaining on an interface after networkd restart, link reconfiguration, or reboot. + A regression could show up as stale qdisc or class state remaining on an interface after networkd restart, link reconfiguration, or reboot. Testers should check `tc qdisc show` and `tc class show` before and after networkd reconfiguration on systems using networkd-managed tc settings. A regression could also show up as traffic queueing or class-based policy not being restored correctly after networkd reload/restart. Systems using HTB, DRR, QFQ, or nested qdisc/class configurations are the most relevant regression targets. If the drop ordering were incorrect, networkd could log failed rtnetlink operations while deleting or recreating tc objects, or it could keep internal qdisc/tclass state that no longer matches kernel state. In that case, symptoms would likely include errors in `journalctl -u systemd-networkd`, failed networkd reconfiguration, or unexpected tc state on the affected link. [ Other Info ] - This was also reproduced on a Google Cloud Kubernetes node running Noble through the following steps: - + This was also reproduced on a Google Cloud Kubernetes node running Noble + through the following steps: + Spin up a GKE node (Google Cloud access required): gcloud container clusters create <your-cluster-name> --zone=<choose-AZ> --enable-dataplane-v2 \ - --enable-ip-alias --cluster-version=<your-chosen-image-serial> --machine-type=n2-highmem-64 \ - --image-type=UBUNTU_CONTAINERD --disk-size=2000 --num-nodes=1 --no-enable-autoupgrade - + --enable-ip-alias --cluster-version=<your-chosen-image-serial> --machine-type=n2-highmem-64 \ + --image-type=UBUNTU_CONTAINERD --disk-size=2000 --num-nodes=1 --no-enable-autoupgrade + List node info: gcloud compute instances list This will output a table containing the EXTERNAL_IP value of the test node, which will be used for ssh connection. Add ssh keys to node: gcloud compute instances add-metadata NODE_VM_NAME \ - --zone <zone> \ - --metadata="ssh-keys=YOUR_USERNAME:$(cat ~/.ssh/<your-key>.pub)" + --zone <zone> \ + --metadata="ssh-keys=YOUR_USERNAME:$(cat ~/.ssh/<your-key>.pub)" ssh onto node: ssh user@<ip.address> Check journalctl logs for systemd-networkd segfault: sudo journalctl | grep -i segfault Example output: Sep 17 13:09:25 gke-systemdtest-13700-default-pool- bce9f8c7-cwgn kernel: systemd-network[1103]: segfault at 7ffe6e2ddff8 ip 00007a2fbddbc473 sp 00007ffe6e2de000 error 6 in libsystemd- shared-255.so[7a2fbdc6a000+248000] likely on CPU 45 (core 13, socket 0) This can be reproduced repeatedly by rebooting the node and checking journalctl logs for the new segfault. Bug: https://github.com/systemd/systemd/issues/32247 Fix patch 1: https://github.com/systemd/systemd/commit/632d321050f58fe1b5bed7cfe769d212377c0301 Fix patch 2: https://github.com/systemd/systemd/commit/ee8f605ded4fea6b93aae018415efae877c26ed2 [ Original Bug Description ] [545051.196840] systemd-network[1395597]: segfault at 7ffd0b106ff8 ip 0000732937dbbb9c sp 00007ffd0b107000 error 6 in libsystemd-shared-255.so[1bbb9c,732937c6a000+247000] likely on CPU 15 (core 15, socket 0) [545051.196849] Code: 49 01 c1 41 f6 c1 07 75 7d 49 8b 01 49 89 42 08 48 83 c4 08 89 d0 5b 41 5c 41 5d 5d c3 90 83 e1 04 0f 84 97 00 00 00 8b 77 28 <e8> af e5 ff ff 41 89 02 89 c2 41 89 41 28 83 fa ff 74 31 41 0f b6 [270850.754339] systemd-network[700716]: segfault at 7ffd54bfbff8 ip 0000717ac19bb897 sp 00007ffd54bfc000 error 6 in libsystemd-shared-255.so[1bb897,717ac186a000+247000] likely on CPU 10 (core 10, socket 0) [270850.754347] Code: 00 00 b8 f4 ff ff ff e9 45 ff ff ff 0f 1f 80 00 00 00 00 49 89 f2 48 85 ff 0f 84 54 04 00 00 55 49 89 f9 48 89 e5 41 55 41 54 <53> 48 83 ec 08 0f b6 4f 2f 89 c8 83 e0 03 3c 01 0f 84 13 01 00 00 [292219.943345] systemd-network[755659]: segfault at 7fff00f4eff8 ip 00007442d3fc4118 sp 00007fff00f4f000 error 6 in libsystemd-shared-255.so[1c4118,7442d3e6a000+247000] likely on CPU 2 (core 2, socket 0) [292219.943350] Code: 35 53 eb 0f 00 48 8d 3d 36 d2 12 00 e8 41 64 00 00 e8 3c a2 ea ff e8 17 fd ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 48 89 e5 <41> 55 49 89 d5 41 54 49 89 cc 53 48 89 fb 48 83 ec 08 e8 51 77 ff [296437.649671] systemd-network[766382]: segfault at 7ffc7a3a5ff8 ip 00007d1ca4dc4122 sp 00007ffc7a3a6000 error 6 in libsystemd-shared-255.so[1c4122,7d1ca4c6a000+247000] likely on CPU 5 (core 5, socket 0) [296437.649677] Code: 12 00 e8 41 64 00 00 e8 3c a2 ea ff e8 17 fd ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 48 89 e5 41 55 49 89 d5 41 54 49 89 cc <53> 48 89 fb 48 83 ec 08 e8 51 77 ff ff 83 f8 ff 0f 84 88 00 00 00 More and more segfaults. Please update systemd package in 24.04 LTS. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: systemd 255.4-1ubuntu8.4 [modified: usr/lib/systemd/system/rc-local.service] Uname: Linux 6.11.3-x64v4-xanmod2 x86_64 ApportVersion: 2.28.1-0ubuntu3.1 Architecture: amd64 CasperMD5CheckResult: pass CloudArchitecture: x86_64 CloudID: none CloudName: none CloudPlatform: none CloudSubPlatform: config Date: Tue Oct 22 06:40:50 2024 InstallationDate: Installed on 2024-10-08 (14 days ago) InstallationMedia: Ubuntu-Server 22.04.5 LTS "Jammy Jellyfish" - Release amd64 (20240911.4) MachineType: Gigabyte Technology Co., Ltd. X670 GAMING X AX V2 ProcEnviron: LANG=C.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR=<set> ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.11.3-x64v4-xanmod2 root=/dev/mapper/ubuntu--vg-ubuntu--lv ro mitigations=off nosmt amd_pstate=disable RebootRequiredPkgs: Error: path contained symlinks. SourcePackage: systemd UpgradeStatus: Upgraded to noble on 2024-10-08 (14 days ago) dmi.bios.date: 05/22/2024 dmi.bios.release: 5.34 dmi.bios.vendor: American Megatrends International, LLC. dmi.bios.version: F30 dmi.board.asset.tag: Default string dmi.board.name: X670 GAMING X AX V2 dmi.board.vendor: Gigabyte Technology Co., Ltd. dmi.board.version: x.x dmi.chassis.asset.tag: Default string dmi.chassis.type: 3 dmi.chassis.vendor: Default string dmi.chassis.version: Default string dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrF30:bd05/22/2024:br5.34:svnGigabyteTechnologyCo.,Ltd.:pnX670GAMINGXAXV2:pvrDefaultstring:rvnGigabyteTechnologyCo.,Ltd.:rnX670GAMINGXAXV2:rvrx.x:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring: dmi.product.family: X670 MB dmi.product.name: X670 GAMING X AX V2 dmi.product.sku: Default string dmi.product.version: Default string dmi.sys.vendor: Gigabyte Technology Co., Ltd. modified.conffile..etc.init.d.apport: [modified] mtime.conffile..etc.init.d.apport: 2024-07-22T16:59:07
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2085314 Title: systemd-networkd segfault (question #819069) To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/2085314/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
