Reproducer for bug #2164889 ** Description changed:
+ [ Impact ] + + On systems where systemd-networkd manages interfaces with nested traffic control state, networkd can crash during teardown of qdisc/tclass objects. + The failure happens when networkd tears down tc state while rebooting or restarting the link. + + Systemd version 255 recursively deletes qdiscs and tclasses while iterating the live link sets. With sufficiently nested tc state, deleting one object + can mutate the same set that is still being walked, causing re-entrant deletion and eventually stack overflow or a segfault-like crash. + For affected users this can make node reboot or network reconfiguration unreliable, because systemd-networkd may crash while processing link cleanup. + + Release: Ubuntu Noble 24.04, systemd 255.4-1ubuntu8.17 + + Fixed in systemd 256 and later + + [ Test Plan ] + + Launch a Noble VM with lxd: lxc launch ubuntu:24.04 tc-crash-test --vm + -c limits.cpu=4 -c limits.memory=4GiB + + Configure default interface with 4 queues: lxc exec tc-crash-test -- + ethtool -L enp5s0 combined 4 + + Download reproducer script for bug #2164889 (attached in comments) + + Upload repro script to VM: lxc file push repro-systemd-networkd-tc- + stack-overflow.sh tc-crash-test/root/ + + Shell into VM: lxc shell tc-crash-test + + Enable execute permissions: chmod +x repro-systemd-networkd-tc-stack- + overflow.sh + + Run with ALLOW_DEFAULT_ROUTE: sudo ALLOW_DEFAULT_ROUTE=1 ./repro- + systemd-networkd-tc-stack-overflow.sh enp5s0 + + Output: REPRODUCED=yes ITERATION=1 CRASH_SIGNATURE=journal-segv + + [ Where problems could occur ] + + The change affects systemd-networkd's traffic control teardown path, specifically qdisc and tclass removal in src/network/tc/qdisc.c and src/network/tc/tclass.c. + If the marking or sweep logic were wrong, networkd could either fail to remove some qdiscs/tclasses or remove more tc state than intended. + + A regression could show up as stale qdisc or class state remaining on an interface after networkd restart, link reconfiguration, or reboot. + Testers should check `tc qdisc show` and `tc class show` before and after networkd reconfiguration on systems using networkd-managed tc settings. + A regression could also show up as traffic queueing or class-based policy not being restored correctly after networkd reload/restart. + Systems using HTB, DRR, QFQ, or nested qdisc/class configurations are the most relevant regression targets. + + If the drop ordering were incorrect, networkd could log failed rtnetlink operations while deleting or recreating tc objects, or it could keep internal + qdisc/tclass state that no longer matches kernel state. In that case, symptoms would likely include errors in `journalctl -u systemd-networkd`, + failed networkd reconfiguration, or unexpected tc state on the affected link. + + [ Other Info ] + + This was also reproduced on a Google Cloud Kubernetes node running Noble through the following steps: + + Spin up a GKE node (Google Cloud access required): + + gcloud container clusters create <your-cluster-name> --zone=<choose-AZ> --enable-dataplane-v2 \ + --enable-ip-alias --cluster-version=<your-chosen-image-serial> --machine-type=n2-highmem-64 \ + --image-type=UBUNTU_CONTAINERD --disk-size=2000 --num-nodes=1 --no-enable-autoupgrade + + List node info: gcloud compute instances list + + This will output a table containing the EXTERNAL_IP value of the test + node, which will be used for ssh connection. + + Add ssh keys to node: gcloud compute instances add-metadata NODE_VM_NAME \ + --zone <zone> \ + --metadata="ssh-keys=YOUR_USERNAME:$(cat ~/.ssh/<your-key>.pub)" + + ssh onto node: ssh user@<ip.address> + + Check journalctl logs for systemd-networkd segfault: + + sudo journalctl | grep -i segfault + + Example output: Sep 17 13:09:25 gke-systemdtest-13700-default-pool- + bce9f8c7-cwgn kernel: systemd-network[1103]: segfault at 7ffe6e2ddff8 ip + 00007a2fbddbc473 sp 00007ffe6e2de000 error 6 in libsystemd- + shared-255.so[7a2fbdc6a000+248000] likely on CPU 45 (core 13, socket 0) + + This can be reproduced repeatedly by rebooting the node and checking + journalctl logs for the new segfault. + + Bug: https://github.com/systemd/systemd/issues/32247 + Fix patch 1: https://github.com/systemd/systemd/commit/632d321050f58fe1b5bed7cfe769d212377c0301 + Fix patch 2: https://github.com/systemd/systemd/commit/ee8f605ded4fea6b93aae018415efae877c26ed2 + + [ Original Bug Description ] + [545051.196840] systemd-network[1395597]: segfault at 7ffd0b106ff8 ip 0000732937dbbb9c sp 00007ffd0b107000 error 6 in libsystemd-shared-255.so[1bbb9c,732937c6a000+247000] likely on CPU 15 (core 15, socket 0) [545051.196849] Code: 49 01 c1 41 f6 c1 07 75 7d 49 8b 01 49 89 42 08 48 83 c4 08 89 d0 5b 41 5c 41 5d 5d c3 90 83 e1 04 0f 84 97 00 00 00 8b 77 28 <e8> af e5 ff ff 41 89 02 89 c2 41 89 41 28 83 fa ff 74 31 41 0f b6 [270850.754339] systemd-network[700716]: segfault at 7ffd54bfbff8 ip 0000717ac19bb897 sp 00007ffd54bfc000 error 6 in libsystemd-shared-255.so[1bb897,717ac186a000+247000] likely on CPU 10 (core 10, socket 0) [270850.754347] Code: 00 00 b8 f4 ff ff ff e9 45 ff ff ff 0f 1f 80 00 00 00 00 49 89 f2 48 85 ff 0f 84 54 04 00 00 55 49 89 f9 48 89 e5 41 55 41 54 <53> 48 83 ec 08 0f b6 4f 2f 89 c8 83 e0 03 3c 01 0f 84 13 01 00 00 [292219.943345] systemd-network[755659]: segfault at 7fff00f4eff8 ip 00007442d3fc4118 sp 00007fff00f4f000 error 6 in libsystemd-shared-255.so[1c4118,7442d3e6a000+247000] likely on CPU 2 (core 2, socket 0) [292219.943350] Code: 35 53 eb 0f 00 48 8d 3d 36 d2 12 00 e8 41 64 00 00 e8 3c a2 ea ff e8 17 fd ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 48 89 e5 <41> 55 49 89 d5 41 54 49 89 cc 53 48 89 fb 48 83 ec 08 e8 51 77 ff [296437.649671] systemd-network[766382]: segfault at 7ffc7a3a5ff8 ip 00007d1ca4dc4122 sp 00007ffc7a3a6000 error 6 in libsystemd-shared-255.so[1c4122,7d1ca4c6a000+247000] likely on CPU 5 (core 5, socket 0) [296437.649677] Code: 12 00 e8 41 64 00 00 e8 3c a2 ea ff e8 17 fd ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 48 89 e5 41 55 49 89 d5 41 54 49 89 cc <53> 48 89 fb 48 83 ec 08 e8 51 77 ff ff 83 f8 ff 0f 84 88 00 00 00 More and more segfaults. Please update systemd package in 24.04 LTS. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: systemd 255.4-1ubuntu8.4 [modified: usr/lib/systemd/system/rc-local.service] Uname: Linux 6.11.3-x64v4-xanmod2 x86_64 ApportVersion: 2.28.1-0ubuntu3.1 Architecture: amd64 CasperMD5CheckResult: pass CloudArchitecture: x86_64 CloudID: none CloudName: none CloudPlatform: none CloudSubPlatform: config Date: Tue Oct 22 06:40:50 2024 InstallationDate: Installed on 2024-10-08 (14 days ago) InstallationMedia: Ubuntu-Server 22.04.5 LTS "Jammy Jellyfish" - Release amd64 (20240911.4) MachineType: Gigabyte Technology Co., Ltd. X670 GAMING X AX V2 ProcEnviron: - LANG=C.UTF-8 - PATH=(custom, no user) - SHELL=/bin/bash - TERM=xterm-256color - XDG_RUNTIME_DIR=<set> + LANG=C.UTF-8 + PATH=(custom, no user) + SHELL=/bin/bash + TERM=xterm-256color + XDG_RUNTIME_DIR=<set> ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.11.3-x64v4-xanmod2 root=/dev/mapper/ubuntu--vg-ubuntu--lv ro mitigations=off nosmt amd_pstate=disable RebootRequiredPkgs: Error: path contained symlinks. SourcePackage: systemd UpgradeStatus: Upgraded to noble on 2024-10-08 (14 days ago) dmi.bios.date: 05/22/2024 dmi.bios.release: 5.34 dmi.bios.vendor: American Megatrends International, LLC. dmi.bios.version: F30 dmi.board.asset.tag: Default string dmi.board.name: X670 GAMING X AX V2 dmi.board.vendor: Gigabyte Technology Co., Ltd. dmi.board.version: x.x dmi.chassis.asset.tag: Default string dmi.chassis.type: 3 dmi.chassis.vendor: Default string dmi.chassis.version: Default string dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrF30:bd05/22/2024:br5.34:svnGigabyteTechnologyCo.,Ltd.:pnX670GAMINGXAXV2:pvrDefaultstring:rvnGigabyteTechnologyCo.,Ltd.:rnX670GAMINGXAXV2:rvrx.x:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring: dmi.product.family: X670 MB dmi.product.name: X670 GAMING X AX V2 dmi.product.sku: Default string dmi.product.version: Default string dmi.sys.vendor: Gigabyte Technology Co., Ltd. modified.conffile..etc.init.d.apport: [modified] mtime.conffile..etc.init.d.apport: 2024-07-22T16:59:07 ** Attachment added: "repro-systemd-networkd-tc-stack-overflow.sh" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2085314/+attachment/6002095/+files/repro-systemd-networkd-tc-stack-overflow.sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2085314 Title: systemd-networkd segfault (question #819069) To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/2085314/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
