Results of the split test from #50. The request to the kernel team follows in the next comment.
SETUP ----- Three identical nodes. All facts below come from the nodes themselves. Board: Raspberry Pi 5 Model B Rev 1.1 (revision d04171), 8 GB Kernel: Ubuntu 26.04 LTS, linux-raspi 7.0.0-1017.17 Ethernet: Cadence GEM rev 0x00070109 (macb), PHY Broadcom BCM54213PE RP1 fw: 23fca1cd776798b50bd3e640ecb242215778e7f8 Bootloader: 2025-12-08 Rings: RX 512 / TX 512 (defaults) swarm01: 2c:cf:67:dc:49:5a swarm02: 88:a2:9e:b2:f1:ac swarm03: 2c:cf:67:dc:4a:ec The RP1 firmware is newer than the one in the first report. The stall is the same. RESULT 1: A CONSTANT MAXIMUM CLOCK DOES NOT STOP THE STALL ---------------------------------------------------------- swarm01 ran with governor "performance" (constant 2400 MHz) and default offloads for 74.2 hours. It stalled 2 times: 2026-09-03 12:06 CEST and 2026-09-05 17:15 CEST. The baseline for this node is about 1 stall per 11.5 hours, so about 6.4 stalls were expected. Both stalls had the usual signature. The tx_packets counter froze with the carrier up. The qdisc backlog went past qlen (1026 and 877 packets). Every MAC error counter stayed at zero. The clock stayed at 2400 MHz. The CPU stayed at 50 C. So the DVFS idea from #50 is not a fix. This agrees with #2 and #3, which I had missed: "performance" was tried in December 2025 and the network still died. The count is lower than expected, so a partial effect is possible. But 2 stalls in 3 days is not a workaround. RESULT 2: "TSO OFF SG OFF" STOPS THE STALL ------------------------------------------ swarm02 and swarm03 ran with "ethtool -K eth0 tso off sg off" and the governor left on "ondemand". They had 0 stalls in 220 node-hours since 2026-09-02 19:09 UTC. On 2026-09-05 21:19 UTC I moved swarm01 to the same setting. It had 0 stalls in the 36 hours since. Total: 0 stalls in about 256 node-hours. The measured baseline is about 1 stall per 14 node-hours (24 stalls in 344 node-hours, #48). About 18 stalls were expected. This is the same result as #34 and #45. CORRECTIONS AND ATTACHMENTS --------------------------- Two comment numbers in #50 were wrong. The 600 MHz report is #46, not #39. The governor flipping attempt is #32, not #26. Attached, as one text file: the full before and after captures of three stalls. Two are from the governor arm on swarm01. One is from swarm03 before the test started. Also attached: the script that takes the captures. The script now also runs "ethtool -d eth0" at stall time, per #51. It has not fired since I added that. AI DISCLOSURE: I have been heavily assisted by Claude Fable when diagnosing this issue ** Attachment added: "lp2133877-stall-captures-3x-before-after.txt" https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2133877/+attachment/5998327/+files/lp2133877-stall-captures-3x-before-after.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2133877 Title: Complete network hang on Raspberry Pi 5 with kernel 6.17 under load - possibly related to CPU frequency scaling To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2133877/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
