Public bug reported:
Complete system lockup on kernel 7.0.0-28.28~24.04.1 (HWE edge on Ubuntu
24.04, Dell laptop). The machine degraded over ~80 minutes and then froze
entirely: blank screen, power and fans on, no response to keyboard, no SSH.
Required a power-cycle.
What happened
-------------
At 18:59:42 the hung-task detector reported ~10 kworkers, all on the
"events" workqueue, all in D-state with the same stack:
Workqueue: events bpf_prog_free_deferred
Call Trace:
__schedule+0x2d0/0x660
schedule+0x27/0x90
schedule_preempt_disabled+0x15/0x30
__mutex_lock.constprop.0+0x4d3/0x9d0
__mutex_lock_slowpath+0x13/0x20
mutex_lock+0x3b/0x50
bpf_prog_pack_free+0x24/0x1e0
bpf_jit_binary_pack_free+0x18/0xc0
bpf_jit_free+0x65/0x190
bpf_prog_free_deferred+0x1c0/0x1f0
process_one_work+0x1af/0x430
worker_thread+0x1bf/0x350
Every report ends with:
INFO: task kworker/N:M is blocked on a mutex likely owned by task
kworker/0:687:1519573.
The owner (kworker/0:687) never released the mutex and its stack was not
dumped. With all "events" kworkers on every CPU stuck behind it, the rest
of the system starved on the shared workqueue:
- 19:00:00 - systemd (PID 1) logged its last message (starting a unit) and
never logged again.
- From 19:00:25 - every D-Bus service activation timed out;
systemd-resolved stopped answering (all DNS lookups failed with i/o
timeout on 127.0.0.53).
- 20:06 - gnome-shell "Failed to make thread 'KMS thread' normally
scheduled: Timeout was reached"; USB device errors.
- 20:23:16 - journald and all monitoring (atop, sar) stopped; total freeze
until power-cycle the next morning.
Memory was healthy throughout (atop shows 5.7 GB free RAM, swap steady at
15.5 GB of 24 GB) - this was not OOM or memory thrash.
Likely trigger
--------------
Heavy BPF program load/free churn. About 2 minutes before the wedge, a
Docker container that runs Calico eBPF dataplane tests (bulk-loading and
freeing many JITed BPF programs) exited. The SentinelOne eBPF agent was
also running. The deadlock began during the burst of deferred
bpf_prog_free work after container teardown.
Additional context
------------------
- Kernel tainted P OE (ZFS out-of-tree module loaded). ZFS was a victim of
the hang (a sanoid snapshot unit was starting when PID 1 wedged), not
obviously involved in the BPF paths above.
- Full hung-task dump is attached to this report as HungTaskTraces
(journalctl -k, 18:59:30-19:01).
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-7.0.0-28-generic 7.0.0-28.28~24.04.1
ProcVersionSignature: Ubuntu 7.0.0-28.28~24.04.1-generic 7.0.12
Uname: Linux 7.0.0-28-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia zfs
ApportVersion: 2.28.3-0ubuntu0.1
Architecture: amd64
CRDA: N/A
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127:
pkexec must be setuid root
Date: Fri Aug 14 10:41:16 2026
InstallationDate: Installed on 2025-02-13 (547 days ago)
InstallationMedia: Ubuntu 24.04.1 LTS "Noble Numbat" - Release amd64
(20240827.1)
IwConfig: lo no wireless extensions.
MachineType: Dell Inc. Precision 5530
PackageArchitecture: amd64
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_pca6v7@/vmlinuz-7.0.0-28-generic
root=ZFS=rpool/ROOT/ubuntu_pca6v7 ro quiet splash mem_sleep_default=deep
vt.handoff=1
RelatedPackageVersions:
linux-restricted-modules-7.0.0-28-generic N/A
linux-backports-modules-7.0.0-28-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.27
RfKill: Error: command ['rfkill', 'list'] failed with exit code 1: rfkill:
cannot open /dev/rfkill: No such file or directory
SourcePackage: linux-hwe-7.0
UpgradeStatus: No upgrade log present (probably fresh install)
_MarkForUpload: True
acpidump: Error: command ['pkexec', '/usr/share/apport/dump_acpi_tables.py']
failed with exit code 127: pkexec must be setuid root
dmi.bios.date: 12/11/2024
dmi.bios.release: 1.42
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.42.0
dmi.board.name: 0FP2W2
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 10
dmi.chassis.vendor: Dell Inc.
dmi.modalias:
dmi:bvnDellInc.:bvr1.42.0:bd12/11/2024:br1.42:svnDellInc.:pnPrecision5530:pvr:rvnDellInc.:rn0FP2W2:rvrA00:cvnDellInc.:ct10:cvr:sku087D:pfaPrecision:
dmi.product.family: Precision
dmi.product.name: Precision 5530
dmi.product.sku: 087D
dmi.sys.vendor: Dell Inc.
** Affects: linux-hwe-7.0 (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug noble wayland-session
** Description changed:
- While running a series of tests that spin docker containers up and down
- and attach/detach BPF programs, the system wedged completely
+ Complete system lockup on kernel 7.0.0-28.28~24.04.1 (HWE edge on Ubuntu
+ 24.04, Dell laptop). The machine degraded over ~80 minutes and then froze
+ entirely: blank screen, power and fans on, no response to keyboard, no SSH.
+ Required a power-cycle.
+
+ What happened
+ -------------
+ At 18:59:42 the hung-task detector reported ~10 kworkers, all on the
+ "events" workqueue, all in D-state with the same stack:
+
+ Workqueue: events bpf_prog_free_deferred
+ Call Trace:
+ __schedule+0x2d0/0x660
+ schedule+0x27/0x90
+ schedule_preempt_disabled+0x15/0x30
+ __mutex_lock.constprop.0+0x4d3/0x9d0
+ __mutex_lock_slowpath+0x13/0x20
+ mutex_lock+0x3b/0x50
+ bpf_prog_pack_free+0x24/0x1e0
+ bpf_jit_binary_pack_free+0x18/0xc0
+ bpf_jit_free+0x65/0x190
+ bpf_prog_free_deferred+0x1c0/0x1f0
+ process_one_work+0x1af/0x430
+ worker_thread+0x1bf/0x350
+
+ Every report ends with:
+
+ INFO: task kworker/N:M is blocked on a mutex likely owned by task
+ kworker/0:687:1519573.
+
+ The owner (kworker/0:687) never released the mutex and its stack was not
+ dumped. With all "events" kworkers on every CPU stuck behind it, the rest
+ of the system starved on the shared workqueue:
+
+ - 19:00:00 - systemd (PID 1) logged its last message (starting a unit) and
+ never logged again.
+ - From 19:00:25 - every D-Bus service activation timed out;
+ systemd-resolved stopped answering (all DNS lookups failed with i/o
+ timeout on 127.0.0.53).
+ - 20:06 - gnome-shell "Failed to make thread 'KMS thread' normally
+ scheduled: Timeout was reached"; USB device errors.
+ - 20:23:16 - journald and all monitoring (atop, sar) stopped; total freeze
+ until power-cycle the next morning.
+
+ Memory was healthy throughout (atop shows 5.7 GB free RAM, swap steady at
+ 15.5 GB of 24 GB) - this was not OOM or memory thrash.
+
+ Likely trigger
+ --------------
+ Heavy BPF program load/free churn. About 2 minutes before the wedge, a
+ Docker container that runs Calico eBPF dataplane tests (bulk-loading and
+ freeing many JITed BPF programs) exited. The SentinelOne eBPF agent was
+ also running. The deadlock began during the burst of deferred
+ bpf_prog_free work after container teardown.
+
+ Additional context
+ ------------------
+ - Kernel tainted P OE (ZFS out-of-tree module loaded). ZFS was a victim of
+ the hang (a sanoid snapshot unit was starting when PID 1 wedged), not
+ obviously involved in the BPF paths above.
+ - Full hung-task dump is attached to this report as HungTaskTraces
+ (journalctl -k, 18:59:30-19:01).
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-7.0.0-28-generic 7.0.0-28.28~24.04.1
ProcVersionSignature: Ubuntu 7.0.0-28.28~24.04.1-generic 7.0.12
Uname: Linux 7.0.0-28-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia zfs
ApportVersion: 2.28.3-0ubuntu0.1
Architecture: amd64
CRDA: N/A
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127:
pkexec must be setuid root
Date: Fri Aug 14 10:41:16 2026
InstallationDate: Installed on 2025-02-13 (547 days ago)
InstallationMedia: Ubuntu 24.04.1 LTS "Noble Numbat" - Release amd64
(20240827.1)
IwConfig: lo no wireless extensions.
MachineType: Dell Inc. Precision 5530
PackageArchitecture: amd64
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_pca6v7@/vmlinuz-7.0.0-28-generic
root=ZFS=rpool/ROOT/ubuntu_pca6v7 ro quiet splash mem_sleep_default=deep
vt.handoff=1
RelatedPackageVersions:
- linux-restricted-modules-7.0.0-28-generic N/A
- linux-backports-modules-7.0.0-28-generic N/A
- linux-firmware 20240318.git3b128b60-0ubuntu2.27
+ linux-restricted-modules-7.0.0-28-generic N/A
+ linux-backports-modules-7.0.0-28-generic N/A
+ linux-firmware 20240318.git3b128b60-0ubuntu2.27
RfKill: Error: command ['rfkill', 'list'] failed with exit code 1: rfkill:
cannot open /dev/rfkill: No such file or directory
SourcePackage: linux-hwe-7.0
UpgradeStatus: No upgrade log present (probably fresh install)
_MarkForUpload: True
acpidump: Error: command ['pkexec', '/usr/share/apport/dump_acpi_tables.py']
failed with exit code 127: pkexec must be setuid root
dmi.bios.date: 12/11/2024
dmi.bios.release: 1.42
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.42.0
dmi.board.name: 0FP2W2
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 10
dmi.chassis.vendor: Dell Inc.
dmi.modalias:
dmi:bvnDellInc.:bvr1.42.0:bd12/11/2024:br1.42:svnDellInc.:pnPrecision5530:pvr:rvnDellInc.:rn0FP2W2:rvrA00:cvnDellInc.:ct10:cvr:sku087D:pfaPrecision:
dmi.product.family: Precision
dmi.product.name: Precision 5530
dmi.product.sku: 087D
dmi.sys.vendor: Dell Inc.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163477
Title:
System-wide hang: bpf_prog_pack_free deadlocks on prog pack mutex
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-hwe-7.0/+bug/2163477/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
