I asked Claude to look for potential root cause, in case it's helpful:
Some upstream archaeology that narrows this down.
The blocked tasks are all waiters on the BPF prog pack mutex
(kernel/bpf/core.c, pack_mutex). The owner (kworker/0:687, stack never
dumped) would have been executing the free path under that mutex:
bpf_prog_pack_free() -> bpf_arch_text_invalidate() (x86 text_poke_set)
and, when a pack empties, bpf_jit_free_exec() -> execmem_free() -> vfree()
of VM_FLUSH_RESET_PERMS memory. Both legs walk init_mm page tables via the
x86 CPA (change page attributes) code and issue cross-CPU TLB flushes. If
the owner wedges in there, every subsequent BPF program free piles up
behind pack_mutex on the "events" workqueue, which matches the observed
system-wide starvation (PID 1, D-Bus, resolved, KMS all stalling).
Upstream has a known, recently-fixed use-after-free in exactly this
machinery: vmap huge-page promotion frees lower page tables with no
locking against concurrent kernel page-table walkers. Fixed in v7.2-rc7 by
commit 26444eb71465 ("mm/vmalloc: acquire init_mm lock on huge vmap to
avoid ptdump UAF"), Cc: stable, Fixes: b6bdb7517c3d — first of a 3-patch
series ("mm: fix UAF caused by race between ptdump and vmap pgtable
freeing", also 27c32e553834 and 9d3277b2c07c). The cover letter for that
series additionally describes a still-unmerged race on the x86 CPA side:
CPA collapse of ranges into huge leaf entries can free a page table that a
concurrent CPA/page-table walk is accessing.
Two things make this kernel (7.0) more exposed than 6.17, where the same
workload ran 45+ days clean:
- v7.0 includes commit 4afb85f5e359 ("mm: remove PMD alignment constraint
in execmem_vmalloc()"), changing how the 2 MB BPF prog packs are
allocated in vmalloc space and hence when huge-vmap promotion/page-table
freeing kicks in for them.
- The workload here (Calico eBPF test suites in containers, plus an eBPF
security agent) loads and frees large numbers of JITed BPF programs
concurrently from multiple CPUs — i.e. constant parallel CPA activity
(set_memory_rox on pack alloc, reset-perms on pack free) interleaved
with vmalloc alloc/free of the packs themselves.
A page-table UAF hit by the CPA/text_poke walker while it holds pack_mutex
would present exactly as observed: the owner never returns, its stack is
never dumped (it is spinning/faulted, not in D state), and all other
kworkers queue up behind the mutex.
Suggested action: consider picking up the 26444eb71465 series (it is
tagged for stable) for the 7.0 HWE kernel.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163477
Title:
System-wide hang: bpf_prog_pack_free deadlocks on prog pack mutex
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-hwe-7.0/+bug/2163477/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
