Public bug reported:
[Expected]
Page-cache reclaim runs without faulting, regardless of the upper MMIO
region splitting System RAM.
[What's happened]
Three of our Azure VMs (`Standard_L16as_v4` with the upper MMIO region
below `0x1000000000` on `linux-azure-6.8 6.8.0-1062.69~22.04.1`) running
as Elasticsearch data-nodes oopsed - two in kswapd0, one in the
elasticsearch process itself. Crashed machines were rebuilt urgently,
non-crashed ones a bit later, so we got apport and /proc/iomem from
them:
- oops-node1-kswapd.log # fragment from crashed node
- oops-node2-kswapd.log # fragment from crashed node
- oops-node3-elasticsearch.log # fragment from crashed node
- proc-iomem-node4.log # output of same set machine, not crashed
- apport-node4.txt # apport of same set machine, not crashed
[Suspect]
An oops in `shrink_folio_list()`.
[Steps to reproduce]
./reproduce.sh azure-1062 # or: noble-31 noble-137 noble-139
azure-1064 azure-1062-read-deleted azure-1062-mglru-off
The script fetches the kernel from Launchpad, builds a rootfs, and boots
QEMU: 4 GiB guest, 4 x 128 MB holes via `memmap=`, XFS on /dev/vda with
a 16 GiB file, read via mmap with jumps to random offsets.
Ext4 is untested.
[Versions tested]
target kernel result
log
noble-31 linux 6.8.0-31.31 no oops,
5,000,000 jumps reproduce-noble-31.log
noble-137 linux 6.8.0-137.137 oops, before
jump 200,000 reproduce-noble-137.log
noble-139 linux 6.8.0-139.139 (proposed) oops, before
jump 400,000 reproduce-noble-139.log
azure-1062 linux-azure-6.8 1062.69~22.04.1 oops, before
jump 200,000 reproduce-azure-1062.log
azure-1064 linux-azure-6.8 1064.72~22.04.1 oops, before
jump 800,000 reproduce-azure-1064.log
azure-1062-read-deleted rebuilt from 1062 source no oops,
5,000,000 jumps reproduce-azure-1062-read-deleted.log
azure-1062-mglru-off rebuilt from 1062 source oops, before
jump 800,000 reproduce-azure-1062-mglru-off.log
In most of these runs the fault came twice: first in kswapd0, then in
the reading process itself. This matches both patterns we saw in
production.
All these logs come from runs with 4 holes and a limit of 5,000,000
jumps. Fewer holes make the fault less likely - with a single hole we
still got it, but the run took much longer. With 4 holes a bad kernel
faults in minutes, so we did not have to run each version for days. We
can run more tests if needed.
137.137 (noble Updates) and 1064.72 (jammy Updates) are the newest
published builds; 31.31 is noble Release before the backport named under
[Cause].
[Cause]
Probably it's in `mm/vmscan.c:1277`
if (folio_test_large(folio) && list_empty(&folio->_deferred_list))
flags |= TTU_SYNC;
The line appeared in `Ubuntu-6.8.0-48.48` with the backport for LP
#2076147.
73bc32875ee9 mm: hold PTL from the first PTE while reclaiming a
large folio
Mainline dropped it - it seems that wasn't backported.
e5a119c4a683 mm/vmscan: drop checking if _deferred_list is empty
before using TTU_SYNC
The `azure-1062-read-deleted` target applies exactly this one-line
removal.
If we filed this against the wrong package, or if you need anything else
from us, please let us know.
** Affects: ubuntu
Importance: Undecided
Status: New
** Tags: cherry-pick jammy kernel-oops noble regression-update testcase
uec-images
** Attachment added: "apport of same set machine, not crashed"
https://bugs.launchpad.net/bugs/2163714/+attachment/5992988/+files/apport-node4.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163714
Title:
oops in shrink_folio_list(): out-of-bounds read of
folio->_deferred_list
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/2163714/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
