Public bug reported:
SUMMARY
-------
[regression] Slow VM boot and guest soft lockups with large-BAR GPU passthrough
returned in 6.8.0-136: SAUCE revert of "vfio/pci: Use unmap_mapping_range()" was
clobbered by the 2026-06-09 upstream stable patchset.
This is a recurrence of LP #2089306.
The SAUCE revert of "vfio/pci: Use unmap_mapping_range()" (upstream
aac6db75a9fc,
v6.10), which shipped in noble linux 6.8.0-53.55, was undone in 6.8.0-136.136
when
the commit was re-applied by a routine upstream stable sync.
"vfio/pci: Insert full vma on mmap'd MMIO fault" (upstream d71a989cf5d9, v6.10)
was NOT re-applied. That leaves the kernel in exactly the intermediate state
that
LP #2089306 explicitly documented as insufficient:
"Reverting this prefaulting behavior does not fully resolve the slowness, as a
VM still experiences extremely slow accesses to the passthrough devices as
VMAs
get faulted in, causing soft lockup warnings in the guest during boot. Thus,
'vfio/pci: Use unmap_mapping_range()' must also be reverted to restore
performance to that of versions prior to 6.8.0-48-generic."
AFFECTED VERSIONS
-----------------
Broken : 6.8.0-136.136 and later (confirmed on 6.8.0-137.137, current
noble-updates)
Last known good : 6.8.0-124.124 (the good window opens at 6.8.0-53.55)
EVIDENCE OF THE RE-APPLICATION
------------------------------
From "apt-get changelog linux-image-unsigned-6.8.0-137-generic", under
linux (6.8.0-136.136):
* Noble update: upstream stable patchset 2026-06-09 (LP: #2156149)
- vfio/pci: Use unmap_mapping_range()
Note this entry carries no CVE annotation, unlike the surrounding entries in
that
patchset.
The earlier reverts appear under linux (6.8.0-53.55):
* vfio_pci soft lockup on VM start while using PCIe passthrough (LP:
#2089306)
- SAUCE: Revert "vfio/pci: Insert full vma on mmap'd MMIO fault"
- SAUCE: Revert "vfio/pci: Use unmap_mapping_range()"
Confirmed in the shipped binary on 6.8.0-137-generic:
$ nm -u vfio-pci-core.ko | grep -E \
"remap_pfn_range|vmf_insert_pfn|unmap_mapping_range|zap_vma_ptes"
U unmap_mapping_range
U vmf_insert_pfn
"remap_pfn_range" and "zap_vma_ptes" are absent, confirming the per-page
vmf_insert_pfn() path with no prefault.
As noted in LP #2089306, noble does not carry ba168b52bf8e ("mm: use rwsem
assertion macros for mmap_lock"), so noble was never exposed to the WARN_ON that
motivated aac6db75a9fc upstream. There is no functional reason for the commit to
be present in noble.
[IMPACT]
--------
On a host with large-BAR PCIe devices passed through, the first access to each
BAR
mapping now faults the region in one 4 KiB page at a time instead of a single
io_remap_pfn_range() call over the whole VMA. For a 256 GiB BAR that is
67,108,864 individual faults.
Measured on an HGX B200 host (8 x NVIDIA B200, PCI ID 10de:2901, BAR1 / Region 2
= 256 GiB each) running an 8-GPU passthrough KVM guest:
6.8.0-124.124 : ~5.4 to 6.7 s per GPU, ~47 s from the first to the last GPU,
no soft lockups, all 8 GPUs usable.
6.8.0-137.137 : ~26 to 28 s per GPU, ~190 s from the first to the last GPU,
23 "watchdog: BUG: soft lockup" warnings inside the guest
spread across all 32 vCPUs (firewalld, khugepaged, systemd,
systemd-udevd, kworkers), and only 6 of 8 GPUs usable.
Guest dmesg during nvidia module load, showing the interval between successive
GPUs:
[ 17.912] nvidia 0000:06:00.0: enabling device (0140 -> 0142)
[ 45.258] nvidia 0000:07:00.0: enabling device (0140 -> 0142) +27.35 s
[ 71.932] nvidia 0000:08:00.0: enabling device (0140 -> 0142) +26.67 s
[ 99.725] nvidia 0000:09:00.0: enabling device (0140 -> 0142) +27.79 s
[ 125.682] nvidia 0000:0a:00.0: enabling device (0140 -> 0142) +25.96 s
[ 152.023] nvidia 0000:0b:00.0: enabling device (0140 -> 0142) +26.34 s
[ 179.230] nvidia 0000:0c:00.0: enabling device (0140 -> 0142) +27.21 s
[ 207.526] nvidia 0000:0d:00.0: enabling device (0140 -> 0142) +28.30 s
The secondary consequence is a hard functional failure, not merely
slowness.
The nvidia module is loaded from a systemd-udevd worker, whose default event
timeout is 180 s. The worker is killed at exactly 180.0 s after its event began:
[ 197.644] systemd-udevd[1006]: 0000:06:00.0: Worker [1027] processing
SEQNUM=6567 killed
Once SIGKILL is pending on the calling task, kthread_create() returns -EINTR, so
every device probed after that point fails:
[ 207.525] NVRM: GPU 0000:0c:00.0: nv_linux_init_open_q() failed!
[ 207.526] nvidia 0000:0c:00.0: probe with driver nvidia failed with error
-1
[ 235.572] NVRM: GPU 0000:0d:00.0: nv_linux_init_open_q() failed!
[ 235.573] nvidia 0000:0d:00.0: probe with driver nvidia failed with error
-1
[ 235.573] NVRM: The NVIDIA probe routine failed for 2 device(s).
The guest therefore sees 6 of 8 GPUs, reproducibly (3/3). Manually re-binding
the
two devices from a shell afterwards succeeds immediately, which confirms the
devices are healthy and that the failure is purely the interrupted probe.
CONFIRMED A/B ON THE SAME HOST
------------------------------
Same physical host (umb-b200-071), same guest image, same libvirt domain
definition, same GPU firmware and NVIDIA driver, and the same systemd-udevd
event
timeout (default 180 s; /etc/udev/udev.conf left unmodified, event_timeout still
commented out). The host kernel was the only variable changed.
Summary:
+----------------------------+------------------------+------------------------+
| Measurement | 6.8.0-137.137 (broken) | 6.8.0-124.124 (good) |
+----------------------------+------------------------+------------------------+
| vfio-pci-core undefined | vmf_insert_pfn | remap_pfn_range |
| symbols (nm -u) | unmap_mapping_range | zap_vma_ptes |
+----------------------------+------------------------+------------------------+
| per-GPU probe interval | 26 - 28 s | 5.4 - 6.7 s |
+----------------------------+------------------------+------------------------+
| first to last of 8 GPUs | 189.6 s | 46.9 s |
+----------------------------+------------------------+------------------------+
| systemd-udevd 180 s event | exceeded; worker | ~1/3 used; ~117 s |
| budget | killed at 197.644 s | margin remaining |
+----------------------------+------------------------+------------------------+
| GPUs visible in guest | 6 of 8 | 8 of 8 |
+----------------------------+------------------------+------------------------+
| guest soft lockup warnings | 23 | 0 |
+----------------------------+------------------------+------------------------+
| udev workers SIGKILLed | ~40 | 0 |
+----------------------------+------------------------+------------------------+
| udev event_timeout | 180 s (default) | 180 s (default) |
+----------------------------+------------------------+------------------------+
"nvidia NNNN:NN:NN.N: enabling device (0140 -> 0142)" timestamps in the
guest:
6.8.0-137 6.8.0-124
[ 17.912] 06:00.0 [ 8.956] 06:00.0
[ 45.258] 07:00.0 +27.35 s [ 19.946] 07:00.0 +10.99 s
[ 71.932] 08:00.0 +26.67 s [ 25.343] 08:00.0 +5.40 s
[ 99.725] 09:00.0 +27.79 s [ 31.408] 09:00.0 +6.06 s
[ 125.682] 0a:00.0 +25.96 s [ 37.277] 0a:00.0 +5.87 s
[ 152.023] 0b:00.0 +26.34 s [ 42.832] 0b:00.0 +5.55 s
[ 179.230] 0c:00.0 +27.21 s [ 49.517] 0c:00.0 +6.68 s
[ 207.526] 0d:00.0 +28.30 s [ 55.885] 0d:00.0 +6.37 s
first to last GPU: 189.6 s 46.9 s
Note that ~6 s per GPU on 6.8.0-124 is still not free: io_remap_pfn_range() must
populate 67 M PTEs for a 256 GiB BAR and the IOMMU mapping must be built. But
the
whole 8-GPU probe then consumes roughly a third of the 180 s udev budget instead
of overrunning it. The upstream v6.12 fix (f9e54c3a2f5b) would reduce even that
remaining cost, since PMD mappings cut 67 M insertions to about 131 k.
STEPS TO REPRODUCE
------------------
1. Ubuntu 24.04 host, kernel 6.8.0-136 or later, booted with
"intel_iommu=on iommu=pt".
2. Bind 8 large-BAR GPUs (BAR1 >= 128 GiB) to vfio-pci.
3. Start a KVM guest with all 8 assigned as <hostdev managed='yes'>.
4. In the guest run: dmesg | grep "enabling device"
Observe ~27 s between devices, plus "soft lockup" warnings.
5. Boot the host on 6.8.0-124.124 and repeat, changing nothing else. Intervals
drop to ~5 to 7 s per GPU, all 8 GPUs become usable, and no soft lockup
warnings occur.
[TEST PLAN]
-----------
As per Steps To Reproduce.
The two code paths can also be distinguished without booting a VM:
sudo cp $(modinfo -n vfio-pci-core) /tmp/v.ko.zst
sudo zstd -df /tmp/v.ko.zst -o /tmp/v.ko
nm -u /tmp/v.ko | grep -E "remap_pfn_range|vmf_insert_pfn"
Fixed kernel: references remap_pfn_range (and zap_vma_ptes)
Broken kernel: references vmf_insert_pfn (and unmap_mapping_range)
REQUESTED FIX
-------------
Re-apply the existing SAUCE revert of "vfio/pci: Use unmap_mapping_range()"
(aac6db75a9fc), as shipped in 6.8.0-53.55, and add whatever guard is appropriate
to prevent future upstream stable syncs from silently reintroducing it.
Please do NOT re-apply "vfio/pci: Insert full vma on mmap'd MMIO fault"
(d71a989cf5d9). That is the commit whose prefaulting behaviour caused the
host-side soft lockups and multi-hour VM start times in the original bug.
The upstream resolution is Peter Xu's "mm: Support huge pfnmaps" series plus
f9e54c3a2f5b ("vfio/pci: implement huge_fault support"), both in v6.12. That was
previously assessed as too invasive to backport to 6.8, which is why the revert
remains the appropriate fix for noble.
[WHERE PROBLEMS COULD OCCUR]
----------------------------
Identical to the original SRU in LP #2089306. The change is confined to
drivers/vfio/pci/vfio_pci_core.c and restores code that shipped in noble from
6.8.0-53.55 through 6.8.0-135.x without known issue.
Because noble does not carry ba168b52bf8e, the mmap_lock WARN_ON that
aac6db75a9fc addresses upstream cannot occur in this series.
[OTHER INFO]
------------
Related bugs:
LP #2089306 - original bug, fixed by the two SAUCE reverts in 6.8.0-53.55
LP #2156149 - the 2026-06-09 noble stable patchset that re-applied the commit
LP #2097389 - separate guest-side large-BAR PCI init slowness
("PCI: Batch BAR sizing operations"), fixed in 6.8.0-56.58
This is not vendor-specific. Any noble host performing large-BAR PCIe
passthrough
is affected, and the cost scales linearly with BAR size, so 256 GiB-BAR parts
are
roughly twice as badly affected as the 128 GiB H100 case that motivated the
original fix.
Host: Ubuntu 24.04.4, QEMU 8.2.2+ds-0ubuntu1.18, libvirt 10.0.0-2ubuntu8.15,
OVMF 2024.02-2ubuntu0.9
Guest: RHEL 9.8, kernel 5.14.0-687.39.1.el9_8.x86_64
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163742
Title:
[regression] Slow VM boot and guest soft lockups with large-BAR GPU
passthrough returned in 6.8.0-136
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2163742/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
