Public bug reported:
[1) Release of Ubuntu]
```
$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04.4 LTS
Release: 24.04
```
[2) Package version]
```
$ apt-cache policy qemu-system-x86 qemu-block-extra
qemu-system-x86:
Installed: 1:8.2.2+ds-0ubuntu1.16
Candidate: 1:8.2.2+ds-0ubuntu1.16
qemu-block-extra:
Installed: 1:8.2.2+ds-0ubuntu1.16
Candidate: 1:8.2.2+ds-0ubuntu1.16
```
Note on our deployment: this is OpenStack 2024.2 deployed with Kolla-Ansible, so
libvirt and QEMU run inside an OCI container built FROM ubuntu:noble. The
packages
are the unmodified ones from the Ubuntu noble archive
(qemu-system-x86 and qemu-block-extra, 1:8.2.2+ds-0ubuntu1.16), and the host is
also Ubuntu 24.04. The bug is in the archive package, not in anything Kolla
changes.
[3) What I expected to happen]
Querying a VM's block-device statistics — virsh domstats --block <domain>, or
any
monitoring agent that does the same — should be a fast, read-only operation
that does
not disturb the running guest.
[4) What happened instead]
The whole VM freezes for as long as the query takes, which under ordinary I/O
load is
tens of seconds. During the freeze the guest is completely unresponsive: the
guest
clock stops, SSH sessions hang, and the console shows RCU stall warnings
(rcu_preempt kthread starved). The guest recovers on its own once the query
returns.
Measured on an RBD-backed Cinder volume with rbd_qos_bps_limit = 8388608
(8 MiB/s):
Condition: discard in flight (mkfs.xfs on the volume)
virsh domstats --block: 60.39 s
Condition: 256 MiB throttled read in flight
virsh domstats --block: 10.9 s
Condition: same load, query-blockstats instead (control)
Result: 0.142 s
The control row is the key comparison: the same VM under the same load answers
query-blockstats in 0.142 s, so the cost is specific to query-named-block-nodes.
[Impact]
qemu_rbd_get_specific_info() in block/rbd.c issues a synchronous, blocking
rbd_read() to probe the image's encryption format every time the block node is
queried:
```
static ImageInfoSpecific *qemu_rbd_get_specific_info(BlockDriverState *bs, ...)
{
char buf[RBD_ENCRYPTION_LUKS_HEADER_VERIFICATION_LEN] = {0};
if (s->image_size >= RBD_ENCRYPTION_LUKS_HEADER_VERIFICATION_LEN) {
r = rbd_read(s->image, 0,
RBD_ENCRYPTION_LUKS_HEADER_VERIFICATION_LEN, buf); /*
blocks */
```
This runs on QEMU's main thread with the BQL held — bdrv_named_nodes_list()
(block.c) begins with GLOBAL_STATE_CODE() and calls bdrv_block_device_info()
for every node, which calls bdrv_get_specific_info() (block/qapi.c).
This 8-byte read (RBD_ENCRYPTION_LUKS_HEADER_VERIFICATION_LEN is 8) is the only
operation on the whole query path that touches the network. Everything else
bdrv_do_query_node_info() does for an RBD node is served from memory:
bdrv_getlength() returns the cached bs->total_sectors (rbd never sets
has_variable_length); bdrv_get_allocated_file_size() returns -ENOTSUP because
rbd
is a protocol driver that does not implement the hook; and
qemu_rbd_co_get_info() just
returns the cached object_size.
Its size is irrelevant — it is a synchronous round-trip to Ceph, so it inherits
whatever
queue depth and throttling the image has. If it is slow — queued behind other
I/O,
throttled by librbd QoS, or on a degraded cluster — the BQL is held for the
entire
duration.
What the read buys is very little. The 8 bytes are compared against four magic
strings
(LUKS, LUKS2, and their layered variants), and the entire result is a single
optional
enum:
```
{ 'struct': 'ImageInfoSpecificRbd',
'data': { '*encryption-format': 'RbdImageEncryptionFormat' } }
```
For any image that does not use RBD encryption — which is every volume in our
deployment, and the default for OpenStack Cinder RBD volumes generally — none
of the
four magics ever match and the field is simply absent. So on an unencrypted
image
the guest can be frozen for a minute in order to report nothing at all.
The value is also not used for any decision inside QEMU; it is reported to the
user as a
hint. The commit's own QAPI documentation change says the probe result can be
altered by
the guest writing a header into its own disk and so should not be trusted.
Guest vCPUs block at
their next MMIO exit (prepare_mmio_access()) and the VM stops executing.
This is reachable in normal production, not only in edge cases:
* libvirt sends query-named-block-nodes unconditionally alongside
query-blockstats whenever anything requests block stats. In
src/qemu/qemu_driver.c, qemuDomainGetStatsBlock() calls
qemuMonitorGetAllBlockStatsInfo() and then
qemuMonitorBlockStatsUpdateCapacityBlockdev() inside the same monitor section,
with no flag to skip the second call.
* That path is used by virsh domstats --block, by OpenStack, and by
prometheus-libvirt-exporter, which scrapes every compute node every few
seconds.
* VIR_CONNECT_GET_ALL_DOMAINS_STATS_NOWAIT does not help: it only affects
acquiring libvirt's own domain job lock, not the QEMU monitor call. We
measured it
making no difference (25.995 s with --nowait vs 17.008 s without, both
freezing
the guest).
* The trigger does not need to be unusual. Any queued or throttled I/O is
enough. In
our OpenStack deployment rbd_qos_iops_limit is set per-image on essentially
every
Cinder volume, so the precondition exists cluster-wide.
[Fix]
Upstream commit 4af976ef398e — "rbd: Fix .bdrv_get_specific_info
implementation",
Kevin Wolf, 2025-08-11.
Buglink:
https://issues.redhat.com/browse/RHEL-105440
From the commit message:
```
The first is that it issues a blocking rbd_read() call in order to probe the
encryption format for the image while querying the node. This means that if the
connection to the server goes down, not only I/O is stuck (which is
unavoidable),
but query-named-block-nodes will actually make the whole QEMU instance
unresponsive. .bdrv_get_specific_info implementations shouldn't perform blocking
operations, but only return what is already known.
```
The fix stores the encryption format in BDRVRBDState at open time — where
blocking
is already expected and harmless — and returns the cached value when queried.
It also
corrects a second bug: for an image already opened with RBD encryption enabled,
the
old code probed the decrypted guest data and could report a misleading
"double encryption" format.
Release status:
* present in QEMU 10.1.0, and backported to stable 10.0.4 (commit
d1a5c0c7a)
* not in any 8.2.x release. stable-8.2 ended at v8.2.10 (2025-03-26), 138 days
before this fix was authored, so upstream will not produce an 8.2.x
containing it.
* noble's 1:8.2.2+ds-0ubuntu1.16 and .18 changelogs contain no rbd
changes.
* no fixed qemu is available from any noble pocket (release / updates /
proposed /
backports) or from the Ubuntu Cloud Archive, which ships no qemu for noble.
Red Hat has backported the same commit for CentOS Stream 10 / RHEL:
https://gitlab.com/redhat/centos-stream/src/qemu-
kvm/-/merge_requests/399
[Confirmed by building a patched package]
We verified the fix resolves this specific bug by backporting the commit
ourselves,
changing nothing else.
Build. apt-get source qemu=1:8.2.2+ds-0ubuntu1.16, added 4af976ef398e as a quilt
patch, rebuilt. The patch applies cleanly to 8.2.2 — no conflicts, only line
offsets (9 hunks in block/rbd.c, 1 in qapi/block-core.json).
Deploy. Replaced only /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so from the
rebuilt qemu-block-extra, then hard-rebooted one guest so a new QEMU process
would
load it. Everything else — host, kernel, Ceph cluster, libvirt, the QEMU binary
itself, the volume, and the QoS limit — was unchanged.
Note for anyone reproducing this: the package version must be left exactly
1:8.2.2+ds-0ubuntu1.16. QEMU's module loader requires a matching build stamp
(util/module.c checks for the qemu_stamp<CONFIG_STAMP> symbol, and
scripts/qemu-stamp.py derives CONFIG_STAMP from a SHA-1 over the meson project
version, the pkgversion, and the contents of configure). Bumping the version
with
dch changes the stamp and the module is rejected with
"Only modules from the same build can be loaded." followed by "Unknown driver
'rbd'".
Results, patched, same host and same 8 MiB/s throttle (verified enforced at
8.4 MB/s across five consecutive dd runs before testing):
Test: virsh domstats --block, discard in flight
Result: 0.151 / 0.150 / 0.141 / 0.153 / 0.150 / 0.144 s
Test: libvirt-exporter scrape, discard in flight
Result: 30 consecutive samples, all 0.19–0.24 s
Test: libvirt-exporter scrape, 256 MiB throttled read
Result: all 0.21–0.22 s
Test: guest clock, 60 s under continuous discard load
Result: continuous, no gaps
The headline comparison is the same command, same VM, same host, same
workload:
60.39 s unpatched → ~0.15 s patched.
[Test Plan]
1. A Ceph cluster with an RBD image attached to a guest as a virtio-blk
device.
2. Throttle it so I/O queues:
```
rbd config image set <pool>/<image> rbd_qos_bps_limit 8388608
```
Verify the limit is actually enforced — a configured limit is not always
live. In
the guest, dd if=/dev/vdb of=/dev/null bs=1M count=64 iflag=direct must
report
roughly 8 MB/s. Repeat it a few times; do not proceed until it is consistent.
3. In the guest, start work that fills the queue, e.g.:
```
mkfs.xfs -f /dev/vdb
```
4. On the host, while that is still in flight:
```
time virsh domstats --block <domain>
```
5. In the guest, in a second session:
```
for i in $(seq 60); do date +%H:%M:%S; sleep 1; done
```
Unfixed: step 4 takes tens of seconds, and step 5 shows that many seconds are
missing
from the guest clock.
Fixed: step 4 returns in roughly 0.15 s and step 5 is continuous.
[Where problems could occur]
The change alters when the encryption-format probe happens (image open instead
of
node query) and what is reported for images opened with RBD encryption enabled.
A
regression would most plausibly appear as a changed encryption-format field in
query-named-block-nodes or qemu-img info output for RBD images.
Guest-visible I/O behaviour is unchanged. No on-disk format, wire protocol, or
migration-stream change is involved. The code change is confined to
block/rbd.c, so
only RBD-backed guests are affected at all; the qapi/block-core.json hunk is
documentation only, clarifying that the probe result is guest-modifiable and
should
not be treated as trusted.
Because the module carries a build stamp tied to the package version, a rebuilt
qemu-block-extra must be installed together with the matching qemu-system-x86.
[Other Info]
Precedent: LP #2126951 was also a query-named-block-nodes fix and was backported
into noble's 1:8.2.2+ds-0ubuntu1.14.
** Affects: qemu (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2166503
Title:
rbd: blocking rbd_read() in .bdrv_get_specific_info freezes guests
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2166503/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
