v4 has been floating around for a while. Let's see if we can find someone
to merge this; or at least give some more feedback ... all patches have
at least one RB.
I realized that resizing RAM blocks while the guest is being migrated
(precopy: resize while still running on the source, postcopy: resize
while already running on the target) is buggy. In case of precopy, we
can simply cancel migration. Postcopy handling is more involved. Resizing
can currently happen during a guest reboot, triggered by ACPI rebuilds.
Along with the fixes, some cleanups.
--------------------------------------------------------------------------
Example to highlight one part of the problem:
1. Start a paused VM (where a ramblock resize will trigger when booting):
sudo build/qemu-system-x86_64 \
--enable-kvm \
-S \
-machine q35,nvdimm=on \
-smp 1 \
-cpu host \
-m size=20G,slots=8,maxmem=22G \
-object memory-backend-file,id=mem0,mem-path=/tmp/nvdimm,size=256M \
-device nvdimm,label-size=131072,memdev=mem0,id=nvdimm0,slot=1 \
-nodefaults \
-chardev stdio,nosignal,id=serial \
-device isa-serial,chardev=serial \
-chardev socket,id=monitor,path=/var/tmp/monitor,server,nowait \
-mon chardev=monitor,mode=readline \
-device vmgenid \
-device intel-iommu \
-nographic
2. Starting precopy and then starting the VM to trigger resizing during
precopy:
QEMU 5.2.95 monitor - type 'help' for more information
(qemu) migrate -d "exec:gzip -c > STATEFILE.gz"
QEMU 5.2.95 monitor - type 'help' for more information
(qemu) cont
3a. Before this series, migration never completes:
QEMU 5.2.95 monitor - type 'help' for more information
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 43826 ms
expected downtime: 300 ms
setup: 5 ms
transferred ram: 65981 kbytes
throughput: 8.27 mbps
remaining ram: 18446744073709551612 kbytes
total ram: 21234188 kbytes
duplicate: 5308454 pages
skipped: 0 pages
normal: 93 pages
normal bytes: 372 kbytes
dirty sync count: 1
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 0
4. With this change, migration is properly aborted:
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: cancelled
total time: 0 ms
--------------------------------------------------------------------------
Cc: Eduardo Habkost <[email protected]>
Cc: Marcel Apfelbaum <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Paolo Bonzini <[email protected]>
Cc: Richard Henderson <[email protected]>
Cc: Juan Quintela <[email protected]>
Cc: "Dr. David Alan Gilbert" <[email protected]>
Cc: Peter Xu <[email protected]>
Cc: Alex Williamson <[email protected]>
v4 -> v5:
- Rephrased some patch descriptions
- Dropped some patches to reduce the footprint
-- "stubs/ram-block: Remove stubs that are no longer needed"
-- "migration/ram: Tolerate partially changed mappings in postcopy code"
- Removed as already upstream now
-- "migration/ram: Consolidate variable reset after placement in
ram_load_postcopy()"
v3 -> v4:
- Rebased and retested
- Added RBs
v2 -> v3:
- Rebased on current master
- Added RBs
- "migration/ram: Tolerate partially changed mappings in postcopy code"
-- Extended the comment for the uffdio unregister part.
v1 -> v2:
- "util: vfio-helpers: Factor out and fix processing of existing ram
blocks"
-- Stringify error
- "migraton/ram: Handle RAM block resizes during precopy"
-- Simplified check if we're migrating on the source
- "exec: Relax range check in ram_block_discard_range()"
-- Added to make discard during resizes actually work
- "migration/ram: Discard new RAM when growing RAM blocks after
ram_postcopy_incoming_init()"
-- Better checks if in the right postcopy mode.
-- Better patch subject/description/comments
- "migration/ram: Handle RAM block resizes during postcopy"
-- Better comments
-- Adapt to changed postcopy checks
- "migrate/ram: Get rid of "place_source" in ram_load_postcopy()"
-- Dropped, as broken
- "migration/ram: Tolerate partially changed mappings in postcopy code"
-- Better comment / description. Clarify that no implicit wakeup will
happen
-- Warn on EINVAL (older kernels)
-- Wake up any waiter explicitly
David Hildenbrand (10):
util: vfio-helpers: Factor out and fix processing of existing ram
blocks
numa: Teach ram block notifiers about resizeable ram blocks
numa: Make all callbacks of ram block notifiers optional
migration/ram: Handle RAM block resizes during precopy
exec: Relax range check in ram_block_discard_range()
migration/ram: Discard RAM when growing RAM blocks after
ram_postcopy_incoming_init()
migration/ram: Simplify host page handling in ram_load_postcopy()
migration/ram: Handle RAM block resizes during postcopy
migration/multifd: Print used_length of memory block
migration/ram: Use offset_in_ramblock() in range checks
hw/core/numa.c | 41 +++++++++--
hw/i386/xen/xen-mapcache.c | 7 +-
include/exec/cpu-common.h | 1 +
include/exec/memory.h | 10 +--
include/exec/ramblock.h | 10 +++
include/exec/ramlist.h | 13 ++--
migration/migration.c | 9 ++-
migration/migration.h | 1 +
migration/multifd.c | 2 +-
migration/postcopy-ram.c | 15 ++++-
migration/ram.c | 135 +++++++++++++++++++++++++++++--------
softmmu/physmem.c | 26 +++++--
target/i386/hax/hax-mem.c | 5 +-
target/i386/sev.c | 18 ++---
util/vfio-helpers.c | 41 ++++-------
15 files changed, 241 insertions(+), 93 deletions(-)
--
2.30.2
