Current memory operations like pinning may take a lot of time at the destination. Currently they are done after the source of the migration is stopped, and before the workload is resumed at the destination. This is a period where neigher traffic can flow, nor the VM workload can continue (downtime).
We can do better as we know the memory layout of the guest RAM at the destination from the moment the migration starts. Moving that operation allows QEMU to communicate the kernel the maps while the workload is still running in the source, so Linux can start mapping them. Also, the destination of the guest memory may finish before the destination QEMU maps all the memory. In this case, the rest of the memory will be mapped at the same time as before applying this series, when the device is starting. So we're only improving with this series. RFC TODO: We should be able to not finish the migration while the memory is still not mapped, but I still need to find how. Suggestions are welcome. Note that further devices setup at the end of the migration may alter the guest memory layout. But same as the previous point, many operations are still done incrementally, like memory pinning, so we're saving time anyway. Only tested with vdpa_sim. I'm sending this before full benchmark, as some work like [1] can be based on it, and Si-Wei agreed on benchmark this series with his experience. This needs to be applied on top of [2], which perform some code reorganization that allows to map the memory without knowing the queue layout the guest configure on the device. Future directions on top of this series may include: * Iterative migration of virtio-net devices, as it may reduce downtime per [1]. vhost-vdpa net can apply the configuration through CVQ in the destination while the source is still migrating. * Move more things ahead of migration time, like DRIVER_OK. * Check that the devices of the destination are valid, and cancel the migration in case it is not. RFC v2: * Delegate map to another thread so it does no block QMP. * Fix not allocating iova_tree if x-svq=on at the destination. * Rebased on latest master. * More cleanups of current code, that might be split from this series too. [1] https://lore.kernel.org/qemu-devel/[email protected]/T/ [2] https://lists.nongnu.org/archive/html/qemu-devel/2023-11/msg05331.html Eugenio Pérez (10): vdpa: do not set virtio status bits if unneeded vdpa: make batch_begin_once early return vdpa: merge _begin_batch into _batch_begin_once vdpa: extract out _dma_end_batch from _listener_commit vdpa: factor out stop path of vhost_vdpa_dev_start vdpa: check for iova tree initialized at net_client_start vdpa: set backend capabilities at vhost_vdpa_init vdpa: add vhost_vdpa_load_setup vdpa: add vhost_vdpa_net_load_setup NetClient callback virtio_net: register incremental migration handlers include/hw/virtio/vhost-vdpa.h | 25 ++++ include/net/net.h | 6 + hw/net/virtio-net.c | 35 +++++ hw/virtio/vhost-vdpa.c | 257 +++++++++++++++++++++++++++------ net/vhost-vdpa.c | 37 ++++- 5 files changed, 312 insertions(+), 48 deletions(-) -- 2.39.3
