Tested this series of patches with virtio-net regression tests,
everything works fine.

Tested-by: Lei Yang <leiy...@redhat.com>


On Fri, Jul 25, 2025 at 6:01 AM Jonah Palmer <jonah.pal...@oracle.com> wrote:
>
>
>
> On 7/23/25 1:51 AM, Jason Wang wrote:
> > On Tue, Jul 22, 2025 at 8:41 PM Jonah Palmer <jonah.pal...@oracle.com> 
> > wrote:
> >>
> >> This series is an RFC initial implementation of iterative live
> >> migration for virtio-net devices.
> >>
> >> The main motivation behind implementing iterative migration for
> >> virtio-net devices is to start on heavy, time-consuming operations
> >> for the destination while the source is still active (i.e. before
> >> the stop-and-copy phase).
> >
> > It would be better to explain which kind of operations were heavy and
> > time-consuming and how iterative migration help.
> >
>
> You're right. Apologies for being vague here.
>
> I did do some profiling of the virtio_load call for virtio-net to try
> and narrow down where exactly most of the downtime is coming from during
> the stop-and-copy phase.
>
> Pretty much the entirety of the downtime comes from the
> vmstate_load_state call for the vmstate_virtio's subsections:
>
> /* Subsections */
> ret = vmstate_load_state(f, &vmstate_virtio, vdev, 1);
> if (ret) {
>      return ret;
> }
>
> More specifically, the vmstate_virtio_virtqueues and
> vmstate_virtio_extra_state subsections.
>
> For example, currently (with no iterative migration), for a virtio-net
> device, the virtio_load call took 13.29ms to finish. 13.20ms of that
> time was spent in vmstate_load_state(f, &vmstate_virtio, vdev, 1).
>
> Of that 13.21ms, ~6.83ms was spent migrating vmstate_virtio_virtqueues
> and ~6.33ms was spent migrating the vmstate_virtio_extra_state
> subsections. And I believe this is from walking VIRTIO_QUEUE_MAX
> virtqueues, twice.
>
> vmstate_load_state virtio-net v11
> vmstate_load_state PCIDevice v2
> vmstate_load_state_end PCIDevice end/0
> vmstate_load_state virtio-net-device v11
> vmstate_load_state virtio-net-queue-tx_waiting v0
> vmstate_load_state_end virtio-net-queue-tx_waiting end/0
> vmstate_load_state virtio-net-vnet v0
> vmstate_load_state_end virtio-net-vnet end/0
> vmstate_load_state virtio-net-ufo v0
> vmstate_load_state_end virtio-net-ufo end/0
> vmstate_load_state virtio-net-tx_waiting v0
> vmstate_load_state virtio-net-queue-tx_waiting v0
> vmstate_load_state_end virtio-net-queue-tx_waiting end/0
> vmstate_load_state virtio-net-queue-tx_waiting v0
> vmstate_load_state_end virtio-net-queue-tx_waiting end/0
> vmstate_load_state virtio-net-queue-tx_waiting v0
> vmstate_load_state_end virtio-net-queue-tx_waiting end/0
> vmstate_load_state_end virtio-net-tx_waiting end/0
> vmstate_load_state_end virtio-net-device end/0
> vmstate_load_state virtio v1
> vmstate_load_state virtio/64bit_features v1
> vmstate_load_state_end virtio/64bit_features end/0
> vmstate_load_state virtio/virtqueues v1
> vmstate_load_state virtqueue_state v1  <--- Queue idx 0
> ...
> vmstate_load_state_end virtqueue_state end/0
> vmstate_load_state virtqueue_state v1  <--- Queue idx 1023
> vmstate_load_state_end virtqueue_state end/0
> vmstate_load_state_end virtio/virtqueues end/0
> vmstate_load_state virtio/extra_state v1
> vmstate_load_state virtio_pci v1
> vmstate_load_state virtio_pci/modern_state v1
> vmstate_load_state virtio_pci/modern_queue_state v1  <--- Queue idx 0
> vmstate_load_state_end virtio_pci/modern_queue_state end/0
> ...
> vmstate_load_state virtio_pci/modern_queue_state v1  <--- Queue idx 1023
> vmstate_load_state_end virtio_pci/modern_queue_state end/0
> vmstate_load_state_end virtio_pci/modern_state end/0
> vmstate_load_state_end virtio_pci end/0
> vmstate_load_state_end virtio/extra_state end/0
> vmstate_load_state virtio/started v1
> vmstate_load_state_end virtio/started end/0
> vmstate_load_state_end virtio end/0
> vmstate_load_state_end virtio-net end/0
> vmstate_downtime_load type=non-iterable idstr=0000:00:03.0/virtio-net
> instance_id=0 downtime=13260
>
> With iterative migration for virtio-net (maybe all virtio devices?), we
> can send this early while the source is still running and then only send
> the deltas during the stop-and-copy phase. It's likely that the source
> wont be using all VIRTIO_QUEUE_MAX virtqueues during the migration
> period, so this could really minimize a large majority of the downtime
> contributed by virtio-net.
>
> This could be one example.
>
> >>
> >> The motivation behind this RFC series specifically is to provide an
> >> initial framework for such an implementation and get feedback on the
> >> design and direction.
> >> -------
> >>
> >> This implementation of iterative live migration for a virtio-net device
> >> is enabled via setting the migration capability 'virtio-iterative' to
> >> on for both the source & destination, e.g. (HMP):
> >>
> >> (qemu) migrate_set_capability virtio-iterative on
> >>
> >> The virtio-net device's SaveVMHandlers hooks are registered/unregistered
> >> during the device's realize/unrealize phase.
> >
> > I wonder about the plan for libvirt support.
> >
>
> Could you elaborate on this a bit?
>
> >>
> >> Currently, this series only sends and loads the vmstate at the start of
> >> migration. The vmstate is still sent (again) during the stop-and-copy
> >> phase, as it is today, to handle any deltas in the state since it was
> >> initially sent. A future patch in this series could avoid having to
> >> re-send and re-load the entire state again and instead focus only on the
> >> deltas.
> >>
> >> There is a slight, modest improvement in guest-visible downtime from
> >> this series. More specifically, when using iterative live migration with
> >> a virtio-net device, the downtime contributed by migrating a virtio-net
> >> device decreased from ~3.2ms to ~1.4ms on average:
> >
> > Are you testing this via a software virtio device or hardware one?
> >
>
> Just software (virtio-device, vhost-net) with these numbers. I can run
> some tests with vDPA hardware though.
>
> Those numbers were from a simple, 1 queue-pair virtio-net device.
>
> >>
> >> Before:
> >> -------
> >> vmstate_downtime_load type=non-iterable idstr=0000:00:03.0/virtio-net
> >>    instance_id=0 downtime=3594
> >>
> >> After:
> >> ------
> >> vmstate_downtime_load type=non-iterable idstr=0000:00:03.0/virtio-net
> >>    instance_id=0 downtime=1607
> >>
> >> This slight improvement is likely due to the initial vmstate_load_state
> >> call "warming up" pages in memory such that, when it's called a second
> >> time during the stop-and-copy phase, allocation and page-fault latencies
> >> are reduced.
> >> -------
> >>
> >> Comments, suggestions, etc. are welcome here.
> >>
> >> Jonah Palmer (6):
> >>    migration: Add virtio-iterative capability
> >>    virtio-net: Reorder vmstate_virtio_net and helpers
> >>    virtio-net: Add SaveVMHandlers for iterative migration
> >>    virtio-net: iter live migration - migrate vmstate
> >>    virtio,virtio-net: skip consistency check in virtio_load for iterative
> >>      migration
> >>    virtio-net: skip vhost_started assertion during iterative migration
> >>
> >>   hw/net/virtio-net.c            | 246 +++++++++++++++++++++++++++------
> >>   hw/virtio/virtio.c             |  32 +++--
> >>   include/hw/virtio/virtio-net.h |   8 ++
> >>   include/hw/virtio/virtio.h     |   7 +
> >>   migration/savevm.c             |   1 +
> >>   qapi/migration.json            |   7 +-
> >>   6 files changed, 247 insertions(+), 54 deletions(-)
> >>
> >> --
> >> 2.47.1
> >
> > Thanks
> >
> >>
> >
>
>


Reply via email to