On Mon, Jul 28, 2025 at 4:51 PM Eugenio Perez Martin
<epere...@redhat.com> wrote:
>
> On Mon, Jul 28, 2025 at 9:36 AM Jason Wang <jasow...@redhat.com> wrote:
> >
> > On Mon, Jul 28, 2025 at 3:09 PM Jason Wang <jasow...@redhat.com> wrote:
> > >
> > > On Fri, Jul 25, 2025 at 5:33 PM Michael S. Tsirkin <m...@redhat.com>
> > > wrote:
> > > >
> > > > On Thu, Jul 24, 2025 at 05:59:20PM -0400, Jonah Palmer wrote:
> > > > >
> > > > >
> > > > > On 7/23/25 1:51 AM, Jason Wang wrote:
> > > > > > On Tue, Jul 22, 2025 at 8:41 PM Jonah Palmer
> > > > > > <jonah.pal...@oracle.com> wrote:
> > > > > > >
> > > > > > > This series is an RFC initial implementation of iterative live
> > > > > > > migration for virtio-net devices.
> > > > > > >
> > > > > > > The main motivation behind implementing iterative migration for
> > > > > > > virtio-net devices is to start on heavy, time-consuming operations
> > > > > > > for the destination while the source is still active (i.e. before
> > > > > > > the stop-and-copy phase).
> > > > > >
> > > > > > It would be better to explain which kind of operations were heavy
> > > > > > and
> > > > > > time-consuming and how iterative migration help.
> > > > > >
> > > > >
> > > > > You're right. Apologies for being vague here.
> > > > >
> > > > > I did do some profiling of the virtio_load call for virtio-net to try
> > > > > and
> > > > > narrow down where exactly most of the downtime is coming from during
> > > > > the
> > > > > stop-and-copy phase.
> > > > >
> > > > > Pretty much the entirety of the downtime comes from the
> > > > > vmstate_load_state
> > > > > call for the vmstate_virtio's subsections:
> > > > >
> > > > > /* Subsections */
> > > > > ret = vmstate_load_state(f, &vmstate_virtio, vdev, 1);
> > > > > if (ret) {
> > > > > return ret;
> > > > > }
> > > > >
> > > > > More specifically, the vmstate_virtio_virtqueues and
> > > > > vmstate_virtio_extra_state subsections.
> > > > >
> > > > > For example, currently (with no iterative migration), for a virtio-net
> > > > > device, the virtio_load call took 13.29ms to finish. 13.20ms of that
> > > > > time
> > > > > was spent in vmstate_load_state(f, &vmstate_virtio, vdev, 1).
> > > > >
> > > > > Of that 13.21ms, ~6.83ms was spent migrating
> > > > > vmstate_virtio_virtqueues and
> > > > > ~6.33ms was spent migrating the vmstate_virtio_extra_state
> > > > > subsections. And
> > > > > I believe this is from walking VIRTIO_QUEUE_MAX virtqueues, twice.
> > > >
> > > > Can we optimize it simply by sending a bitmap of used vqs?
> > >
> > > +1.
> > >
> > > For example devices like virtio-net may know exactly the number of
> > > virtqueues that will be used.
> >
> > Ok, I think it comes from the following subsections:
> >
> > static const VMStateDescription vmstate_virtio_virtqueues = {
> > .name = "virtio/virtqueues",
> > .version_id = 1,
> > .minimum_version_id = 1,
> > .needed = &virtio_virtqueue_needed,
> > .fields = (const VMStateField[]) {
> > VMSTATE_STRUCT_VARRAY_POINTER_KNOWN(vq, struct VirtIODevice,
> > VIRTIO_QUEUE_MAX, 0, vmstate_virtqueue, VirtQueue),
> > VMSTATE_END_OF_LIST()
> > }
> > };
> >
> > static const VMStateDescription vmstate_virtio_packed_virtqueues = {
> > .name = "virtio/packed_virtqueues",
> > .version_id = 1,
> > .minimum_version_id = 1,
> > .needed = &virtio_packed_virtqueue_needed,
> > .fields = (const VMStateField[]) {
> > VMSTATE_STRUCT_VARRAY_POINTER_KNOWN(vq, struct VirtIODevice,
> > VIRTIO_QUEUE_MAX, 0, vmstate_packed_virtqueue,
> > VirtQueue),
> > VMSTATE_END_OF_LIST()
> > }
> > };
> >
> > A rough idea is to disable those subsections and use new subsections
> > instead (and do the compatibility work) like virtio_save():
> >
> > for (i = 0; i < VIRTIO_QUEUE_MAX; i++) {
> > if (vdev->vq[i].vring.num == 0)
> > break;
> > }
> >
> > qemu_put_be32(f, i);
> > ....
> >
>
> While I think this is a very good area to explore, I think we will get
> more benefits by pre-warming vhost-vdpa devices, as they take one or
> two orders of magnitude more than sending and processing the
> virtio-net state (1s~10s vs 10~100ms).
Expanding on this,
This is a great base to start from! My proposal is to perform these as
next steps:
1) Track in the destination what members changes from the vmstate sent
in the iterative phase and the downtime phase. I would start by
creating a copy of the last VirtIODevice and VirtIONet, at least for a
first RFC on top of this one.
2) Start the vhost-vdpa net device by the time the first iterative
state reaches us. Just creating the virtqueues should be noticeable,
but sending the CVQ messages here gives us even more downtime
reduction. Do not send these messages in the downtime if the
properties have not changed!
If we want to take a small detour to make a first revision simpler, we
could do the same with vhost-net first: Pre-warm the device creation
with the mq etc.
Thanks!
