Re: [PATCH V5 04/38] vfio/container: preserve descriptors

[Steven Sistare]

On 6/23/2025 5:07 AM, Duan, Zhenzhong wrote:

-----Original Message-----
From: Steve Sistare <steven.sist...@oracle.com>
Subject: [PATCH V5 04/38] vfio/container: preserve descriptors

At vfio creation time, save the value of vfio container, group, and device
descriptors in CPR state.  On qemu restart, vfio_realize() finds and uses
the saved descriptors.

During reuse, device and iommu state is already configured, so operations
in vfio_realize that would modify the configuration, such as vfio ioctl's,
are skipped.  The result is that vfio_realize constructs qemu data
structures that reflect the current state of the device.

Signed-off-by: Steve Sistare <steven.sist...@oracle.com>
Reviewed-by: Cédric Le Goater <c...@redhat.com>
Reviewed-by: Zhenzhong Duan <zhenzhong.d...@intel.com>
---
include/hw/vfio/vfio-cpr.h |  6 +++++
hw/vfio/container.c        | 67 +++++++++++++++++++++++++++++++++++----------
-
hw/vfio/cpr-legacy.c       | 42 +++++++++++++++++++++++++++++
3 files changed, 100 insertions(+), 15 deletions(-)

diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h
index d4e0bd5..5a2e5f6 100644
--- a/include/hw/vfio/vfio-cpr.h
+++ b/include/hw/vfio/vfio-cpr.h
@@ -13,6 +13,7 @@

struct VFIOContainer;
struct VFIOContainerBase;
+struct VFIOGroup;

typedef struct VFIOContainerCPR {
     Error *blocker;
@@ -30,4 +31,9 @@ bool vfio_cpr_register_container(struct VFIOContainerBase
*bcontainer,
                                  Error **errp);
void vfio_cpr_unregister_container(struct VFIOContainerBase *bcontainer);

+int vfio_cpr_group_get_device_fd(int d, const char *name);
+
+bool vfio_cpr_container_match(struct VFIOContainer *container,
+                              struct VFIOGroup *group, int fd);
+
#endif /* HW_VFIO_VFIO_CPR_H */
diff --git a/hw/vfio/container.c b/hw/vfio/container.c
index 93cdf80..5caae4c 100644
--- a/hw/vfio/container.c
+++ b/hw/vfio/container.c
@@ -31,6 +31,8 @@
#include "system/reset.h"
#include "trace.h"
#include "qapi/error.h"
+#include "migration/cpr.h"
+#include "migration/blocker.h"
#include "pci.h"
#include "hw/vfio/vfio-container.h"
#include "vfio-helpers.h"
@@ -425,7 +427,12 @@ static VFIOContainer *vfio_create_container(int fd,
VFIOGroup *group,
         return NULL;
     }

-    if (!vfio_set_iommu(fd, group->fd, &iommu_type, errp)) {
+    /*
+     * During CPR, just set the container type and skip the ioctls, as the
+     * container and group are already configured in the kernel.
+     */
+    if (!cpr_is_incoming() &&
+        !vfio_set_iommu(fd, group->fd, &iommu_type, errp)) {
         return NULL;
     }

@@ -592,6 +599,11 @@ static bool vfio_container_group_add(VFIOContainer
*container, VFIOGroup *group,
     group->container = container;
     QLIST_INSERT_HEAD(&container->group_list, group, container_next);
     vfio_group_add_kvm_device(group);
+    /*
+     * Remember the container fd for each group, so we can attach to the same
+     * container after CPR.
+     */
+    cpr_resave_fd("vfio_container_for_group", group->groupid, container->fd);

I know this is already merged. Just out of curious, It looks cpr_save_fd is 
enough?

vfio_container_group_add is called from multiple places.  In some, we know that 
the fd
is being saved for the first time, in others we do not know.  resave avoids 
creating
a duplicate entry.

- Steve