On Thu, 10 Sept 2026 at 14:53, Anatoly Burakov <[email protected]> wrote: > > This patchset introduces a major refactor of the VFIO subsystem in DPDK to > support character device (cdev) interface introduced in Linux kernel, as well > as > make the API more streamlined and useful. The goal is to simplify device > management, improve compatibility, make the code readable, and clarify API. > > The following sections outline the key issues addressed by this patchset and > the > corresponding changes introduced. > > 1. Only group mode is supported > =============================== > > Since kernel version 4.14.327 (LTS), VFIO supports the new character device > (cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). > This > is a device-centric mode and does away with all the complexity regarding > groups > and IOMMU types, delegating it all to the kernel, and exposes a much simpler > interface to userspace. The old group-based implementation will still be > around, > and will need to be kept in DPDK for compatibility reasons. > > To enable this, VFIO is heavily refactored, so that the code can support both > modes while relying on (mostly) common infrastructure. > > Additionally, new `vfio_get_mode` API is added for those cases that need > some introspection into VFIO's internals, with two modes: group (old-style), > and cdev (the new mode). > > Historically, no-IOMMU mode was technically a variant of group mode, the > distinction is largely irrelevant to the user, as all usages of noiommu checks > in our codebase are for deciding whether to use IOVA or PA, not anything to do > with managing groups. However, now that upcoming kernel versions will support > no-IOMMU for both group, cdev compatibility, and full cdev paths, a new > `vfio_get_iommu_mode` is also added, with two modes: safe (full IOMMU > backing), > and unsafe (no-IOMMU mode). The naming is chosen explicitly to emphasize that > using no-IOMMU mode is not ideal. > > 2. Custom container assignment API does not map to cdev mode > ============================================================ > > The existing `rte_vfio_device_setup/release` model is fundamentally > incompatible > with cdev mode, because for custom container cases, the expected flow is that > the user binds the IOMMU group (and thus, implicitly, the device itself) to a > specific container using `rte_vfio_container_group_bind`, whereas this step is > not needed for cdev as the device fd is assigned to the container straight > away. > > Therefore, what we do instead is introduce a new API for container device > assignment which, semantically, will assign a device to specified container, > so > that when it is mapped using `rte_pci_map_device`, the appropriate container > is > selected. Under the hood though, we essentially transition to getting device > fd > straight away at assign stage, so that by the time the PCI bus attempts to map > the device, it is already mapped and we just return an fd. There is no > "unassign" API because `release_device` already performs that function. > > Because the API is now unified around device assignment, the old > group-specific > API's can be removed and, where appropriate, reimplemented using new API. > There > were other users of VFIO which relied on group API but only for convenience > purposes; no actual VFIO functionality depended on those API's. > > List of removed API's: > > * `rte_vfio_get_group_fd` > * `rte_vfio_clear_group` > * `rte_vfio_container_group_bind` (replaced by container assign API) > * `rte_vfio_container_group_unbind` > * `rte_vfio_noiommu_is_enabled` (replaced by new mode API) > > 3. The API responsibilities aren't clear and bleed into each other > ================================================================== > > Some API's do multiple things at once. In particular: > > * `rte_vfio_get_device_info` will setup the device > * `rte_vfio_setup_device` will get device info > > These API's have been adjusted to do one thing only. > > 4. The API does not need to be public > ===================================== > > The initial idea for exposing VFIO API was to enable userspace applications to > directly map memory for DMA, but it turns out that in practice only drivers > use > this API. Therefore, the entire VFIO API is made internal, driver-only, and is > renamed from `rte_vfio` to `dev_vfio`.
Thanks for the cleanup! I am still in the process of reviewing. Some first comments. - About patch 1, I am not sure I understand your intention. Without applying it, a conflict appears later in the series. - Do you know if some Linux capability is needed for using the cdev mode? Did you test this change in (unpriviledged) containers for example? - It was a ugly/gray area so far, but should we stop exposing an API that do nothing on Windows and FreeBSD? Especially now that we make it internal. Drivers calling the VFIO API may be broken/falsely announcing support on other OSes. I prefer a clear broken build rather than some runtime failure on those OSes the day someone starts testing. --vfio-intr / --vfio-vf-token EAL options are already Linux only. So EAL common code looks already ready. There may be one complication on the PCI bus side, with its calls to vfio_dma_map/unmap but it seems doable (implement per OS dma_map/dma_unmap internal symbols ?). Something like: https://github.com/david-marchand/dpdk/commit/cc77440883b0d38b842d6c520292b540eabb78c4 https://github.com/david-marchand/dpdk/commit/8f426089f949e0694e5201193be88de673c4283d - Probably for later, but I see one more constant added in config/meson.build. All VFIO objects seems to be local (or exchanged over MP messages). How much effort would it take to remove those constants in config/meson.build? I am thinking about RTE_MAX_VFIO_CONTAINERS, RTE_MAX_VFIO_GROUPS, RTE_MAX_VFIO_DEVICES, EAL_VFIO_MAX_USER_MEM_MAPS. - rte_eal_check_module() only user is VFIO. No opensource project use it. It could be removed in the future. -- David Marchand
