This RFC extends RISC-V CBQRI/resctrl support to external requesters
behind a RISC-V IOMMU.
The RISC-V IOMMU QoS ID extension associates a resource-control ID
(RCID) and monitoring ID (MCID) with IOMMU-related memory requests [2].
For a translated device request, the IOMMU obtains the IDs from the
device context translation attributes (DC.ta) and supplies them to the
I/O bridge after a successful translation. The bridge and downstream
CBQRI-controlled resources can then apply their existing allocation and
monitoring policy to the request.
The series is based directly on Drew Fustini's v4 RISC-V Ssqosid and
CBQRI resctrl series [1]. That dependency provides CPU-side QoS ID
plumbing and CBQRI-backed resctrl support. This series extends the same
resctrl identity to device requests translated by a RISC-V IOMMU.
The RISC-V names map to the existing resctrl identifiers as follows:
resctrl CLOSID -> RISC-V RCID
resctrl RMID -> RISC-V MCID
Userspace first creates a resctrl resource group. The new "devices"
file then assigns an external requester to that group:
userspace
|
| mkdir /sys/fs/resctrl/<group>
|
| echo "iommu_group:<id>" > /sys/fs/resctrl/<group>/devices
v
generic resctrl devices file
|
| (closid, rmid)
v
RISC-V CBQRI IOMMU binding backend
|
| (rcid, mcid)
v
RISC-V IOMMU DC.ta.{RCID,MCID}
The "devices" file is an assignment and membership interface only. It
does not describe a new resctrl resource, schema, or monitoring domain;
resource policy remains in the existing schemata and info files.
Common IOMMU helpers:
The userspace ABI names an IOMMU group by the numeric ID already used by
/sys/kernel/iommu_groups. Patch 1 adds iommu_group_get_by_id() as a
narrow, reference-counted lookup for that existing identifier.
Updating a group also needs stronger iteration semantics than separate
iommu_group_for_each_dev() calls can provide. All devices must be
validated before any device context is changed, and group membership
must not change between validation and update. Patch 2 therefore adds
iommu_group_update_devices():
lock group->mutex
for each device: check() /* may fail, no state changed */
for each device: update() /* non-failing callback */
unlock group->mutex
The helper contains no resctrl or QoS policy. It only provides an
all-or-none validation boundary for a group-wide update while keeping
IOMMU group internals private. The RISC-V IOMMU QoS code is its first
caller.
RISC-V IOMMU programming:
For translated device requests, the per-device-context QoS IDs are
programmed in DC.ta. Active contexts are invalidated after an update so
cached translations do not retain stale IDs. BARE mode cannot safely
represent per-IOMMU-group assignment because only the per-IOMMU
iommu_qosid global default is available, so group assignment rejects
BARE mode.
The RISC-V IOMMU also has a per-IOMMU iommu_qosid register. The series
exposes it as:
/sys/class/iommu/<iommu>/qosid
This sysfs attribute programs the global RCID and MCID used for
IOMMU-originated DDT, CQ, FQ, PQ, and MSI accesses. In BARE mode,
device-originated requests also use these IDs. It is not a resctrl group
assignment interface.
Open questions for this RFC:
* whether a common resctrl "devices" file is the right ABI for
associating external requesters with a resource group;
* whether iommu_group_get_by_id() is an acceptable narrow common
helper for resolving the numeric IOMMU group ID passed through that
ABI; and
* whether iommu_group_update_devices() provides the right common
contract for checked group-wide updates.
[1] Drew Fustini's CBQRI v4 series:
https://lore.kernel.org/linux-riscv/[email protected]/
[2] RISC-V IOMMU QoS ID extension:
https://docs.riscv.org/reference/cbqri/qos_iommu.html
Zhanpeng Zhang (7):
iommu: Add group lookup by ID
iommu: Add checked group device update helper
resctrl: Add a devices file for external requester assignment
iommu/riscv: Program QoS IDs for assigned groups
iommu/riscv: Expose global QoS IDs in sysfs
riscv_cbqri: Assign IOMMU groups to resource groups
selftests/iommu: Add RISC-V IOMMU QoS smoke test
.../ABI/testing/sysfs-class-iommu-riscv-iommu | 27 +
Documentation/filesystems/resctrl.rst | 26 +
MAINTAINERS | 12 +
arch/Kconfig | 6 +
arch/riscv/include/asm/qos.h | 28 +
drivers/iommu/iommu.c | 140 +++-
drivers/iommu/riscv/iommu-bits.h | 15 +
drivers/iommu/riscv/iommu.c | 419 ++++++++++-
drivers/iommu/riscv/iommu.h | 22 +-
drivers/resctrl/Kconfig | 6 +
drivers/resctrl/Makefile | 1 +
drivers/resctrl/cbqri_iommu.c | 276 ++++++++
fs/resctrl/rdtgroup.c | 206 +++++-
include/linux/iommu.h | 30 +
include/linux/resctrl.h | 45 ++
tools/testing/selftests/iommu/Makefile | 2 +
.../selftests/iommu/iommu_qos_smoke.sh | 649 ++++++++++++++++++
17 files changed, 1886 insertions(+), 24 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-riscv-iommu
create mode 100644 drivers/resctrl/cbqri_iommu.c
create mode 100755 tools/testing/selftests/iommu/iommu_qos_smoke.sh
base-commit: ab754eba620f5a66e7277b7b2d0e706aa7743e8a
--
2.50.1 (Apple Git-155)
