To address this, patch #6 in the series introduces a new helper
function pci_setup_iommu_per_bus(), which explicitly sets the
iommu_per_bus field in the PCIBus structure. This allows
pci_device_get_iommu_bus_devfn() to retrieve IOMMU ops based
on the specific bus.
This patch series introduces support for a user-creatable SMMUv3 device
(-device arm-smmuv3) in QEMU.
Tested-by: Nathan Chen <nath...@nvidia.com>
I re-ran the test from v3 [0] and am able to create 16 SMMUv3 devices in
a qemu VM with emulated devices properly associated with the guest SMMUs
in guest sysfs - verified with some guest SMMUs having two or three
emulated NICs assigned to them while other guest SMMUs have a minimum of
one assigned.
Removing SMMUv3 devices from the VM config described above, I do not
observe the problematic behavior where devices behind PXBs without SMMUs
erroneously use the address space from pcie.0's SMMU. I removed SMMUv3
devices from PXBs with one, two, and three emulated NICs assigned to
them. Below are the guest topology and qemu command used where SMMUv3
devices are excluded from the original test:
nvidia@lego-cg1-dvt-59:~$ lspci -tv
-+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
| +-01.0 Red Hat, Inc. QEMU NVM Express Controller
| +-02.0 Intel Corporation 82540EM Gigabit Ethernet Controller
| +-03.0 Red Hat, Inc. Virtio network device
| +-04.0 Red Hat, Inc. Virtio network device
| +-05.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-06.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-07.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-08.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-09.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0a.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0b.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0c.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0d.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0e.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0f.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-10.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-11.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-12.0 Red Hat, Inc. QEMU PCIe Expander bridge
| \-13.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-[0000:90]-+-00.0-[91]----00.0 Red Hat, Inc. Virtio 1.0 network device
| \-01.0-[92]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:94]---00.0-[95]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:98]---00.0-[99]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:9c]---00.0-[9d]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:a0]---00.0-[a1]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:a4]---00.0-[a5]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:a8]---00.0-[a9]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:ac]---00.0-[ad]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:b0]---00.0-[b1]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:b4]---00.0-[b5]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:b8]---00.0-[b9]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:bc]---00.0-[bd]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:c0]---00.0-[c1]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:c4]---00.0-[c5]----00.0 Red Hat, Inc. Virtio 1.0 network device
\-[0000:c8]-+-00.0-[c9]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-01.0-[ca]----00.0 Red Hat, Inc. Virtio 1.0 network device
\-02.0-[cb]----00.0 Red Hat, Inc. Virtio 1.0 network device
qemu-system-aarch64 \
-machine hmat=on -machine virt,accel=kvm,gic-version=3,ras=on \
-cpu host -smp cpus=4 -m size=16G,slots=4,maxmem=32G -nographic \
-bios /usr/share/AAVMF/AAVMF_CODE.fd \
-device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
-drive
file=/localhome/local-nathanc/noble-server-cloudimg-arm64.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0
\
-device
e1000,romfile=/localhome/local-nathanc/efi-e1000.rom,netdev=net0,bus=pcie.0
\
-netdev
user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586 \
-netdev user,id=net1 \
-netdev user,id=net2 \
-netdev user,id=net3 \
-netdev user,id=net4 \
-netdev user,id=net5 \
-netdev user,id=net6 \
-netdev user,id=net7 \
-netdev user,id=net8 \
-netdev user,id=net9 \
-netdev user,id=net10 \
-netdev user,id=net11 \
-netdev user,id=net12 \
-netdev user,id=net13 \
-netdev user,id=net14 \
-netdev user,id=net15 \
-netdev user,id=net16 \
-netdev user,id=net17 \
-netdev user,id=net18 \
-netdev user,id=net19 \
-netdev user,id=net20 \
-device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0 \
-device virtio-net-pci,bus=pcie.0,netdev=net1 \
-device virtio-net-pci,bus=pcie.0,netdev=net20 \
-device pxb-pcie,id=pcie.1,bus_nr=200,bus=pcie.0 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1,slot=1,chassis=1 \
-device virtio-net-pci,bus=pcie.port1,netdev=net2 \
-device
pcie-root-port,id=pcie.port17,bus=pcie.1,slot=17,chassis=17 \
-device virtio-net-pci,bus=pcie.port17,netdev=net18 \
-device
pcie-root-port,id=pcie.port18,bus=pcie.1,slot=18,chassis=18 \
-device virtio-net-pci,bus=pcie.port18,netdev=net19 \
-device pxb-pcie,id=pcie.2,bus_nr=196,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.2 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,slot=2,chassis=2 \
-device virtio-net-pci,bus=pcie.port2,netdev=net3 \
-device pxb-pcie,id=pcie.3,bus_nr=192,bus=pcie.0 \
-device pcie-root-port,id=pcie.port3,bus=pcie.3,slot=3,chassis=3 \
-device virtio-net-pci,bus=pcie.port3,netdev=net4 \
-device pxb-pcie,id=pcie.4,bus_nr=188,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.4,id=smmuv3.4 \
-device pcie-root-port,id=pcie.port4,bus=pcie.4,slot=4,chassis=4 \
-device virtio-net-pci,bus=pcie.port4,netdev=net5 \
-device pxb-pcie,id=pcie.5,bus_nr=184,bus=pcie.0 \
-device pcie-root-port,id=pcie.port5,bus=pcie.5,slot=5,chassis=5 \
-device virtio-net-pci,bus=pcie.port5,netdev=net6 \
-device pxb-pcie,id=pcie.6,bus_nr=180,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.6,id=smmuv3.6 \
-device pcie-root-port,id=pcie.port6,bus=pcie.6,slot=6,chassis=6 \
-device virtio-net-pci,bus=pcie.port6,netdev=net7 \
-device pxb-pcie,id=pcie.7,bus_nr=176,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.7,id=smmuv3.7 \
-device pcie-root-port,id=pcie.port7,bus=pcie.7,slot=7,chassis=7 \
-device virtio-net-pci,bus=pcie.port7,netdev=net8 \
-device pxb-pcie,id=pcie.8,bus_nr=172,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.8,id=smmuv3.8 \
-device pcie-root-port,id=pcie.port8,bus=pcie.8,slot=8,chassis=8 \
-device virtio-net-pci,bus=pcie.port8,netdev=net9 \
-device pxb-pcie,id=pcie.9,bus_nr=168,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.9,id=smmuv3.9 \
-device pcie-root-port,id=pcie.port9,bus=pcie.9,slot=9,chassis=9 \
-device virtio-net-pci,bus=pcie.port9,netdev=net10 \
-device pxb-pcie,id=pcie.10,bus_nr=164,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.10,id=smmuv3.10 \
-device
pcie-root-port,id=pcie.port10,bus=pcie.10,slot=10,chassis=10 \
-device virtio-net-pci,bus=pcie.port10,netdev=net11 \
-device pxb-pcie,id=pcie.11,bus_nr=160,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.11,id=smmuv3.11 \
-device
pcie-root-port,id=pcie.port11,bus=pcie.11,slot=11,chassis=11 \
-device virtio-net-pci,bus=pcie.port11,netdev=net12 \
-device pxb-pcie,id=pcie.12,bus_nr=156,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.12,id=smmuv3.12 \
-device
pcie-root-port,id=pcie.port12,bus=pcie.12,slot=12,chassis=12 \
-device virtio-net-pci,bus=pcie.port12,netdev=net13 \
-device pxb-pcie,id=pcie.13,bus_nr=152,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.13,id=smmuv3.13 \
-device
pcie-root-port,id=pcie.port13,bus=pcie.13,slot=13,chassis=13 \
-device virtio-net-pci,bus=pcie.port13,netdev=net14 \
-device pxb-pcie,id=pcie.14,bus_nr=148,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.14,id=smmuv3.14 \
-device
pcie-root-port,id=pcie.port14,bus=pcie.14,slot=14,chassis=14 \
-device virtio-net-pci,bus=pcie.port14,netdev=net15 \
-device pxb-pcie,id=pcie.15,bus_nr=144,bus=pcie.0 \
-device
pcie-root-port,id=pcie.port15,bus=pcie.15,slot=15,chassis=15 \
-device virtio-net-pci,bus=pcie.port15,netdev=net16 \
-device
pcie-root-port,id=pcie.port16,bus=pcie.15,slot=16,chassis=16 \
-device virtio-net-pci,bus=pcie.port16,netdev=net17
[0]
https://lore.kernel.org/all/90957693-dc12-4731-960f-0ee295d29...@nvidia.com/
Thanks,
Nathan
