I have a Ubuntu VM (4.15.0-48-generic) to which I pass through a PCI device,
specifically a virtual NVMe controller. The problem I have is that only one I/O
queue
is initialized, while there should be more (e.g. four). I'm using upstream QEMU
v4.1.0 confgiured without any additional options. Most likely there's something
broken with my virtual device implementation but I can't figure out exactly
what, I was hoping to get some debugging directions.
I run QEMU as follows:
~/src/qemu/x86_64-softmmu/qemu-system-x86_64 \
-kernel bionic-server-cloudimg-amd64-vmlinuz-generic \
-smp cores=2,sockets=2 \
-nographic \
-append "console=ttyS0 root=/dev/sda1 single nvme.sgl_threshold=0
nokaslr nvme.io_queue_depth=4" \
-initrd bionic-server-cloudimg-amd64-initrd-generic \
-hda bionic-server-cloudimg-amd64.raw \
-hdb data.raw \
-m 1G \
-object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1073741824
-numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
-device
vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000000 \
-trace enable=vfio*,file=qemu.trace \
-net none \
-s
This is what QEMU thinks of the devices:
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 8086:1237
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
ISA bridge: PCI device 8086:7000
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 1:
IDE controller: PCI device 8086:7010
PCI subsystem 1af4:1100
BAR4: I/O at 0xc000 [0xc00f].
id ""
Bus 0, device 1, function 3:
Bridge: PCI device 8086:7113
PCI subsystem 1af4:1100
IRQ 9.
id ""
Bus 0, device 2, function 0:
VGA controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
BAR2: 32 bit memory at 0xfebf4000 [0xfebf4fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
id ""
Bus 0, device 3, function 0:
Class 0264: PCI device 4e58:0001
PCI subsystem 0000:0000
IRQ 11.
BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
id ""
And this is what the guest thinks of the device in question:
root@ubuntu:~# lspci -vvv -s 00:03.0
00:03.0 Non-Volatile memory controller: Device 4e58:0001 (prog-if 02 [NVM
Express])
Physical Slot: 3
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 24
NUMA node: 0
Region 0: Memory at febf0000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 0
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable- 64bit+
Address: 00000000fee01004 Data: 4023
Capabilities: [60] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
L1 <1us
ExtTag- AttnBtn+ AttnInd- PwrInd- RBE- FLReset-
SlotPowerLimit 128.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed unknown, Width x0, ASPM not supported,
Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: nvme
Kernel modules: nvme
I tried debugging the guest kernel and I think the following if is taken in
pci_msi_domain_check_cap():
if (pci_msi_desc_is_multi_msi(desc) &&
!(info->flags & MSI_FLAG_MULTI_PCI_MSI))
return 1;
because flags is 0x3b (MSI_FLAG_MULTI_PCI_MSI is 0x4). And this I think means
that MSI_FLAG_MULTI_PCI_MSI is not set for that msi_domain_info.
# grep -i msi qemu.trace
1327@1571926064.595365:vfio_msi_setup 00000000-0000-0000-0000-000000000000 PCI
MSI CAP @0x48
1334@1571926073.489691:vfio_msi_enable (00000000-0000-0000-0000-000000000000)
Enabled 1 MSI vectors
1334@1571926073.501741:vfio_msi_disable (00000000-0000-0000-0000-000000000000)
1334@1571926073.507127:vfio_msi_enable (00000000-0000-0000-0000-000000000000)
Enabled 1 MSI vectors
1327@1571926073.520840:vfio_msi_interrupt
(00000000-0000-0000-0000-000000000000) vector 0 0xfee01004/0x4023
... more vfio_msi_interrupt ...
How can I further debug this?
