Here is the reply from Mellanox support :
==================
• There is a limitations on the firmware/driver for ConnectX-5 Socket direct
adapter
• There is a limitation on the number of total PFs+VFs that wecan
allocate on the adapter which is 256.
• The max number of VF's of each port/PF is 63 VF's => 63*4=252 .
Currently on your setup NUM_PF_MSIX==63. for all PFs it is 252.
In order to bring up the max supported number of VF'S=63 , please reduce the
NUM_PF_MSIX to 32
- mlxconfig -d /dev/mst/mt4119_pciconf0 set NUM_PF_MSIX=32
- mlxconfig -d /dev/mst/mt4119_pciconf0.1 set NUM_PF_MSIX=32
- mlxconfig -d /dev/mst/mt4119_pciconf1 set NUM_PF_MSIX=32
- mlxconfig -d /dev/mst/mt4119_pciconf1.1 set NUM_PF_MSIX=32
==================
And indeed, setting NUM_PF_MSIX to 32 allowed me to create 4*63 VFs.
** Changed in: linux (Ubuntu)
Status: Confirmed => Invalid
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1821345
Title:
Mellanox MT27800 / mlx5_core : cannot bring up VFs if the total number
of VFs is >= 64 : alloc irq vectors failed
Status in linux package in Ubuntu:
Invalid
Bug description:
Hi,
For the Mellanox NIC MT27800 (MCX556M-ECAT-S25 to be precise), the
maximum number of VF per PF can be configured in the firmware, by
running e.g. :
$ sudo mstconfig -d 32:00.1 set NUM_OF_VFS=127
If I set this NUM_OF_VFS to 64 or more, the creation of even a single
VF fails mid-way :
$ echo 1 | sudo tee /sys/class/net/enp50s0f0/device/sriov_numvfs
1
dmesg shows :
[101418.090970] (0000:32:00.0): E-Switch: E-Switch enable SRIOV: nvfs(1) mode
(1)
[101418.182736] (0000:32:00.0): E-Switch: SRIOV enabled: active vports(2)
[101418.289642] pci 0000:32:00.2: [15b3:1018] type 00 class 0x020000
[101418.290235] pci 0000:32:00.2: Max Payload Size set to 512 (was 128, max
512)
[101418.290261] pci 0000:32:00.2: enabling Extended Tags
[101418.292432] iommu: Adding device 0000:32:00.2 to group 139
[101418.292580] iommu: Using direct mapping for device 0000:32:00.2
[101418.292979] mlx5_core 0000:32:00.2: enabling device (0000 -> 0002)
[101418.293719] mlx5_core 0000:32:00.2: firmware version: 16.24.1000
[101418.550589] mlx5_core 0000:32:00.2: alloc irq vectors failed
[101418.974082] mlx5_core 0000:32:00.2: mlx5_load_one failed with error code
-28
[101418.981656] mlx5_core: probe of 0000:32:00.2 failed with error -28
This is a problem with both the Ubuntu kernel driver and the OFED
driver.
Thanks
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-4.15.0-46-generic 4.15.0-46.49
ProcVersionSignature: User Name 4.15.0-46.49-generic 4.15.18
Uname: Linux 4.15.0-46-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Mar 21 08:33 seq
crw-rw---- 1 root audio 116, 33 Mar 21 08:33 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord':
'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
Date: Fri Mar 22 12:43:26 2019
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Supermicro AS -2023US-TR4
PciMultimedia:
ProcEnviron:
TERM=screen-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-46-generic
root=UUID=52d72a05-517b-4c16-8dcb-486bdebcc695 ro console=ttyS1,115200 nosplash
iommu=pt amd_iommu=on processor.max_cstate=0 l1tf=full cpuidle.off=1
RelatedPackageVersions:
linux-restricted-modules-4.15.0-46-generic N/A
linux-backports-modules-4.15.0-46-generic N/A
linux-firmware 1.173.3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/04/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.1c
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: H11DSU-iN
dmi.board.vendor: Supermicro
dmi.board.version: 1.02A
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 1
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias:
dmi:bvnAmericanMegatrendsInc.:bvr1.1c:bd10/04/2018:svnSupermicro:pnAS-2023US-TR4:pvr0123456789:rvnSupermicro:rnH11DSU-iN:rvr1.02A:cvnSupermicro:ct1:cvr0123456789:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: AS -2023US-TR4
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1821345/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
