Hi Jegadish! :) Thanks a lot for taking a look! On paper in Python, the sysfs writes look totally synchronous and clean. But on real Mellanox silicon, the card's firmware is running its own race behind the scenes :)
Here is what happens inside `mlx5_core`: 1. When Netplan writes `sriov_numvfs` with autoprobe=1, the kernel immediately probes each VF in legacy mode, and the card firmware starts allocating hardware queues. 2. Netplan then calls `unbind`, and right after that flips devlink to `switchdev`. For the Mellanox ASIC, switching e-switch mode is a heavy job (rebuilding internal switching tables and creating representors). 3. When `netplan-sriov-rebind` immediately calls `bind`, the driver tries to send commands to the firmware command mailbox. But the firmware is still choking on the mode switch, so the driver semaphore hits a timeout: `-ETIMEDOUT` (-110). Flipping `sriov_drivers_autoprobe=0` before writing `sriov_numvfs` saves the day completely: - VFs spawn "cold" without any driver probing. - The card firmware does not waste cycles spinning up legacy queues just to kill them a millisecond later. - Devlink switches to `switchdev` on clean, idle VFs. - Then the rebind service binds them once and it succeeds right away without any firmware lockups. About Testflinger: the physical ConnectX-6 card here is on @zabaj1's box, while I tested the driver sequence on my local testbed. Do you have a bare-metal node with a Mellanox CX5 or CX6 in Canonical's Testflinger pool? That would be awesome to verify it together ;) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2166228 Title: Mellanox NIC is unable to enter switchdev mode To manage notifications about this bug go to: https://bugs.launchpad.net/netplan/+bug/2166228/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
