The bug was verified in Xenial kernel 4.4.0-142.168 available in -proposed.
I'm running in an AWS 2-cpu instance, which exhibits the issue if we run a small reproducer script (a loop that basically changes IRQ affinity for the NVMe MSIs/legacy interrupt among the CPUs and performs a 4K write to the device + sync): $ uptime 17:42:11 up 4:00, 2 users, load average: 0.19, 0.14, 0.08 $ uname -rmv 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 $ dmesg [ 2218.252634] nvme 0000:00:04.0: I/O 6 QID 2 timeout, completion polled [ 2451.245962] nvme 0000:00:04.0: I/O 22 QID 2 timeout, completion polled [ 6672.249406] nvme 0000:00:04.0: I/O 3 QID 1 timeout, completion polled [ 8425.253863] nvme 0000:00:04.0: I/O 28 QID 2 timeout, completion polled <notice I've cleared the dmesg after the boot> Cheers, Guilherme ** Tags removed: verification-needed-xenial ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1807393 Title: nvme - Polling on timeout Status in linux package in Ubuntu: In Progress Status in linux source package in Xenial: Fix Committed Bug description: [Impact] * NVMe controllers potentially could miss to send an interrupt, specially due to bugs in virtual devices(which are common those days - qemu has its own NVMe virtual device, so does AWS). This would be a difficult to debug situation, because NVMe driver only reports the request timeout, not the reason. * The upstream patch proposed to SRU here here, 7776db1ccc12 ("NVMe/pci: Poll CQ on timeout") was designed to provide more information in these cases, by pro-actively polling the CQEs on request timeouts, to check if the specific request was completed and some issue (probably a missed interrupt) prevented the driver to notice, or if the request really wasn't completed, which indicates more severe issues. * Although quite useful for debugging, this patch could help to mitigate issues in cloud environments like AWS, in case we may have jitter in request completion and the i/o timeout was set to low values, or even in case of atypical bugs in the virtual NVMe controller. With this patch, if polling succeeds the NVMe driver will continue working instead of trying a reset controller procedure, which may lead to fails in the rootfs - refer to https://launchpad.net/bugs/1788035. [Test Case] * It's a bit tricky to artificially create a situation of missed interrupt; one idea was to implement a small hack in the NVMe qemu virtual device that given a trigger in guest kernel, will induce the virtual device to skip an interrupt. The hack patch is present in comment #1 below. * To trigger such hack from guest kernel, all is needed is to issue a raw admin command from nvme-cli: "nvme admin-passthru -o 0xff /dev/nvme0" After that, just perform some I/Os to see one of them aborting - one could use dd for this goal, like "dd if=/dev/zero of=/dev/nvme0n1 count=5". [Regression Potential] * There are no clear risks in adding such polling mechanism to the NVMe driver; one bad thing that was neverreported but could happen with this patch is the device could be in a bad state IRQ-wise that a reset would fix, but the patch could cause all requests to be completed via polling, which prevents the adapter reset. This is however a very hypothetical situation, which would also happen in the mainline kernel (since it has the patch). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1807393/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
