Thanks. By the way, also confirming that the patch works here (have been running the patched kernel for 3 days now). Running on hardware Super Micro X9SCL with an Intel 82574L NIC.
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1730550 Title: e1000e in 4.4.0-97-generic breaks 82574L under heavy load. Status in linux package in Ubuntu: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Bug description: == SRU Justification == This issue was first reported on the netdev email list by Lennart Sorensen: https://www.mail-archive.com/netdev@vger.kernel.org/msg178170.html Commit 16ecba59bc333d6282ee057fb02339f77a880beb causes link drops on the 82574L under heavy load. "Unfortunately this commit changed the driver to assume that the Other Causes interrupt can only mean link state change and hence sets the flag that (unfortunately) means both link is down and link state should be checked. Since this now happens 3000 times per second, the chances of it happening while the watchdog_task is checking the link state becomes pretty high, and it if does happen to coincice, then the watchdog_task will reset the adapter, which causes a real loss of link." The original reported experienced this issue on a Supermicro X7SPA-HF-D525 server board. However, the bug is now seen on many servers running X9DBL-1F server boards. This bug is fixed by commits 19110cfbb34 and 4aea7a5c5e9, which were both added to mainline in v4.15-rc1. The commit that introduced this bug,16ecba5, was added to mainlien in v4.5-rc1. However, Xenial recived this commit as well as commit 531ff577a. Bionic master-next does not need these commits, since it got them via bug 1735843 and the 4.14.3 updates. == Fixes == 19110cfbb34 ("e1000e: Separate signaling for link check/link up") 4aea7a5c5e9 ("e1000e: Avoid receiver overrun interrupt bursts") == Regression Potential == These commits are specific to e1000. == Test Case == A test kernel was built with these patches and tested by the original bug reporter. The bug reporter states the test kernel resolved the bug. == Original Bug Descriptio == This issue was first reported on the netdev email list by Lennart Sorensen: https://www.mail-archive.com/netdev@vger.kernel.org/msg178170.html Commit 16ecba59bc333d6282ee057fb02339f77a880beb causes link drops on the 82574L under heavy load. "Unfortunately this commit changed the driver to assume that the Other Causes interrupt can only mean link state change and hence sets the flag that (unfortunately) means both link is down and link state should be checked. Since this now happens 3000 times per second, the chances of it happening while the watchdog_task is checking the link state becomes pretty high, and it if does happen to coincice, then the watchdog_task will reset the adapter, which causes a real loss of link." A fix for this issue was accepted into the net-next branch, along with other e1000e/igb patches: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net- next.git/commit/?id=f44dea3421b47d355a835e9cfcc59ca7318575a9 The original reported experienced this issue on a Supermicro X7SPA- HF-D525 server board. We see this issue on many servers running X9DBL- 1F server boards. Both boards use the Intel 82574L for the network interfaces. We see messages like this under heavy load: [Nov 6 15:42] e1000e: eth0 NIC Link is Down [ +0.001670] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [Nov 6 16:10] e1000e: eth0 NIC Link is Down [ +0.008505] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [Nov 7 00:49] e1000e: eth0 NIC Link is Down [ +2.235111] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx We have confirmed that the connected switch sees the link drops also, to these are not false alarms from the e1000e driver. # lsb_release -rd Description: Ubuntu 16.04.2 LTS Release: 16.04 I could not cleanly apply the net-next patch to 4.4.0 so I tested with just the following cherry picked changes on the latest 4.4.0 kernel source package. https://patchwork.ozlabs.org/patch/823942/ https://patchwork.ozlabs.org/patch/823945/ https://patchwork.ozlabs.org/patch/823940/ https://patchwork.ozlabs.org/patch/823941/ https://patchwork.ozlabs.org/patch/823939/ Although it's my understanding the first two are the critical ones for the race condition. I have been running with the patches e1000e kernel driver, under network load for 7 days and I no longer see the network interface drops. Could we pull these changes into the Ubuntu 4.4.0 kernel ? Thanks --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 19 07:34 seq crw-rw---- 1 root audio 116, 33 Jul 19 07:34 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.10 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=49ca52b8-cf08-4485-b296-0dffb098e557 IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Supermicro X9DBL-3F/X9DBL-iF Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_GB.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic root=UUID=957d7126-5452-4606-942d-1d58adbeb253 ro net.ifnames=0 biosdevname=0 quiet splash nomdmonddf nomdmonisw ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 RelatedPackageVersions: linux-restricted-modules-4.4.0-83-generic N/A linux-backports-modules-4.4.0-83-generic N/A linux-firmware 1.157.11 RfKill: Error: [Errno 2] No such file or directory Tags: xenial xenial Uname: Linux 4.4.0-83-generic x86_64 UnreportableReason: The report belongs to a package that is not installed. UpgradeStatus: Upgraded to xenial on 2016-12-05 (337 days ago) UserGroups: _MarkForUpload: False dmi.bios.date: 12/28/2012 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.00 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: X9DBL-3F/X9DBL-iF dmi.board.vendor: Supermicro dmi.board.version: 0123456789 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 3 dmi.chassis.vendor: Supermicro dmi.chassis.version: 0123456789 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.00:bd12/28/2012:svnSupermicro:pnX9DBL-3F/X9DBL-iF:pvr0123456789:rvnSupermicro:rnX9DBL-3F/X9DBL-iF:rvr0123456789:cvnSupermicro:ct3:cvr0123456789: dmi.product.name: X9DBL-3F/X9DBL-iF dmi.product.version: 0123456789 dmi.sys.vendor: Supermicro To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1730550/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
