You have been subscribed to a public bug:
After upgrading 3 servers from linux-image-5.3.0-40-generic to linux-
image-5.4.0-48-generic I have started seeing the following queue
timeouts from IP-over-Infiniband (ipoib) devices. The devices in
question are (with newest available firmware, 2.42.5000):
# lspci -nnk -s 83:00.0
83:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family
[ConnectX-3] [15b3:1003]
Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:0027]
Kernel driver in use: mlx4_core
Below is the WARN from one machine's syslog. The others are practically
identical. When the WARN happens on any of the machines, other 2 will
_also_ exhibit queue timeouts. Additionally, other (unrelated) machines
connected to the same infiniband fabric will exhibit a 12-second
transmission delay. This could conceivably be caused by these 3 servers
also being Subnet Managers (opensm package).
The infiniband fabric is partitioned, with the affected partition (8011)
seeing most of the traffic.
--------------------
kernel: [52642.480066] ------------[ cut here ]------------
kernel: [52642.480092] NETDEV WATCHDOG: ib0.8011 (): transmit queue 0 timed out
kernel: [52642.480120] WARNING: CPU: 13 PID: 0 at
/build/linux-hwe-5.4-8m2I8l/linux-hwe-5.4-5.4.0/net/sched/sch_generic.c:448
dev_watchdog+0x264/0x270
kernel: [52642.480121] Modules linked in: aufs overlay ip6table_raw
ip6table_mangle ip6table_nat iptable_raw iptable_mangle iptable_nat nf_tables
nfnetlink cfg80211 ebtable_filter ebtables ip6table_filter ip6_tables
iptable_filter bpfilter mst_pciconf(OE) mst_pci(OE) 8021q garp mrp stp llc
nls_iso8859_1 intel_rapl_msr lz4 lz4_compress intel_rapl_common ib_iser rdma_cm
sb_edac iw_cm iscsi_tcp libiscsi_tcp libiscsi x86_pkg_temp_thermal
scsi_transport_iscsi intel_powerclamp zram veth vhost_net tap coretemp vhost
kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel kvm openvswitch nsh
nf_conncount nf_nat nf_conntrack rapl nf_defrag_ipv6 nf_defrag_ipv4 joydev
input_leds intel_cstate ib_ipoib mei_me mei ib_cm ioatdma ib_umad lpc_ich
acpi_pad acpi_power_meter mac_hid ipmi_si ipmi_ssif ipmi_devintf
ipmi_msghandler kyber_iosched sch_fq_codel tcp_highspeed ip_tables x_tables
autofs4 zfs(POE) zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE)
spl(OE) zlua(POE) btrfs zstd_compress raid10 raid456
kernel: [52642.480182] async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid0 multipath linear dm_mirror dm_region_hash
dm_log mlx4_ib ib_uverbs ib_core hid_generic raid1 ses enclosure usbhid hid ast
drm_vram_helper i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect
sysimgblt aesni_intel fb_sys_fops ixgbe glue_helper mpt3sas nvme xfrm_algo
crypto_simd ahci raid_class dca cryptd mlx4_core drm megaraid_sas nvme_core
libahci scsi_transport_sas mdio wmi
kernel: [52642.480221] CPU: 13 PID: 0 Comm: swapper/13 Tainted: P OE
5.4.0-48-generic #52~18.04.1-Ubuntu
kernel: [52642.480223] Hardware name: Supermicro Super Server/X10DRW-iT, BIOS
2.0b 04/13/2017
kernel: [52642.480226] RIP: 0010:dev_watchdog+0x264/0x270
kernel: [52642.480229] Code: 48 85 c0 75 e6 eb a0 4c 89 ef c6 05 42 c1 e7 00 01
e8 30 b8 fa ff 89 d9 48 89 c2 4c 89 ee 48 c7 c7 50 05 63 ae e8 4c 31 71 ff <0f>
0b eb 82 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41
kernel: [52642.480230] RSP: 0018:ffffb4998c970e48 EFLAGS: 00010282
kernel: [52642.480233] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
000000000000083f
kernel: [52642.480234] RDX: 0000000000000000 RSI: 00000000000000f6 RDI:
000000000000083f
kernel: [52642.480235] RBP: ffffb4998c970e78 R08: 00000000000008fd R09:
0000000000000003
kernel: [52642.480237] R10: ffffb4998c970ee8 R11: 0000000000000001 R12:
0000000000000001
kernel: [52642.480238] R13: ffff93d302465000 R14: ffff93d302465480 R15:
ffff93f2d293c880
kernel: [52642.480240] FS: 0000000000000000(0000) GS:ffff93f33f640000(0000)
knlGS:0000000000000000
kernel: [52642.480242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [52642.480243] CR2: fffff90140127000 CR3: 0000002a26e0a004 CR4:
00000000001626e0
kernel: [52642.480245] Call Trace:
kernel: [52642.480247] <IRQ>
kernel: [52642.480252] ? pfifo_fast_reset+0x110/0x110
kernel: [52642.480255] call_timer_fn+0x32/0x130
kernel: [52642.480258] run_timer_softirq+0x443/0x480
kernel: [52642.480262] ? ktime_get+0x43/0xa0
kernel: [52642.480268] ? lapic_next_deadline+0x26/0x30
kernel: [52642.480273] __do_softirq+0xe4/0x2da
kernel: [52642.480278] irq_exit+0xae/0xb0
kernel: [52642.480282] smp_apic_timer_interrupt+0x79/0x130
kernel: [52642.480285] apic_timer_interrupt+0xf/0x20
kernel: [52642.480286] </IRQ>
kernel: [52642.480292] RIP: 0010:cpuidle_enter_state+0xbc/0x440
kernel: [52642.480294] Code: ff e8 98 f6 80 ff 80 7d d3 00 74 17 9c 58 0f 1f 44
00 00 f6 c4 02 0f 85 54 03 00 00 31 ff e8 2b 74 87 ff fb 66 0f 1f 44 00 00 <45>
85 ed 0f 88 1a 03 00 00 4c 2b 7d c8 48 ba cf f7 53 e3 a5 9b c4
kernel: [52642.480295] RSP: 0018:ffffb4998c5afe48 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffff13
kernel: [52642.480298] RAX: ffff93f33f66ad00 RBX: ffffffffae9579e0 RCX:
000000000000001f
kernel: [52642.480299] RDX: 00002fe0c852cf32 RSI: 0000000037a6f80a RDI:
0000000000000000
kernel: [52642.480300] RBP: ffffb4998c5afe88 R08: 0000000000000002 R09:
000000000002a580
kernel: [52642.480302] R10: ffffb4998c5afe18 R11: 0000000000002171 R12:
ffffd4997f844500
kernel: [52642.480303] R13: 0000000000000004 R14: ffffffffae957b78 R15:
00002fe0c852cf32
kernel: [52642.480307] ? cpuidle_enter_state+0x98/0x440
kernel: [52642.480311] cpuidle_enter+0x2e/0x40
kernel: [52642.480316] call_cpuidle+0x23/0x40
kernel: [52642.480319] do_idle+0x1f6/0x270
kernel: [52642.480323] cpu_startup_entry+0x1d/0x20
kernel: [52642.480326] start_secondary+0x166/0x1c0
kernel: [52642.480331] secondary_startup_64+0xa4/0xb0
kernel: [52642.480334] ---[ end trace 770c6aafc2e53202 ]---
kernel: [52642.480339] ib0.8011: transmit timeout: latency 1420 msecs
kernel: [52642.480343] ib0.8011: queue stopped 1, tx_head 30, tx_tail 30,
global_tx_head 50968261, global_tx_tail 50968133
kernel: [52642.608061] ib0: transmit timeout: latency 1512 msecs
kernel: [52642.608081] ib0: queue stopped 1, tx_head 1293, tx_tail 1293,
global_tx_head 38440435, global_tx_tail 38440307
--------------------
# apt-cache policy linux-image-5.4.0-48-generic
linux-image-5.4.0-48-generic:
Installed: 5.4.0-48.52~18.04.1
# lsb_release -rd
Description: Ubuntu 18.04.5 LTS
Release: 18.04
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
--
Infiniband transmit queue timeouts after upgrading to linux-hwe-5.4
https://bugs.launchpad.net/bugs/1898057
You received this bug notification because you are a member of Kernel Packages,
which is subscribed to linux in Ubuntu.
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
