Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 382.545090] watchdog:
BUG: soft lockup - CPU#1 stuck for 21s! [ksoftirqd/1:17]
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] Modules
linked in: nls_iso8859_1 xt_conntrack nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 hv_balloon xt_owner xt_tcpudp serio_raw iptable_security
bpfilter joydev sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs
zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_core
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel
crypto_simd hid_hyperv cryptd pata_acpi glue_helper hyperv_keyboard hyperv_fb
hid hv_netvsc hv_utils
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] CPU: 1 PID:
17 Comm: ksoftirqd/1 Not tainted 5.4.0-1088-azure #93
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] Hardware
name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008
12/07/2018
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] RIP:
0010:find_next_and_bit+0x1c/0x70
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] Code: 47 c2
c3 c3 66 2e 0f 1f 84 00 00 00 00 00 48 89 d0 48 39 ca 76 63 49 89 c8 49 c1 e8
06 4a 8b 14 c7 48 85 f6 74 04 4a 23 14 c6 <49> c7 c0 ff ff ff ff 49 d3 e0 48 83
e1 c0 49 21 d0 75 2d 48 83 c1
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] RSP:
0018:ffffb409cc6fbac8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff12
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] RAX:
0000000000000040 RBX: ffffb409cc6fbb28 RCX: 0000000000000001
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] RDX:
0000000000000001 RSI: ffff89239f429050 RDI: ffff89239d4d44e0
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] RBP:
ffffb409cc6fbad0 R08: 0000000000000000 R09: 0000000000005000
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] R10:
00000000ffffffff R11: 0000000000000002 R12: ffffb409cc6fbd28
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] R13:
0000000000000003 R14: ffff89239d4d44c0 R15: 0000000000000000
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] FS:
0000000000000000(0000) GS:ffff89239fa40000(0000) knlGS:0000000000000000
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] CS: 0010 DS:
0000 ES: 0000 CR0: 0000000080050033
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] CR2:
00007efe0d000000 CR3: 000000106e436000 CR4: 00000000003506e0
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] Call Trace:
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] ?
cpumask_next_and+0x1e/0x20
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
update_sd_lb_stats+0x14a/0x7d0
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
find_busiest_group+0x49/0x520
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
load_balance+0x16f/0xaf0
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] ?
wake_up_process+0x15/0x20
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
rebalance_domains+0x268/0x310
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
run_rebalance_domains+0x82/0xb0
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
__do_softirq+0xcb/0x2b5
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
run_ksoftirqd+0x1e/0x30
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
smpboot_thread_fn+0xd0/0x170
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
kthread+0x104/0x140
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] ?
sort_range+0x30/0x30
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461] ?
kthread_park+0x90/0x90
Jul 22 19:47:31 selfprovisioned-rtg-bionic kernel: [ 390.001461]
ret_from_fork+0x35/0x40
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] watchdog:
BUG: soft lockup - CPU#0 stuck for 21s! [swapper/0:0]
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] Modules
linked in: nls_iso8859_1 xt_conntrack nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 hv_balloon xt_owner xt_tcpudp serio_raw iptable_security
bpfilter joydev sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs
zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_core
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel
crypto_simd hid_hyperv cryptd pata_acpi glue_helper hyperv_keyboard hyperv_fb
hid hv_netvsc hv_utils
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] CPU: 0 PID: 0
Comm: swapper/0 Tainted: G L 5.4.0-1088-azure #93
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] Hardware
name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008
12/07/2018
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] RIP:
0010:__do_softirq+0x72/0x2b5
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] Code: 01 00
00 89 75 a8 0f b7 f2 89 f1 c7 45 ac 0a 00 00 00 48 89 45 c0 48 89 45 b0 65 66
c7 05 94 de 82 54 00 00 fb b8 ff ff ff ff <48> c7 c3 c0 50 40 ac 41 89 cf 0f bc
c1 89 c6 83 c6 01 89 75 d4 74
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] RSP:
0018:ffffb409c0003f60 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff12
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] RAX:
00000000ffffffff RBX: 0000000000000000 RCX: 0000000000000282
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] RDX:
0000000000000282 RSI: 0000000000000282 RDI: ffff89239da617c0
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] RBP:
ffffb409c0003fb8 R08: 0000000000000000 R09: 0000005ac338d498
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] R10:
0000005ac2fbcb98 R11: 0000000003a147e1 R12: 0000000000000000
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] R13:
0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.214339] FS:
0000000000000000(0000) GS:ffff89239fa00000(0000) knlGS:0000000000000000
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] CS: 0010 DS:
0000 ES: 0000 CR0: 0000000080050033
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] CR2:
00007ffcfdcaf998 CR3: 000000106ec98000 CR4: 00000000003506f0
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] Call Trace:
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] <IRQ>
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014]
irq_exit+0xdf/0xf0
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014]
hv_stimer0_vector_handler+0x66/0x80
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014]
hv_stimer0_callback_vector+0xf/0x20
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] </IRQ>
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] RIP:
0010:default_idle+0x29/0x150
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] Code: 90 0f
1f 44 00 00 55 48 89 e5 41 55 41 54 65 44 8b 25 3b 41 bc 54 53 0f 1f 44 00 00
e9 07 00 00 00 0f 00 2d 1b 52 5b 00 fb f4 <65> 44 8b 25 1f 41 bc 54 0f 1f 44 00
00 5b 41 5c 41 5d 5d c3 65 8b
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] RSP:
0018:ffffffffac403e20 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] RAX:
ffffffffab451200 RBX: 0000000000000000 RCX: 0000000000000001
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] RDX:
0000000000000000 RSI: ffffffffac403df8 RDI: 0000005abf2b3b98
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] RBP:
ffffffffac403e38 R08: 0000000000000001 R09: 0000000000000102
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] R10:
0000000000000010 R11: 0000000000000005 R12: 0000000000000000
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.502014] R13:
ffffffffac413780 R14: 0000000000000000 R15: 0000000000000000
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189] ?
__cpuidle_text_start+0x8/0x8
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
arch_cpu_idle+0x15/0x20
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
default_idle_call+0x23/0x30
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
do_idle+0x1f5/0x250
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
cpu_startup_entry+0x20/0x30
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
rest_init+0xae/0xb0
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
arch_call_rest_init+0xe/0x1b
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
start_kernel+0x536/0x557
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
x86_64_start_reservations+0x24/0x26
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
x86_64_start_kernel+0x75/0x79
Jul 22 19:47:52 selfprovisioned-rtg-bionic kernel: [ 410.566189]
secondary_startup_64+0xa4/0xb0
** Also affects: linux-azure (Ubuntu Focal)
Importance: Undecided
Status: New
** Changed in: linux-azure (Ubuntu Focal)
Importance: Undecided => High
** Changed in: linux-azure (Ubuntu Focal)
Status: New => In Progress
** Changed in: linux-azure (Ubuntu Focal)
Assignee: (unassigned) => Tim Gardner (timg-tpi)
** Changed in: linux-azure (Ubuntu)
Status: New => Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1982613
Title:
Azure: multi-MSI patches break fio tests on NVMe
Status in linux-azure package in Ubuntu:
Fix Released
Status in linux-azure source package in Focal:
In Progress
Bug description:
SRU Justification
[Impact]
Recent additions to focal/azure (5.4.0-1087.92) to support multi-MSI
on the Azure hypervisor break fio tests on NVMe disks by causing a
hung thread. The instance type used to demonstrate the failure was
Standard_L64s_v2.
The initial fix is to revert the 4 patches.
06ad0de407e7ab357f5be5c0dc2290e39ddbf936 PCI: hv: Fix interrupt mapping for
multi-MSI
6f7265bdd6161702cc1530ce7a2d126c49977d3d PCI: hv: Reuse existing IRTE
allocation in compose_msi_msg()
c6a7a02229c2acd4e19602f7b2fcd95ea163eada PCI: hv: Fix hv_arch_irq_unmask()
for multi-MSI
9dc3b7f1d164efda04c4756002011e2ddc9f1c73 PCI: hv: Fix multi-MSI to allow more
than one MSI vector
[Test Plan]
Run the attached reproducer script.
[Where things could go wrong]
Focal will not have support for multi-MSI, but since this is such a
new feature there is likely little consequence.
[Other]
SF: #00339521
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1982613/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
