Public bug reported:
SRU Justification
[Impact]
Below are the kernel messages showing the VF being removed at 12:21:35,
and then being re-added starting at 12:22:44. You can see the stack
traces trying to make an order 7 allocation, and then and order 8
allocation, both of which fail.
[Sun Jan 23 12:21:34 2022] infiniband mlx5_0: wait_for_async_commands:745:(pid
15782): done with all pending requests
[Sun Jan 23 12:21:35 2022] hv_netvsc 000d3a7c-6aa5-000d-3a7c-6aa5000d3a7c eth0:
Data path switched from VF: eth1
[Sun Jan 23 12:21:35 2022] hv_netvsc 000d3a7c-6aa5-000d-3a7c-6aa5000d3a7c eth0:
VF unregistering: eth1
[Sun Jan 23 12:22:44 2022] hv_pci 61367817-ab6d-44c7-8b57-457fcc1b66c2: PCI
VMBus probing: Using version 0x10002
[Sun Jan 23 12:22:44 2022] hv_pci 61367817-ab6d-44c7-8b57-457fcc1b66c2: PCI
host bridge to bus ab6d:00
[Sun Jan 23 12:22:44 2022] pci_bus ab6d:00: root bus resource [mem
0xfe0000000-0xfe00fffff window]
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: [15b3:1016] type 00 class 0x020000
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: reg 0x10: [mem
0xfe0000000-0xfe00fffff 64bit pref]
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: 0.000 Gb/s available PCIe
bandwidth, limited by Unknown speed x0 link at ab6d:00:02.0 (capable of 63.008
Gb/s with 8 GT/s x8 link)
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: BAR 0: assigned [mem
0xfe0000000-0xfe00fffff 64bit pref]
[Sun Jan 23 12:22:44 2022] mlx5_core ab6d:00:02.0: firmware version: 14.30.1210
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 24 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 25 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 26 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 27 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 28 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 29 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 30 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 31 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 32 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 33 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 34 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 35 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 36 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 37 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 38 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 39 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 40 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 41 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 42 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 43 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 44 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 45 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 46 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 47 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 48 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 49 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 50 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 51 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 52 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 53 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 54 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 55 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] kworker/18:1: page allocation failure: order:7,
mode:0x80d0
[Sun Jan 23 12:22:45 2022] CPU: 18 PID: 11869 Comm: kworker/18:1 Kdump: loaded
Tainted: G ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
[Sun Jan 23 12:22:45 2022] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[Sun Jan 23 12:22:45 2022] Workqueue: hv_pri_chan vmbus_add_channel_work
[hv_vmbus]
[Sun Jan 23 12:22:45 2022] Call Trace:
[Sun Jan 23 12:22:45 2022] [<ffffffffa67835a9>] dump_stack+0x19/0x1b
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c46c0>] warn_alloc_failed+0x110/0x180
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c925f>]
__alloc_pages_nodemask+0x9df/0xbe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa603346f>]
dma_generic_alloc_coherent+0x8f/0x140
[Sun Jan 23 12:22:45 2022] [<ffffffffa606eb01>]
x86_swiotlb_alloc_coherent+0x21/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffc0763474>]
mlx5_dma_zalloc_coherent_node+0xb4/0x110 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0763d69>] mlx5_buf_alloc_node+0x89/0x120
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0763e14>] mlx5_buf_alloc+0x14/0x20
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075d9bb>] create_map_eq+0xcb/0x3c0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075dd0a>] create_async_eq+0x5a/0x80
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075ea35>]
mlx5_eq_table_create+0x1a5/0x7e0 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffa607649a>] ? ioremap_nocache+0x1a/0x20
[Sun Jan 23 12:22:45 2022] [<ffffffffc075f55d>] ? alloc_uars_page+0x18d/0x2c0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075ddb0>] ?
mlx5_eq_create_generic+0x80/0x80 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc07565d9>] mlx5_load_one+0x599/0xa30
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0756f11>] init_one+0x411/0x5c0 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d6d1a>] local_pci_probe+0x4a/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d8469>] pci_device_probe+0x109/0x160
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb87f>] pci_bus_add_device+0x4f/0xa0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb909>] pci_bus_add_devices+0x39/0x80
[Sun Jan 23 12:22:45 2022] [<ffffffffc028437f>] hv_pci_probe+0x9ef/0xcf0
[pci_hyperv]
[Sun Jan 23 12:22:45 2022] [<ffffffffc024db81>] vmbus_probe+0x41/0xa0 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64ba9b8>] bus_probe_device+0x98/0xd0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b825f>] device_add+0x4ff/0x7c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b853a>] device_register+0x1a/0x20
[Sun Jan 23 12:22:45 2022] [<ffffffffc024e5a6>]
vmbus_device_register+0x66/0x100 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0251e5d>]
vmbus_add_channel_work+0x4cd/0x640 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bde5f>] process_one_work+0x17f/0x440
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bef76>] worker_thread+0x126/0x3c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bee50>] ?
manage_workers.isra.26+0x2a0/0x2a0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5e31>] kthread+0xd1/0xe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ? insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] [<ffffffffa6795df7>]
ret_from_fork_nospec_begin+0x21/0x21
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ? insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] Mem-Info:
[Sun Jan 23 12:22:45 2022] active_anon:30843105 inactive_anon:323158
isolated_anon:0
active_file:377528 inactive_file:381332 isolated_file:0
unevictable:0 dirty:9 writeback:0 unstable:0
slab_reclaimable:182369 slab_unreclaimable:52365
mapped:9123 shmem:1052916 pagetables:61434 bounce:0
free:173238 free_pcp:171 free_cma:0
[Sun Jan 23 12:22:45 2022] Node 0 DMA free:15892kB min:8kB low:8kB high:12kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 775 128751 128751
[Sun Jan 23 12:22:45 2022] Node 0 DMA32 free:512612kB min:404kB low:504kB
high:604kB active_anon:253160kB inactive_anon:160kB active_file:36kB
inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:1032128kB managed:794528kB mlocked:0kB dirty:0kB writeback:0kB
mapped:4kB shmem:444kB slab_reclaimable:11272kB slab_unreclaimable:1972kB
kernel_stack:192kB pagetables:2768kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
no
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 0 127975 127975
[Sun Jan 23 12:22:45 2022] Node 0 Normal free:164448kB min:67168kB low:83960kB
high:100752kB active_anon:123119260kB inactive_anon:1292472kB
active_file:1510076kB inactive_file:1525280kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:133169152kB managed:131050148kB
mlocked:0kB dirty:36kB writeback:0kB mapped:36488kB shmem:4211220kB
slab_reclaimable:718204kB slab_unreclaimable:207472kB kernel_stack:9712kB
pagetables:242968kB unstable:0kB bounce:0kB free_pcp:676kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 0 0 0
[Sun Jan 23 12:22:45 2022] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB
(U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) =
15892kB
[Sun Jan 23 12:22:45 2022] Node 0 DMA32: 242*4kB (UM) 184*8kB (UEM) 121*16kB
(UEM) 69*32kB (UEM) 69*64kB (UEM) 25*128kB (UEM) 11*256kB (UM) 2*512kB (UE)
1*1024kB (U) 1*2048kB (U) 120*4096kB (M) = 512632kB
[Sun Jan 23 12:22:45 2022] Node 0 Normal: 8431*4kB (UEM) 7451*8kB (UEM)
2808*16kB (UEM) 442*32kB (UEM) 104*64kB (UEM) 44*128kB (UEM) 10*256kB (UM)
4*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 170324kB
[Sun Jan 23 12:22:45 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 23 12:22:45 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 23 12:22:45 2022] 1811902 total pagecache pages
[Sun Jan 23 12:22:45 2022] 0 pages in swap cache
[Sun Jan 23 12:22:45 2022] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 23 12:22:45 2022] Free swap = 0kB
[Sun Jan 23 12:22:45 2022] Total swap = 0kB
[Sun Jan 23 12:22:45 2022] 33554318 pages RAM
[Sun Jan 23 12:22:45 2022] 0 pages HighMem/MovableOnly
[Sun Jan 23 12:22:45 2022] 589172 pages reserved
[Sun Jan 23 12:22:45 2022] kworker/18:1: page allocation failure: order:8,
mode:0xc0d0
[Sun Jan 23 12:22:45 2022] CPU: 18 PID: 11869 Comm: kworker/18:1 Kdump: loaded
Tainted: G ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
[Sun Jan 23 12:22:45 2022] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[Sun Jan 23 12:22:45 2022] Workqueue: hv_pri_chan vmbus_add_channel_work
[hv_vmbus]
[Sun Jan 23 12:22:45 2022] Call Trace:
[Sun Jan 23 12:22:45 2022] [<ffffffffa67835a9>] dump_stack+0x19/0x1b
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c46c0>] warn_alloc_failed+0x110/0x180
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c925f>]
__alloc_pages_nodemask+0x9df/0xbe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa6218ea8>] alloc_pages_current+0x98/0x110
[Sun Jan 23 12:22:45 2022] [<ffffffffa61e5ad8>] kmalloc_order+0x18/0x40
[Sun Jan 23 12:22:45 2022] [<ffffffffa6224876>] kmalloc_order_trace+0x26/0xa0
[Sun Jan 23 12:22:45 2022] [<ffffffffa614f93e>] ?
__irq_put_desc_unlock+0x1e/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa6228801>] __kmalloc+0x211/0x230
[Sun Jan 23 12:22:45 2022] [<ffffffffc07734d6>] mlx5_init_fc_stats+0x76/0x1d0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc077231d>] mlx5_init_fs+0x2d/0x840
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0756823>] mlx5_load_one+0x7e3/0xa30
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0756f11>] init_one+0x411/0x5c0 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d6d1a>] local_pci_probe+0x4a/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d8469>] pci_device_probe+0x109/0x160
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb87f>] pci_bus_add_device+0x4f/0xa0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb909>] pci_bus_add_devices+0x39/0x80
[Sun Jan 23 12:22:45 2022] [<ffffffffc028437f>] hv_pci_probe+0x9ef/0xcf0
[pci_hyperv]
[Sun Jan 23 12:22:45 2022] [<ffffffffc024db81>] vmbus_probe+0x41/0xa0 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64ba9b8>] bus_probe_device+0x98/0xd0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b825f>] device_add+0x4ff/0x7c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b853a>] device_register+0x1a/0x20
[Sun Jan 23 12:22:45 2022] [<ffffffffc024e5a6>]
vmbus_device_register+0x66/0x100 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0251e5d>]
vmbus_add_channel_work+0x4cd/0x640 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bde5f>] process_one_work+0x17f/0x440
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bef76>] worker_thread+0x126/0x3c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bee50>] ?
manage_workers.isra.26+0x2a0/0x2a0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5e31>] kthread+0xd1/0xe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ? insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] [<ffffffffa6795df7>]
ret_from_fork_nospec_begin+0x21/0x21
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ? insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] Mem-Info:
[Sun Jan 23 12:22:45 2022] active_anon:30843203 inactive_anon:323158
isolated_anon:0
active_file:377553 inactive_file:380392 isolated_file:0
unevictable:0 dirty:11 writeback:0 unstable:0
slab_reclaimable:182369 slab_unreclaimable:51851
mapped:9232 shmem:1052917 pagetables:61460 bounce:0
free:174818 free_pcp:239 free_cma:0
[Sun Jan 23 12:22:45 2022] Node 0 DMA free:15892kB min:8kB low:8kB high:12kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 775 128751 128751
[Sun Jan 23 12:22:45 2022] Node 0 DMA32 free:512632kB min:404kB low:504kB
high:604kB active_anon:253160kB inactive_anon:160kB active_file:36kB
inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:1032128kB managed:794528kB mlocked:0kB dirty:0kB writeback:0kB
mapped:4kB shmem:444kB slab_reclaimable:11272kB slab_unreclaimable:1972kB
kernel_stack:192kB pagetables:2768kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
no
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 0 127975 127975
[Sun Jan 23 12:22:45 2022] Node 0 Normal free:170748kB min:67168kB low:83960kB
high:100752kB active_anon:123119652kB inactive_anon:1292472kB
active_file:1510176kB inactive_file:1521520kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:133169152kB managed:131050148kB
mlocked:0kB dirty:44kB writeback:0kB mapped:36924kB shmem:4211224kB
slab_reclaimable:718204kB slab_unreclaimable:205416kB kernel_stack:9776kB
pagetables:243072kB unstable:0kB bounce:0kB free_pcp:928kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[Sun Jan 23 12:22:46 2022] lowmem_reserve[]: 0 0 0 0
[Sun Jan 23 12:22:46 2022] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB
(U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) =
15892kB
[Sun Jan 23 12:22:46 2022] Node 0 DMA32: 242*4kB (UM) 184*8kB (UEM) 121*16kB
(UEM) 69*32kB (UEM) 69*64kB (UEM) 25*128kB (UEM) 11*256kB (UM) 2*512kB (UE)
1*1024kB (U) 1*2048kB (U) 120*4096kB (M) = 512632kB
[Sun Jan 23 12:22:46 2022] Node 0 Normal: 9561*4kB (UEM) 7451*8kB (UEM)
2806*16kB (UEM) 437*32kB (UEM) 103*64kB (UEM) 15*128kB (UEM) 9*256kB (UM)
6*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 170620kB
[Sun Jan 23 12:22:46 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 23 12:22:46 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 23 12:22:46 2022] 1810845 total pagecache pages
[Sun Jan 23 12:22:46 2022] 0 pages in swap cache
[Sun Jan 23 12:22:46 2022] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 23 12:22:46 2022] Free swap = 0kB
[Sun Jan 23 12:22:46 2022] Total swap = 0kB
[Sun Jan 23 12:22:46 2022] 33554318 pages RAM
[Sun Jan 23 12:22:46 2022] 0 pages HighMem/MovableOnly
[Sun Jan 23 12:22:46 2022] 589172 pages reserved
[Sun Jan 23 12:22:46 2022] mlx5_core ab6d:00:02.0: Failed to init flow steering
[Sun Jan 23 12:22:46 2022] mlx5_core ab6d:00:02.0: mlx5_load_one failed with
error code -12
[Sun Jan 23 12:22:46 2022] mlx5_core: probe of ab6d:00:02.0 failed with error
-12
[Fix]
26bf30902c10473ba38f220d3401a61c56d8db3b
48f02eef7f764f33e520ed8009d293396ca690cd
2fdeb4f4c2aea53a6a2e8adb69e811cf304a0ae5
38a54cae6f76c3e6a1e6c1e52c2e43a069fa78cb
b247f32aecad09e6cf7edff7739e6f2c9dc5fca9
[Test Case]
Microsoft tested
[Where things might go wrong]
VM allocations could continue to fail
[Other Info]
SF: #00327011
** Affects: linux-azure (Ubuntu)
Importance: Medium
Assignee: Tim Gardner (timg-tpi)
Status: In Progress
** Package changed: linux (Ubuntu) => linux-azure (Ubuntu)
** Changed in: linux-azure (Ubuntu)
Importance: Undecided => Medium
** Changed in: linux-azure (Ubuntu)
Status: New => In Progress
** Changed in: linux-azure (Ubuntu)
Assignee: (unassigned) => Tim Gardner (timg-tpi)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1961632
Title:
linux-azure: Case VM fails to initialize CX4 VF due to mem
fragmentation
Status in linux-azure package in Ubuntu:
In Progress
Bug description:
SRU Justification
[Impact]
Below are the kernel messages showing the VF being removed at
12:21:35, and then being re-added starting at 12:22:44. You can see
the stack traces trying to make an order 7 allocation, and then and
order 8 allocation, both of which fail.
[Sun Jan 23 12:21:34 2022] infiniband mlx5_0:
wait_for_async_commands:745:(pid 15782): done with all pending requests
[Sun Jan 23 12:21:35 2022] hv_netvsc 000d3a7c-6aa5-000d-3a7c-6aa5000d3a7c
eth0: Data path switched from VF: eth1
[Sun Jan 23 12:21:35 2022] hv_netvsc 000d3a7c-6aa5-000d-3a7c-6aa5000d3a7c
eth0: VF unregistering: eth1
[Sun Jan 23 12:22:44 2022] hv_pci 61367817-ab6d-44c7-8b57-457fcc1b66c2: PCI
VMBus probing: Using version 0x10002
[Sun Jan 23 12:22:44 2022] hv_pci 61367817-ab6d-44c7-8b57-457fcc1b66c2: PCI
host bridge to bus ab6d:00
[Sun Jan 23 12:22:44 2022] pci_bus ab6d:00: root bus resource [mem
0xfe0000000-0xfe00fffff window]
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: [15b3:1016] type 00 class
0x020000
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: reg 0x10: [mem
0xfe0000000-0xfe00fffff 64bit pref]
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: 0.000 Gb/s available PCIe
bandwidth, limited by Unknown speed x0 link at ab6d:00:02.0 (capable of 63.008
Gb/s with 8 GT/s x8 link)
[Sun Jan 23 12:22:44 2022] pci ab6d:00:02.0: BAR 0: assigned [mem
0xfe0000000-0xfe00fffff 64bit pref]
[Sun Jan 23 12:22:44 2022] mlx5_core ab6d:00:02.0: firmware version:
14.30.1210
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 24 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 25 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 26 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 27 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 28 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 29 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 30 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 31 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 32 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 33 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 34 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 35 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 36 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 37 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 38 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 39 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 40 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 41 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 42 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 43 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 44 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 45 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 46 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 47 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 48 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 49 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 50 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 51 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 52 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 53 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 54 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] mlx5_core ab6d:00:02.0: irq 55 for MSI/MSI-X
[Sun Jan 23 12:22:45 2022] kworker/18:1: page allocation failure: order:7,
mode:0x80d0
[Sun Jan 23 12:22:45 2022] CPU: 18 PID: 11869 Comm: kworker/18:1 Kdump:
loaded Tainted: G ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
[Sun Jan 23 12:22:45 2022] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[Sun Jan 23 12:22:45 2022] Workqueue: hv_pri_chan vmbus_add_channel_work
[hv_vmbus]
[Sun Jan 23 12:22:45 2022] Call Trace:
[Sun Jan 23 12:22:45 2022] [<ffffffffa67835a9>] dump_stack+0x19/0x1b
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c46c0>] warn_alloc_failed+0x110/0x180
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c925f>]
__alloc_pages_nodemask+0x9df/0xbe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa603346f>]
dma_generic_alloc_coherent+0x8f/0x140
[Sun Jan 23 12:22:45 2022] [<ffffffffa606eb01>]
x86_swiotlb_alloc_coherent+0x21/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffc0763474>]
mlx5_dma_zalloc_coherent_node+0xb4/0x110 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0763d69>]
mlx5_buf_alloc_node+0x89/0x120 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0763e14>] mlx5_buf_alloc+0x14/0x20
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075d9bb>] create_map_eq+0xcb/0x3c0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075dd0a>] create_async_eq+0x5a/0x80
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075ea35>]
mlx5_eq_table_create+0x1a5/0x7e0 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffa607649a>] ? ioremap_nocache+0x1a/0x20
[Sun Jan 23 12:22:45 2022] [<ffffffffc075f55d>] ? alloc_uars_page+0x18d/0x2c0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc075ddb0>] ?
mlx5_eq_create_generic+0x80/0x80 [mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc07565d9>] mlx5_load_one+0x599/0xa30
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0756f11>] init_one+0x411/0x5c0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d6d1a>] local_pci_probe+0x4a/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d8469>] pci_device_probe+0x109/0x160
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb87f>] pci_bus_add_device+0x4f/0xa0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb909>] pci_bus_add_devices+0x39/0x80
[Sun Jan 23 12:22:45 2022] [<ffffffffc028437f>] hv_pci_probe+0x9ef/0xcf0
[pci_hyperv]
[Sun Jan 23 12:22:45 2022] [<ffffffffc024db81>] vmbus_probe+0x41/0xa0
[hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64ba9b8>] bus_probe_device+0x98/0xd0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b825f>] device_add+0x4ff/0x7c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b853a>] device_register+0x1a/0x20
[Sun Jan 23 12:22:45 2022] [<ffffffffc024e5a6>]
vmbus_device_register+0x66/0x100 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0251e5d>]
vmbus_add_channel_work+0x4cd/0x640 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bde5f>] process_one_work+0x17f/0x440
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bef76>] worker_thread+0x126/0x3c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bee50>] ?
manage_workers.isra.26+0x2a0/0x2a0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5e31>] kthread+0xd1/0xe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ?
insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] [<ffffffffa6795df7>]
ret_from_fork_nospec_begin+0x21/0x21
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ?
insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] Mem-Info:
[Sun Jan 23 12:22:45 2022] active_anon:30843105 inactive_anon:323158
isolated_anon:0
active_file:377528 inactive_file:381332 isolated_file:0
unevictable:0 dirty:9 writeback:0 unstable:0
slab_reclaimable:182369 slab_unreclaimable:52365
mapped:9123 shmem:1052916 pagetables:61434 bounce:0
free:173238 free_pcp:171 free_cma:0
[Sun Jan 23 12:22:45 2022] Node 0 DMA free:15892kB min:8kB low:8kB high:12kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 775 128751 128751
[Sun Jan 23 12:22:45 2022] Node 0 DMA32 free:512612kB min:404kB low:504kB
high:604kB active_anon:253160kB inactive_anon:160kB active_file:36kB
inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:1032128kB managed:794528kB mlocked:0kB dirty:0kB writeback:0kB
mapped:4kB shmem:444kB slab_reclaimable:11272kB slab_unreclaimable:1972kB
kernel_stack:192kB pagetables:2768kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
no
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 0 127975 127975
[Sun Jan 23 12:22:45 2022] Node 0 Normal free:164448kB min:67168kB
low:83960kB high:100752kB active_anon:123119260kB inactive_anon:1292472kB
active_file:1510076kB inactive_file:1525280kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:133169152kB managed:131050148kB
mlocked:0kB dirty:36kB writeback:0kB mapped:36488kB shmem:4211220kB
slab_reclaimable:718204kB slab_unreclaimable:207472kB kernel_stack:9712kB
pagetables:242968kB unstable:0kB bounce:0kB free_pcp:676kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 0 0 0
[Sun Jan 23 12:22:45 2022] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB
2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB
(M) = 15892kB
[Sun Jan 23 12:22:45 2022] Node 0 DMA32: 242*4kB (UM) 184*8kB (UEM) 121*16kB
(UEM) 69*32kB (UEM) 69*64kB (UEM) 25*128kB (UEM) 11*256kB (UM) 2*512kB (UE)
1*1024kB (U) 1*2048kB (U) 120*4096kB (M) = 512632kB
[Sun Jan 23 12:22:45 2022] Node 0 Normal: 8431*4kB (UEM) 7451*8kB (UEM)
2808*16kB (UEM) 442*32kB (UEM) 104*64kB (UEM) 44*128kB (UEM) 10*256kB (UM)
4*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 170324kB
[Sun Jan 23 12:22:45 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 23 12:22:45 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 23 12:22:45 2022] 1811902 total pagecache pages
[Sun Jan 23 12:22:45 2022] 0 pages in swap cache
[Sun Jan 23 12:22:45 2022] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 23 12:22:45 2022] Free swap = 0kB
[Sun Jan 23 12:22:45 2022] Total swap = 0kB
[Sun Jan 23 12:22:45 2022] 33554318 pages RAM
[Sun Jan 23 12:22:45 2022] 0 pages HighMem/MovableOnly
[Sun Jan 23 12:22:45 2022] 589172 pages reserved
[Sun Jan 23 12:22:45 2022] kworker/18:1: page allocation failure: order:8,
mode:0xc0d0
[Sun Jan 23 12:22:45 2022] CPU: 18 PID: 11869 Comm: kworker/18:1 Kdump:
loaded Tainted: G ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
[Sun Jan 23 12:22:45 2022] Hardware name: Microsoft Corporation Virtual
Machine/Virtual Machine, BIOS 090008 12/07/2018
[Sun Jan 23 12:22:45 2022] Workqueue: hv_pri_chan vmbus_add_channel_work
[hv_vmbus]
[Sun Jan 23 12:22:45 2022] Call Trace:
[Sun Jan 23 12:22:45 2022] [<ffffffffa67835a9>] dump_stack+0x19/0x1b
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c46c0>] warn_alloc_failed+0x110/0x180
[Sun Jan 23 12:22:45 2022] [<ffffffffa61c925f>]
__alloc_pages_nodemask+0x9df/0xbe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa6218ea8>] alloc_pages_current+0x98/0x110
[Sun Jan 23 12:22:45 2022] [<ffffffffa61e5ad8>] kmalloc_order+0x18/0x40
[Sun Jan 23 12:22:45 2022] [<ffffffffa6224876>] kmalloc_order_trace+0x26/0xa0
[Sun Jan 23 12:22:45 2022] [<ffffffffa614f93e>] ?
__irq_put_desc_unlock+0x1e/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa6228801>] __kmalloc+0x211/0x230
[Sun Jan 23 12:22:45 2022] [<ffffffffc07734d6>] mlx5_init_fc_stats+0x76/0x1d0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc077231d>] mlx5_init_fs+0x2d/0x840
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0756823>] mlx5_load_one+0x7e3/0xa30
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0756f11>] init_one+0x411/0x5c0
[mlx5_core]
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d6d1a>] local_pci_probe+0x4a/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63d8469>] pci_device_probe+0x109/0x160
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb87f>] pci_bus_add_device+0x4f/0xa0
[Sun Jan 23 12:22:45 2022] [<ffffffffa63cb909>] pci_bus_add_devices+0x39/0x80
[Sun Jan 23 12:22:45 2022] [<ffffffffc028437f>] hv_pci_probe+0x9ef/0xcf0
[pci_hyperv]
[Sun Jan 23 12:22:45 2022] [<ffffffffc024db81>] vmbus_probe+0x41/0xa0
[hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb955>] driver_probe_device+0xc5/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbc70>] ?
driver_probe_device+0x3e0/0x3e0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bbcb3>] __device_attach+0x43/0x50
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b95d5>] bus_for_each_drv+0x75/0xc0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64bb790>] device_attach+0x90/0xb0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64ba9b8>] bus_probe_device+0x98/0xd0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b825f>] device_add+0x4ff/0x7c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa64b853a>] device_register+0x1a/0x20
[Sun Jan 23 12:22:45 2022] [<ffffffffc024e5a6>]
vmbus_device_register+0x66/0x100 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffc0251e5d>]
vmbus_add_channel_work+0x4cd/0x640 [hv_vmbus]
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bde5f>] process_one_work+0x17f/0x440
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bef76>] worker_thread+0x126/0x3c0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60bee50>] ?
manage_workers.isra.26+0x2a0/0x2a0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5e31>] kthread+0xd1/0xe0
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ?
insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] [<ffffffffa6795df7>]
ret_from_fork_nospec_begin+0x21/0x21
[Sun Jan 23 12:22:45 2022] [<ffffffffa60c5d60>] ?
insert_kthread_work+0x40/0x40
[Sun Jan 23 12:22:45 2022] Mem-Info:
[Sun Jan 23 12:22:45 2022] active_anon:30843203 inactive_anon:323158
isolated_anon:0
active_file:377553 inactive_file:380392 isolated_file:0
unevictable:0 dirty:11 writeback:0 unstable:0
slab_reclaimable:182369 slab_unreclaimable:51851
mapped:9232 shmem:1052917 pagetables:61460 bounce:0
free:174818 free_pcp:239 free_cma:0
[Sun Jan 23 12:22:45 2022] Node 0 DMA free:15892kB min:8kB low:8kB high:12kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 775 128751 128751
[Sun Jan 23 12:22:45 2022] Node 0 DMA32 free:512632kB min:404kB low:504kB
high:604kB active_anon:253160kB inactive_anon:160kB active_file:36kB
inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:1032128kB managed:794528kB mlocked:0kB dirty:0kB writeback:0kB
mapped:4kB shmem:444kB slab_reclaimable:11272kB slab_unreclaimable:1972kB
kernel_stack:192kB pagetables:2768kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
no
[Sun Jan 23 12:22:45 2022] lowmem_reserve[]: 0 0 127975 127975
[Sun Jan 23 12:22:45 2022] Node 0 Normal free:170748kB min:67168kB
low:83960kB high:100752kB active_anon:123119652kB inactive_anon:1292472kB
active_file:1510176kB inactive_file:1521520kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:133169152kB managed:131050148kB
mlocked:0kB dirty:44kB writeback:0kB mapped:36924kB shmem:4211224kB
slab_reclaimable:718204kB slab_unreclaimable:205416kB kernel_stack:9776kB
pagetables:243072kB unstable:0kB bounce:0kB free_pcp:928kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[Sun Jan 23 12:22:46 2022] lowmem_reserve[]: 0 0 0 0
[Sun Jan 23 12:22:46 2022] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB
2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB
(M) = 15892kB
[Sun Jan 23 12:22:46 2022] Node 0 DMA32: 242*4kB (UM) 184*8kB (UEM) 121*16kB
(UEM) 69*32kB (UEM) 69*64kB (UEM) 25*128kB (UEM) 11*256kB (UM) 2*512kB (UE)
1*1024kB (U) 1*2048kB (U) 120*4096kB (M) = 512632kB
[Sun Jan 23 12:22:46 2022] Node 0 Normal: 9561*4kB (UEM) 7451*8kB (UEM)
2806*16kB (UEM) 437*32kB (UEM) 103*64kB (UEM) 15*128kB (UEM) 9*256kB (UM)
6*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 170620kB
[Sun Jan 23 12:22:46 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 23 12:22:46 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 23 12:22:46 2022] 1810845 total pagecache pages
[Sun Jan 23 12:22:46 2022] 0 pages in swap cache
[Sun Jan 23 12:22:46 2022] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 23 12:22:46 2022] Free swap = 0kB
[Sun Jan 23 12:22:46 2022] Total swap = 0kB
[Sun Jan 23 12:22:46 2022] 33554318 pages RAM
[Sun Jan 23 12:22:46 2022] 0 pages HighMem/MovableOnly
[Sun Jan 23 12:22:46 2022] 589172 pages reserved
[Sun Jan 23 12:22:46 2022] mlx5_core ab6d:00:02.0: Failed to init flow
steering
[Sun Jan 23 12:22:46 2022] mlx5_core ab6d:00:02.0: mlx5_load_one failed with
error code -12
[Sun Jan 23 12:22:46 2022] mlx5_core: probe of ab6d:00:02.0 failed with error
-12
[Fix]
26bf30902c10473ba38f220d3401a61c56d8db3b
48f02eef7f764f33e520ed8009d293396ca690cd
2fdeb4f4c2aea53a6a2e8adb69e811cf304a0ae5
38a54cae6f76c3e6a1e6c1e52c2e43a069fa78cb
b247f32aecad09e6cf7edff7739e6f2c9dc5fca9
[Test Case]
Microsoft tested
[Where things might go wrong]
VM allocations could continue to fail
[Other Info]
SF: #00327011
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1961632/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
