Is anybody knows the steps to reproduce this issue? We are also facing the same below TB in our testbed and we are planning to take the patch mentioned in comment #15.
Even though we are using this kernel from long time and seen this issue on very few nodes. Appreciate your help in this regard. Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.026015] TCP: request_sock_TCP: Possible SYN flooding on port 8033. Sending cookies. Check SNMP counters. Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.027529] BUG: kernel NULL pointer dereference, address: 0000000000000008 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.035339] #PF: supervisor read access in kernel mode Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.041083] #PF: error_code(0x0000) - not-present page Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.046838] PGD 0 P4D 0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.049670] Oops: 0000 [#1] SMP NOPTI Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.053764] CPU: 36 PID: 230 Comm: ksoftirqd/36 Not tainted 5.4.0-122-generic #138~18.04.1 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.063007] Hardware name: Cisco Systems Inc DN2-HW-APL-L/UCSC-C220-M5SX, BIOS C220M5.4.1.3i.0.0713210713 07/13/2021 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.074770] RIP: 0010:tcp_create_openreq_child+0x2e1/0x3e0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.080907] Code: 08 00 00 41 8b 84 24 18 01 00 00 48 c7 83 80 08 00 00 00 00 00 00 4c 89 e6 4c 89 ef 89 83 c4 05 00 00 49 8b 84 24 f8 00 00 00 <48> 8b 40 08 e8 96 b8 41 00 48 85 c0 0f b7 83 68 05 00 00 74 0a 83 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.101919] RSP: 0018:ffff97b88d207a28 EFLAGS: 00010246 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.107764] RAX: 0000000000000000 RBX: ffff8abb9d1bc600 RCX: 0000000000000007 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.115745] RDX: 0000000000000020 RSI: ffff8abb3c6e1560 RDI: ffff8acdef1b9180 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.123722] RBP: ffff97b88d207a48 R08: 0000000000000000 R09: ffff8aacffc07800 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.131699] R10: 0000000000000514 R11: ffff97b88d207b0f R12: ffff8abb3c6e1560 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.139678] R13: ffff8acdef1b9180 R14: ffff8abdfc1a7500 R15: ffff8adc75529ec0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.147655] FS: 0000000000000000(0000) GS:ffff8adcff000000(0000) knlGS:0000000000000000 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.156705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.163144] CR2: 0000000000000008 CR3: 000000594ea0a005 CR4: 00000000007606e0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.171120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.179099] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.187076] PKRU: 55555554 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.190101] Call Trace: Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.192841] tcp_v4_syn_recv_sock+0x5a/0x3d0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.197616] tcp_get_cookie_sock+0x48/0x140 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.202284] cookie_v4_check+0x561/0x660 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.206672] tcp_v4_do_rcv+0x1a0/0x1d0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.210864] tcp_v4_rcv+0xa86/0xad0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.214766] ip_protocol_deliver_rcu+0x31/0x1b0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.219831] ip_local_deliver_finish+0x48/0x50 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.224807] ip_local_deliver+0x7e/0xe0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.229095] ? ip_protocol_deliver_rcu+0x1b0/0x1b0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.234456] ip_rcv_finish+0x84/0xa0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.238453] ip_rcv+0xbc/0xd0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.241773] __netif_receive_skb_one_core+0x86/0xa0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.247226] __netif_receive_skb+0x18/0x60 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.251806] process_backlog+0xa9/0x170 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.256093] net_rx_action+0x140/0x3e0 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.260287] ? __switch_to_asm+0x34/0x70 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.264664] __do_softirq+0xe4/0x2da Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.268666] run_ksoftirqd+0x2b/0x40 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.272662] smpboot_thread_fn+0xfc/0x170 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.277146] kthread+0x121/0x140 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.280755] ? sort_range+0x30/0x30 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.284655] ? kthread_park+0x90/0x90 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.289259] ret_from_fork+0x1f/0x40 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.293714] Modules linked in: binfmt_misc ipt_rpfilter xt_CT iptable_raw ip_set_hash_ip ip_set_hash_net ipip ip_tunnel xt_multiport xt_nat xt_tcpudp xt_set veth ip_vs_lc ip_set_hash_ipportnet ip_set_hash_ipport ip_set_bitmap_port ip_set_hash_ipportip ip_set dummy ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_nat ip6_tables xt_comment xt_mark xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key crypto_user algif_hash xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter bridge stp llc aufs iptable_mangle bpfilter bonding dm_crypt algif_skcipher af_alg ipmi_ssif intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel nls_iso8859_1 kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl mei_me input_l eds joydev mei intel_cstate ioatdma lpc_ich ipmi_si Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.293744] ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter mac_hid sch_fq_codel sunrpc ip_tables x_tables autofs4 overlay ses mgag200 enclosure drm_vram_helper scsi_transport_sas i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ixgbe fb_sys_fops hid_generic xfrm_algo usbhid uas dca i40e megaraid_sas mdio drm hid usb_storage ahci libahci wmi Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.432873] CR2: 0000000000000008 Feb 22 22:20:47 maglev-master-192-168-70-10 kernel: [14501.437117] ---[ end trace 1d44478aec8706e7 ]--- -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-5.4 in Ubuntu. https://bugs.launchpad.net/bugs/1981658 Title: BUG: kernel NULL pointer dereference, address: 0000000000000008 Status in linux package in Ubuntu: Confirmed Status in linux-hwe-5.4 package in Ubuntu: Confirmed Status in linux source package in Bionic: Fix Released Status in linux-hwe-5.4 source package in Bionic: Confirmed Bug description: Hi, On one of the main US Ubuntu Archive servers (banjo), we decided to reboot into a HWE kernel. The latest being 5.4.0-122 but on doing so, ran into this kernel panic: | [ 350.776585] BUG: kernel NULL pointer dereference, address: 0000000000000008 | [ 350.783674] #PF: supervisor read access in kernel mode | [ 350.788846] #PF: error_code(0x0000) - not-present page | [ 350.794019] PGD 0 P4D 0 | [ 350.796631] Oops: 0000 [#1] SMP NOPTI | [ 350.800425] CPU: 8 PID: 0 Comm: swapper/8 Not tainted 5.4.0-122-generic #138~18.04.1-Ubuntu | [ 350.808918] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 02/10/2022 | [ 350.817666] RIP: 0010:tcp_create_openreq_child+0x2e1/0x3e0 | [ 350.823187] Code: 08 00 00 41 8b 84 24 18 01 00 00 48 c7 83 80 08 00 00 00 00 00 00 4c 89 e6 4c 89 ef 89 83 c4 05 00 00 49 8b 84 24 f8 00 00 00 <48> 8b 40 08 e8 96 28 42 00 48 85 c0 0f b7 83 68 05 00 00 74 0a 83 | [ 350.842068] RSP: 0018:ffff9a958cce8858 EFLAGS: 00010246 | [ 350.847324] RAX: 0000000000000000 RBX: ffff897618739c80 RCX: 0000000000000007 | [ 350.854502] RDX: 0000000000000020 RSI: ffff897607afb0b0 RDI: ffff897605c85580 | [ 350.861682] RBP: ffff9a958cce8878 R08: 0000000000000178 R09: ffff89763e407800 | [ 350.868859] R10: 00000000000004c4 R11: ffff9a958cce89c7 R12: ffff897607afb0b0 | [ 350.876039] R13: ffff897605c85580 R14: ffff8976205fbe00 R15: ffff89762688b400 | [ 350.883219] FS: 0000000000000000(0000) GS:ffff89763ec00000(0000) knlGS:0000000000000000 | [ 350.891358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 | [ 350.897138] CR2: 0000000000000008 CR3: 0000001fd7914000 CR4: 0000000000340ee0 | [ 350.904319] Call Trace: | [ 350.906787] <IRQ> | [ 350.908824] tcp_v6_syn_recv_sock+0x8d/0x710 | [ 350.913259] ? ip6_route_output_flags_noref+0xd0/0x110 | [ 350.918435] tcp_get_cookie_sock+0x48/0x140 | [ 350.922688] cookie_v6_check+0x5a2/0x700 | [ 350.926714] tcp_v6_do_rcv+0x36c/0x3e0 | [ 350.930589] ? tcp_v6_do_rcv+0x36c/0x3e0 | [ 350.934589] tcp_v6_rcv+0xa16/0xa60 | [ 350.938102] ip6_protocol_deliver_rcu+0xd8/0x4d0 | [ 350.942750] ip6_input+0x41/0xb0 | [ 350.946000] ip6_sublist_rcv_finish+0x42/0x60 | [ 350.950385] ip6_sublist_rcv+0x235/0x260 | [ 350.954333] ? __netif_receive_skb_core+0x19d/0xc60 | [ 350.959245] ipv6_list_rcv+0x110/0x140 | [ 350.963018] __netif_receive_skb_list_core+0x157/0x260 | [ 350.968192] ? build_skb+0x17/0x80 | [ 350.971615] netif_receive_skb_list_internal+0x187/0x2a0 | [ 350.976961] gro_normal_list.part.131+0x1e/0x40 | [ 350.981519] napi_complete_done+0x94/0x120 | [ 350.985700] mlx5e_napi_poll+0x178/0x630 [mlx5_core] | [ 350.990697] net_rx_action+0x140/0x3e0 | [ 350.994475] __do_softirq+0xe4/0x2da | [ 350.998079] irq_exit+0xae/0xb0 | [ 351.001239] do_IRQ+0x59/0xe0 | [ 351.004228] common_interrupt+0xf/0xf | [ 351.007913] </IRQ> | [ 351.010029] RIP: 0010:cpuidle_enter_state+0xbc/0x440 | [ 351.015023] Code: ff e8 b8 ca 80 ff 80 7d d3 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 54 03 00 00 31 ff e8 4b 4f 87 ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 1a 03 00 00 4c 2b 7d c8 48 ba cf f7 53 e3 a5 9b c4 | [ 351.033952] RSP: 0018:ffff9a958026fe48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd6 | [ 351.041633] RAX: ffff89763ec2fe00 RBX: ffffffff84b66b40 RCX: 000000000000001f | [ 351.048816] RDX: 00000051abe96150 RSI: 000000002abf3234 RDI: 0000000000000000 | [ 351.055997] RBP: ffff9a958026fe88 R08: 0000000000000002 R09: 000000000002f680 | [ 351.063176] R10: ffff9a958026fe18 R11: 0000000000000115 R12: ffff8976274c3800 | [ 351.070355] R13: 0000000000000001 R14: ffffffff84b66bb8 R15: 00000051abe96150 | [ 351.077540] ? cpuidle_enter_state+0x98/0x440 | [ 351.081930] ? menu_select+0x377/0x600 | [ 351.085706] cpuidle_enter+0x2e/0x40 | [ 351.089310] call_cpuidle+0x23/0x40 | [ 351.092821] do_idle+0x1f6/0x270 | [ 351.096069] cpu_startup_entry+0x1d/0x20 | [ 351.100024] start_secondary+0x166/0x1c0 | [ 351.103977] secondary_startup_64+0xa4/0xb0 | [ 351.108186] Modules linked in: binfmt_misc bonding nls_iso8859_1 ipmi_ssif edac_mce_amd kvm_amd kvm hpilo ccp ipmi_si ipmi_devintf ipmi_msghandler acpi_tad k10temp mac_hid acpi_power_meter sch_fq tcp_bbr ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib raid1 ses enclosure ib_uverbs ib_core mgag200 drm_vram_helper ttm drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect mlx5_core crc32_pclmul sysimgblt smartpqi fb_sys_fops uas ghash_clmulni_intel aesni_intel crypto_simd igb pci_hyperv_intf cryptd glue_helper usb_storage dca tls drm i2c_algo_bit scsi_transport_sas mlxfw nvme i2c_piix4 nvme_core wmi | [ 351.180156] CR2: 0000000000000008 | [ 351.183629] ---[ end trace 23210cdf0c6d5851 ]--- | [ 351.322276] RIP: 0010:tcp_create_openreq_child+0x2e1/0x3e0 | [ 351.327974] Code: 08 00 00 41 8b 84 24 18 01 00 00 48 c7 83 80 08 00 00 00 00 00 00 4c 89 e6 4c 89 ef 89 83 c4 05 00 00 49 8b 84 24 f8 00 00 00 <48> 8b 40 08 e8 96 28 42 00 48 85 c0 0f b7 83 68 05 00 00 74 0a 83 | [ 351.346878] RSP: 0018:ffff9a958cce8858 EFLAGS: 00010246 | [ 351.352166] RAX: 0000000000000000 RBX: ffff897618739c80 RCX: 0000000000000007 | [ 351.359348] RDX: 0000000000000020 RSI: ffff897607afb0b0 RDI: ffff897605c85580 | [ 351.366526] RBP: ffff9a958cce8878 R08: 0000000000000178 R09: ffff89763e407800 | [ 351.373705] R10: 00000000000004c4 R11: ffff9a958cce89c7 R12: ffff897607afb0b0 | [ 351.380886] R13: ffff897605c85580 R14: ffff8976205fbe00 R15: ffff89762688b400 | [ 351.388065] FS: 0000000000000000(0000) GS:ffff89763ec00000(0000) knlGS:0000000000000000 | [ 351.396203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 | [ 351.401982] CR2: 0000000000000008 CR3: 0000001fd7914000 CR4: 0000000000340ee0 | [ 351.409162] Kernel panic - not syncing: Fatal exception in interrupt | [ 351.415613] Kernel Offset: 0x2000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) | [ 351.437793] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- Per ~IS-Outage on Mattermost, tried various other older kernels and it seems -121 is working fine so looks to be introduced in -122. | [hloeung@banjo ~]$ lsb_release -a | No LSB modules are available. | Distributor ID: Ubuntu | Description: Ubuntu 18.04.6 LTS | Release: 18.04 | Codename: bionic --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 14 03:13 seq crw-rw---- 1 root audio 116, 33 Jul 14 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.28 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 18.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: HPE ProLiant DL385 Gen10 Package: linux-hwe-5.4 PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.utf8 SHELL=/bin/bash ProcFB: 0 mgag200drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-121-generic root=UUID=a5a2675d-dc52-48f0-9273-2c6dadac446f ro console=ttyS0,115200 nosplash ProcVersionSignature: Ubuntu 5.4.0-121.137~18.04.1-generic 5.4.189 RelatedPackageVersions: linux-restricted-modules-5.4.0-121-generic N/A linux-backports-modules-5.4.0-121-generic N/A linux-firmware 1.173.21 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic Uname: Linux 5.4.0-121-generic x86_64 UpgradeStatus: Upgraded to bionic on 2019-12-16 (940 days ago) UserGroups: adm _MarkForUpload: True dmi.bios.date: 02/10/2022 dmi.bios.vendor: HPE dmi.bios.version: A40 dmi.board.name: ProLiant DL385 Gen10 dmi.board.vendor: HPE dmi.chassis.type: 23 dmi.chassis.vendor: HPE dmi.modalias: dmi:bvnHPE:bvrA40:bd02/10/2022:svnHPE:pnProLiantDL385Gen10:pvr:rvnHPE:rnProLiantDL385Gen10:rvr:cvnHPE:ct23:cvr: dmi.product.family: ProLiant dmi.product.name: ProLiant DL385 Gen10 dmi.product.sku: 878615-B21 dmi.sys.vendor: HPE To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1981658/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
