Hi Matthew, This looks like the symptoms of an Intel processor errata, 'Unexpected Page Faults in Guest Virtualization Environment'.
It should be fixed with Intel microcode updates of 2019/11/15. This has been shipped on Ubuntu intel-microcode package version 3.20191115.1ubuntu0.18.04.1 for Bionic [1]. However, in a cloud environment, you have to check with your cloud provider whether your hypervisor has the fix, as the guest cannot update the processor microcode. Hope this helps, Mauricio [1] https://launchpad.net/ubuntu/+source/intel- microcode/3.20191115.1ubuntu0.18.04.1 ** Changed in: linux (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1885944 Title: Oops: 0010 [#1] SMP PTI, at <do_page_fault>: "push %rbp" address Status in linux package in Ubuntu: Invalid Bug description: Kernel is Oopsing with following message: "BUG: unable to handle kernel paging request at ffffffff81a65350" Ubuntu: 16.04 Deb package: linux-image-4.15.0-1055-azure Frequency: Occured once a month. After upgrading to ~4.15.0-1067-azure~, didn't occur (?yet?). CONTEXT: #+begin_quote KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-1055-azure DUMPFILE: dump.202005300113 [PARTIAL DUMP] CPUS: 16 DATE: Sat May 30 01:09:00 2020 UPTIME: 14 days, 09:00:57 LOAD AVERAGE: 8.11, 7.01, 7.21 TASKS: 2264 RELEASE: 4.15.0-1055-azure VERSION: #60-Ubuntu SMP Thu Aug 8 18:29:07 UTC 2019 MACHINE: x86_64 (2693 Mhz) MEMORY: 32 GB PANIC: "BUG: unable to handle kernel paging request at ffffffff81a65350" PID: 52632 COMMAND: "nscd" TASK: ffff936ba34e5940 [THREAD_INFO: ffff936ba34e5940] CPU: 8 STATE: TASK_RUNNING (PANIC) #+end_quote PANIC INFO ("modules used" included): #+begin_quote [4967328.882291] unable to execute userspace code (SMEP?) (uid: 112) [4967328.884021] BUG: unable to handle kernel paging request at ffffffff81a65350 [4967328.884021] IP: do_page_fault+0x0/0x130 [4967328.884021] PGD 88ba0c067 P4D 88ba0c067 PUD 88ba0d063 PMD 88a8000e1 [4967328.884021] Oops: 0010 [#1] SMP PTI [4967328.884021] Modules linked in: binfmt_misc unix_diag ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_security ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner iptable_filter ip_tables x_tables kvm_intel kvm irqbypass hv_balloon serio_raw joydev ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel hid_generic aes_x86_64 crypto_simd hyperv_fb glue_helper hid_hyperv cfbfillrect cfbimgblt cryptd pata_acpi hid hv_netvsc hv_utils cfbcopyarea hyperv_keyboard [4967328.956026] CPU: 8 PID: 52632 Comm: nscd Not tainted 4.15.0-1055-azure #60-Ubuntu [4967328.956026] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017 [4967328.956026] RIP: 0010:do_page_fault+0x0/0x130 [4967328.956026] RSP: 0018:fffffe0000167920 EFLAGS: 00010093 [4967328.956026] RAX: 00000000824009f7 RBX: 0000000000000000 RCX: ffffffff824009f7 [4967328.956026] RDX: 00000000ffff9372 RSI: 0000000000000010 RDI: fffffe0000167928 [4967328.956026] RBP: fffffe0000167929 R08: 0000000000000000 R09: 0000000000000000 [4967328.956026] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [4967328.956026] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [4967328.956026] FS: 00007f4722c94740(0000) GS:ffff93729d800000(0000) knlGS:0000000000000000 [4967328.956026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [4967328.956026] CR2: ffffffff81a65350 CR3: 0000000872f4a002 CR4: 00000000003606e0 [4967328.956026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [4967328.956026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [4967328.956026] Call Trace: [4967328.956026] <#DF> [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967328.956026] ? native_iret+0x7/0x7 [4967328.956026] ? __do_page_fault+0x4d0/0x4d0 [4967328.956026] ? page_fault+0x25/0x50 [4967329.124013] ? handle_stack_overflow+0x50/0x50 [4967329.124013] ? double_fault+0x1e/0x30 [4967329.124013] ? native_iret+0x7/0x7 [4967329.124013] ? page_fault+0xa/0x50 [4967329.124013] </#DF> [4967329.124013] Code: 84 01 00 48 89 de 48 c7 c7 c8 a2 a7 82 c6 05 57 c6 3d 01 01 e8 c2 b4 07 00 e9 60 fd ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 41 57 41 56 41 55 49 89 f5 41 54 49 89 fc 53 0f 20 [4967329.152018] RIP: do_page_fault+0x0/0x130 RSP: fffffe0000167920 [4967329.152018] CR2: ffffffff81a65350 #+end_quote SYSTEM INFORMATION: #+begin_quote DMI_BIOS_VENDOR: American Megatrends Inc. DMI_BIOS_VERSION: 090007 DMI_BIOS_DATE: 06/02/2017 DMI_SYS_VENDOR: Microsoft Corporation DMI_PRODUCT_NAME: Virtual Machine DMI_PRODUCT_VERSION: 7.0 DMI_PRODUCT_SERIAL: 0000-0007-0266-2085-8407-7986-85 DMI_PRODUCT_UUID: 71928361-BF2C-9D47-8CF3-354C9F5EA739 DMI_BOARD_VENDOR: Microsoft Corporation DMI_BOARD_NAME: Virtual Machine DMI_BOARD_VERSION: 7.0 DMI_BOARD_SERIAL: 0000-0010-1565-2018-7858-1676-09 DMI_CHASSIS_VENDOR: Microsoft Corporation DMI_CHASSIS_TYPE: 3 DMI_CHASSIS_VERSION: 7.0 DMI_CHASSIS_SERIAL: 9935-0359-4239-3718-1407-0515-72 DMI_CHASSIS_ASSET_TAG: 7783-7084-3265-9085-8269-3286-77 #+end_quote MEM INFO: #+begin_quote PAGES TOTAL PERCENTAGE TOTAL MEM 8196822 31.3 GB ---- FREE 3012013 11.5 GB 36% of TOTAL MEM USED 5184809 19.8 GB 63% of TOTAL MEM SHARED 2023123 7.7 GB 24% of TOTAL MEM BUFFERS 30113 117.6 MB 0% of TOTAL MEM CACHED 2094507 8 GB 25% of TOTAL MEM SLAB 871805 3.3 GB 10% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 0 0 ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 0 0 0% of TOTAL SWAP COMMIT LIMIT 4098411 15.6 GB ---- COMMITTED 6184598 23.6 GB 150% of TOTAL LIMIT #+end_quote STACKS (FOR ALL ON-CPU PROCESSES) NOT ACCESSSIBLE (PID:52632 is faulting one): #+begin_quote PID: 0 TASK: ffffffff82c11480 CPU: 0 COMMAND: "swapper/0" bt: seek error: kernel virtual address: fffffe0000007000 type: "stack contents" bt: read of stack at fffffe0000007000 failed PID: 0 TASK: ffff937299909dc0 CPU: 1 COMMAND: "swapper/1" bt: seek error: kernel virtual address: fffffe0000033000 type: "stack contents" bt: read of stack at fffffe0000033000 failed PID: 0 TASK: ffff937299915940 CPU: 2 COMMAND: "swapper/2" bt: seek error: kernel virtual address: fffffe000005f000 type: "stack contents" bt: read of stack at fffffe000005f000 failed PID: 0 TASK: ffff937299911dc0 CPU: 3 COMMAND: "swapper/3" bt: seek error: kernel virtual address: fffffe000008b000 type: "stack contents" bt: read of stack at fffffe000008b000 failed PID: 52637 TASK: ffff936f01521dc0 CPU: 4 COMMAND: "chrome" bt: seek error: kernel virtual address: fffffe00000b7000 type: "stack contents" bt: read of stack at fffffe00000b7000 failed PID: 0 TASK: ffff937299910000 CPU: 5 COMMAND: "swapper/5" bt: seek error: kernel virtual address: fffffe00000e3000 type: "stack contents" bt: read of stack at fffffe00000e3000 failed PID: 0 TASK: ffff937299945940 CPU: 6 COMMAND: "swapper/6" bt: seek error: kernel virtual address: fffffe000010f000 type: "stack contents" bt: read of stack at fffffe000010f000 failed PID: 51862 TASK: ffff93715362d940 CPU: 7 COMMAND: "chrome" bt: seek error: kernel virtual address: fffffe000013b000 type: "stack contents" bt: read of stack at fffffe000013b000 failed PID: 52632 TASK: ffff936ba34e5940 CPU: 8 COMMAND: "nscd" bt: seek error: kernel virtual address: fffffe0000167000 type: "stack contents" bt: read of stack at fffffe0000167000 failed PID: 0 TASK: ffff937299940000 CPU: 9 COMMAND: "swapper/9" bt: seek error: kernel virtual address: fffffe0000193000 type: "stack contents" bt: read of stack at fffffe0000193000 failed PID: 0 TASK: ffff937299969dc0 CPU: 10 COMMAND: "swapper/10" bt: seek error: kernel virtual address: fffffe00001bf000 type: "stack contents" bt: read of stack at fffffe00001bf000 failed PID: 0 TASK: ffff93729996bb80 CPU: 11 COMMAND: "swapper/11" bt: seek error: kernel virtual address: fffffe00001eb000 type: "stack contents" bt: read of stack at fffffe00001eb000 failed PID: 0 TASK: ffff937299968000 CPU: 12 COMMAND: "swapper/12" bt: seek error: kernel virtual address: fffffe0000217000 type: "stack contents" bt: read of stack at fffffe0000217000 failed PID: 0 TASK: ffff93729996d940 CPU: 13 COMMAND: "swapper/13" bt: seek error: kernel virtual address: fffffe0000243000 type: "stack contents" bt: read of stack at fffffe0000243000 failed PID: 0 TASK: ffff937299971dc0 CPU: 14 COMMAND: "swapper/14" bt: seek error: kernel virtual address: fffffe000026f000 type: "stack contents" bt: read of stack at fffffe000026f000 failed PID: 0 TASK: ffff937299973b80 CPU: 15 COMMAND: "swapper/15" bt: seek error: kernel virtual address: fffffe000029b000 type: "stack contents" bt: read of stack at fffffe000029b000 failed #+end_quote FAULTING ADDRESS (~0xffffffff81a65350~): #+begin_quote VIRTUAL PHYSICAL ffffffff81a65350 88a865350 PGD DIRECTORY: ffffffff82c0a000 PAGE DIRECTORY: 88ba0c067 PUD: 88ba0cff0 => 88ba0d063 PMD: 88ba0d068 => 88a8000e1 PAGE: 88a800000 (2MB) PTE PHYSICAL FLAGS 88a8000e1 88a800000 (PRESENT|ACCESSED|DIRTY|PSE) PAGE PHYSICAL MAPPING INDEX CNT FLAGS fffff4da222a1940 88a865000 0 0 1 17ffffc0000800 reserved #+end_quote Is memory location accessibility information under "%RSP" needed? AND CODE: #+begin_quote /build/linux-azure-1_z7X4/linux-azure-4.15.0/arch/x86/mm/fault.c: 1534 0xffffffff81a65350 <do_page_fault>: push %rbp 0xffffffff81a65351 <do_page_fault+1>: mov %rsp,%rbp 0xffffffff81a65354 <do_page_fault+4>: push %r15 0xffffffff81a65356 <do_page_fault+6>: push %r14 0xffffffff81a65358 <do_page_fault+8>: push %r13 0xffffffff81a6535a <do_page_fault+10>: mov %rsi,%r13 0xffffffff81a6535d <do_page_fault+13>: push %r12 0xffffffff81a6535f <do_page_fault+15>: mov %rdi,%r12 0xffffffff81a65362 <do_page_fault+18>: push %rbx /build/linux-azure-1_z7X4/linux-azure-4.15.0/arch/x86/include/asm/paravirt.h: 57 0xffffffff81a65363 <do_page_fault+19>: mov %cr2,%rax 0xffffffff81a65366 <do_page_fault+22>: nopl 0x0(%rax) #+end_quote TAINTED MASK: #+begin_quote TAINTED_MASK: 0 #+end_quote BOOT LOG (includes boot params): see file:boot.log attache NOTE: no access to faulting instance of virtual machine image. Following comes from another instance of the same vm image: #+begin_quote ProblemType: Bug ApportVersion: 2.20.1-0ubuntu2.19 Architecture: amd64 Date: Wed Jul 1 16:57:32 2020 Dependencies: adduser 3.113+nmu3ubuntu4 apt 1.2.32 apt-utils 1.2.32 coreutils 8.25-2ubuntu3~16.04 debconf 1.5.58ubuntu2 debconf-i18n 1.5.58ubuntu2 debianutils 4.7 dmsetup 2:1.02.110-1ubuntu10 dpkg 1.18.4ubuntu1.6 gcc-5-base 5.4.0-6ubuntu1~16.04.12 gcc-6-base 6.0.1-0ubuntu1 gettext-base 0.19.7-2ubuntu3.1 gnupg 1.4.20-1ubuntu3.3 gpgv 1.4.20-1ubuntu3.3 grub-common 2.02~beta2-36ubuntu3.22 grub-gfxpayload-lists 0.7 grub-pc 2.02~beta2-36ubuntu3.22 grub-pc-bin 2.02~beta2-36ubuntu3.22 grub2-common 2.02~beta2-36ubuntu3.22 init-system-helpers 1.29ubuntu4 kmod 22-1ubuntu5.2 libacl1 2.2.52-3 libapt-inst2.0 1.2.32 libapt-pkg5.0 1.2.32 libasprintf0v5 0.19.7-2ubuntu3.1 libattr1 1:2.4.47-2 libaudit-common 1:2.4.5-1ubuntu2.1 libaudit1 1:2.4.5-1ubuntu2.1 libblkid1 2.27.1-6ubuntu3.7 libbz2-1.0 1.0.6-8ubuntu0.2 libc6 2.23-0ubuntu11 libdb5.3 5.3.28-11ubuntu0.2 libdevmapper1.02.1 2:1.02.110-1ubuntu10 libfdisk1 2.27.1-6ubuntu3.7 libfreetype6 2.6.1-0.1ubuntu2.3 libfuse2 2.9.4-1ubuntu3.1 libgcc1 1:6.0.1-0ubuntu1 libgcrypt20 1.6.5-2ubuntu0.5 libgpg-error0 1.21-2ubuntu1 libgpm2 1.20.4-6.1 libkmod2 22-1ubuntu5.2 liblocale-gettext-perl 1.07-1build1 liblz4-1 0.0~r131-2ubuntu2 liblzma5 5.1.1alpha+20120614-2ubuntu2 libmount1 2.27.1-6ubuntu3.7 libncursesw5 6.0+20160213-1ubuntu1 libpam-modules 1.1.8-3.2ubuntu2.1 libpam-modules-bin 1.1.8-3.2ubuntu2.1 libpam0g 1.1.8-3.2ubuntu2.1 libpcre3 2:8.38-3.1 libpng12-0 1.2.54-1ubuntu1.1 libreadline6 6.3-8ubuntu2 libselinux1 2.4-3build2 libsemanage-common 2.3-1build3 libsemanage1 2.3-1build3 libsepol1 2.4-2 libsmartcols1 2.27.1-6ubuntu3.7 libstdc++6 5.4.0-6ubuntu1~16.04.12 libsystemd0 229-4ubuntu21.22 libtext-charwidth-perl 0.04-7build5 libtext-iconv-perl 1.7-5build4 libtext-wrapi18n-perl 0.06-7.1 libtinfo5 6.0+20160213-1ubuntu1 libudev1 229-4ubuntu21.22 libusb-0.1-4 2:0.1.12-28 libustr-1.0-1 1.0.4-5 libuuid1 2.27.1-6ubuntu3.7 linux-base 4.5ubuntu1~16.04.1 linux-modules-4.15.0-1055-azure 4.15.0-1055.60 lsb-base 9.20160110ubuntu0.2 multiarch-support 2.23-0ubuntu11 os-prober 1.70ubuntu3.3 passwd 1:4.2-3.1ubuntu5.4 perl-base 5.22.1-9ubuntu0.6 readline-common 6.3-8ubuntu2 sensible-utils 0.0.9ubuntu0.16.04.1 sysvinit-utils 2.88dsf-59.3ubuntu2 tar 1.28-2.1ubuntu0.1 ubuntu-keyring 2012.05.19 ucf 3.0036 util-linux 2.27.1-6ubuntu3.7 uuid-runtime 2.27.1-6ubuntu3.7 zlib1g 1:1.2.8.dfsg-2ubuntu4.1 DistroRelease: Ubuntu 16.04 Package: linux-image-4.15.0-1055-azure 4.15.0-1055.60 PackageArchitecture: amd64 ProcCpuinfoMinimal: processor : 15 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz stepping : 4 microcode : 0xffffffff cpu MHz : 2693.724 cache size : 33792 KB physical id : 0 siblings : 16 core id : 7 cpu cores : 8 apicid : 15 initial apicid : 15 fpu : yes fpu_exception : yes cpuid level : 21 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs bogomips : 5387.44 clflush size : 64 cache_alignment : 64 address sizes : 44 bits physical, 48 bits virtual power management: ProcEnviron: LC_TIME=pl_PL.UTF-8 LC_MONETARY=pl_PL.UTF-8 LC_CTYPE='' TERM=dumb PATH=(custom, no user) LC_ADDRESS=pl_PL.UTF-8 XDG_RUNTIME_DIR=<set> LC_TELEPHONE=pl_PL.UTF-8 LANG=en_US.UTF-8 SHELL=/bin/bash LC_NAME=pl_PL.UTF-8 LC_MEASUREMENT=pl_PL.UTF-8 LC_IDENTIFICATION=pl_PL.UTF-8 LC_NUMERIC=pl_PL.UTF-8 LC_PAPER=pl_PL.UTF-8 ProcVersionSignature: User Name 4.15.0-1055.60-azure 4.15.18 SourcePackage: linux-signed-azure Tags: xenial uec-images Uname: Linux 4.15.0-1055-azure x86_64 UpgradeStatus: No upgrade log present (probably fresh install) _MarkForUpload: True #+end_quote To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1885944/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
