Hi Marcelo, This issue can also been seen on the latest Ubuntu 18.04 image. It is also seen in the 5.10-rc7 Linux kernel as well as 5.4.
The issue can be reproduced 100% of the time using a more simple setup as described below. A basic "hello world" C program compiled for 32 bit crashes the Azure Linux host if a standard VM is first started as described below using qemu-kvm. // the Azure host kernel panic reproduction with 32 bit hello world. Includes the kernel panic traceback at the end. // Install a standard ubuntu image on the host like below (if the 11/25/20 is no longer available, the current one will do). root@jnpronevm1:~# wget https://cloud-images.ubuntu.com/bionic/20201125/bionic-server-cloudimg-amd64.img --2020-11-30 19:43:20-- https://cloud-images.ubuntu.com/bionic/20201125/bionic-server-cloudimg-amd64.img Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 91.189.88.247, 91.189.88.248, 2001:67c:1360:8001::33, ... Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|91.189.88.247|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 360775680 (344M) [application/octet-stream] Saving to: ‘bionic-server-cloudimg-amd64.img.1’ bionic-server-cloudim 100%[======================>] 344.06M 20.4MB/s in 18s 2020-11-30 19:43:39 (19.2 MB/s) - ‘bionic-server-cloudimg-amd64.img.1’ saved [360775680/360775680] // make sure qemu-kvm is installed and kvm acceleration can be used. root@jnpronevm1:~# root@jnpronevm1:~# kvm-ok INFO: /dev/kvm exists KVM acceleration can be used root@jnpronevm1:~# cat hello.c #include<stdio.h> int main() { printf("Hello world\n"); } // compile about in 32 bit mode, 64 bit does not show the issue root@jnpronevm1:~# gcc -m32 hello.c -o hello // start the VM from the above downloaded image. root@jnpronevm1:~#/usr/bin/qemu-system-x86_64 -daemonize -name u1 -machine pc-i440fx-bionic,accel=kvm -m 1024 -smp 2,sockets=1 -uuid 772495e6-8658-4306-ba5f-f59c15f42f69 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=bionic-server-cloudimg-amd64.img,if=none,id=drive-virtio-disk0,format=raw,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev socket,id=charserial0,host=127.0.0.1,port=8601,telnet,server,nowait,logfile=/var/log/console.log -vnc 127.0.0.1:0 -vga cirrus qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2] root@jnpronevm1:~# ps aux | grep qemu root 2966 45.6 0.1 1655976 41064 ? Sl 20:02 0:02 /usr/bin/qemu-system-x86_64 -daemonize -name u1 -machine pc-i440fx-bionic,accel=kvm -m 1024 -smp 2,sockets=1 -uuid 772495e6-8658-4306-ba5f-f59c15f42f69 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=bionic-server-cloudimg-amd64.img,if=none,id=drive-virtio-disk0,format=raw,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev socket,id=charserial0,host=127.0.0.1,port=8601,telnet,server,nowait,logfile=/var/log/console.log -vnc 127.0.0.1:0 -vga cirrus root 2979 0.0 0.0 14864 1064 pts/0 S+ 20:02 0:00 grep qemu root@jnpronevm1:~# ./hello The host VM hung up due to kernel panic after above step. ### the host vm hit kernel panic after running hello program. [ 7858.522920] PANIC: double fault, error_code: 0x0 [ 7858.525679] Kernel panic - not syncing: Machine halted. [ 7858.525679] CPU: 5 PID: 4746 Comm: hello Not tainted 4.15.18+test #1 [ 7858.525679] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [ 7858.525679] Call Trace: [ 7858.525679] <#DF> [ 7858.525679] dump_stack+0x63/0x8b [ 7858.525679] panic+0xe4/0x244 [ 7858.525679] df_debug+0x2d/0x30 [ 7858.525679] do_double_fault+0x9a/0x130 [ 7858.525679] double_fault+0x1e/0x30 [ 7858.525679] RIP: 0010:0x1a80 [ 7858.525679] RSP: 0018:000000000000e200 EFLAGS: 00010086 [ 7858.525679] RAX: 00000000000000c5 RBX: 0000000000000001 RCX: 00000000ffc98f2c [ 7858.525679] RDX: 00000000000007d4 RSI: 00000000f7ef2d80 RDI: 00000000f7ef2000 [ 7858.525679] RBP: 00000000ffc98ed8 R08: 0000000000000000 R09: 0000000000000000 [ 7858.525679] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 7858.525679] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7858.525679] </#DF> [ 7858.525679] Kernel Offset: 0xf000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 7858.525679] ---[ end Kernel panic - not syncing: Machine halted. [ 7858.525679] WARNING: CPU: 5 PID: 4746 at kernel/sched/core.c:1192 set_task_cpu+0x162/0x170 [ 7858.525679] Modules linked in: xt_owner iptable_security nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat br_netfilter xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables aufs ip6table_filter ip6_tables iptable_filter overlay openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack nls_iso8859_1 mlx4_en sb_edac mlx4_core devlink kvm_intel kvm irqbypass input_leds joydev mac_hid intel_rapl_perf hv_balloon pci_hyperv serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov [ 7858.525679] async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc hid_hyperv aesni_intel hv_utils hv_storvsc ptp hid hyperv_keyboard scsi_transport_fc hv_netvsc pps_core aes_x86_64 hyperv_fb crypto_simd glue_helper cryptd psmouse pata_acpi i2c_piix4 floppy hv_vmbus [ 7858.525679] CPU: 5 PID: 4746 Comm: hello Not tainted 4.15.18+test #1 [ 7858.525679] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [ 7858.525679] RIP: 0010:set_task_cpu+0x162/0x170 [ 7858.525679] RSP: 0018:ffff960e7fd43ca0 EFLAGS: 00010046 [ 7858.525679] RAX: 0000000000000200 RBX: ffff960e55de16c0 RCX: 0000000000000001 [ 7858.525679] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff960e55de16c0 [ 7858.525679] RBP: ffff960e7fd43cc0 R08: 0000000000000000 R09: 00000000000000ff [ 7858.525679] R10: 0000000000000000 R11: 0000000000000010 R12: ffff960e55de21ec [ 7858.525679] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000022880 [ 7858.525679] FS: 0000000000000000(0000) GS:ffff960e7fd40000(0063) knlGS:00000000f7f040c0 [ 7858.525679] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 7858.525679] CR2: 000000000000e1f8 CR3: 000000086b978002 CR4: 00000000003626e0 [ 7858.525679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7858.525679] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7858.525679] Call Trace: [ 7858.525679] <IRQ> [ 7858.525679] try_to_wake_up+0x14f/0x4a0 [ 7858.525679] default_wake_function+0x12/0x20 [ 7858.525679] autoremove_wake_function+0x12/0x40 [ 7858.525679] __wake_up_common+0x8c/0x130 [ 7858.525679] __wake_up_common_lock+0x80/0xc0 [ 7858.525679] __wake_up+0x13/0x20 [ 7858.525679] wake_up_klogd_work_func+0x40/0x60 [ 7858.525679] irq_work_run_list+0x55/0x80 [ 7858.525679] ? tick_sched_do_timer+0x40/0x40 [ 7858.525679] irq_work_tick+0x40/0x50 [ 7858.525679] update_process_times+0x42/0x60 [ 7858.525679] tick_sched_handle+0x2a/0x60 [ 7858.525679] tick_sched_timer+0x39/0x80 [ 7858.525679] __hrtimer_run_queues+0xe7/0x230 [ 7858.525679] hrtimer_interrupt+0xb1/0x200 [ 7858.525679] vmbus_isr+0x16c/0x2a0 [hv_vmbus] [ 7858.525679] hyperv_vector_handler+0x3f/0x6e [ 7858.525679] hyperv_callback_vector+0x84/0x90 [ 7858.525679] </IRQ> [ 7858.525679] <#DF> [ 7858.525679] RIP: 0010:panic+0x1fe/0x244 [ 7858.525679] RSP: 0018:fffffe00000e3e90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff0c [ 7858.525679] RAX: 0000000000000034 RBX: fffffe00000e3f00 RCX: 0000000000000006 [ 7858.525679] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff960e7fd56490 [ 7858.525679] RBP: fffffe00000e3f08 R08: 0000000000000000 R09: 0000000000000268 [ 7858.525679] R10: 0000000000000000 R11: 0000000000000038 R12: 0000000000000000 [ 7858.525679] R13: 0000000000000000 R14: 000000086b979802 R15: 0000000000000000 [ 7858.525679] df_debug+0x2d/0x30 [ 7858.525679] do_double_fault+0x9a/0x130 [ 7858.525679] double_fault+0x1e/0x30 [ 7858.525679] RIP: 0010:0x1a80 [ 7858.525679] RSP: 0018:000000000000e200 EFLAGS: 00010086 [ 7858.525679] RAX: 00000000000000c5 RBX: 0000000000000001 RCX: 00000000ffc98f2c [ 7858.525679] RDX: 00000000000007d4 RSI: 00000000f7ef2d80 RDI: 00000000f7ef2000 [ 7858.525679] RBP: 00000000ffc98ed8 R08: 0000000000000000 R09: 0000000000000000 [ 7858.525679] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 7858.525679] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7858.525679] </#DF> [ 7858.525679] Code: 2d ff ff ff 80 8b 64 08 00 00 04 e9 49 ff ff ff 0f 0b e9 e8 fe ff ff f7 43 5c fd ff ff ff 0f 84 f2 fe ff ff 0f 0b e9 eb fe ff ff <0f> 0b e9 fa fe ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 [ 7858.525679] ---[ end trace 21e7449a655e2326 ]--- [ 7858.652373] ------------[ cut here ]------------ [ 7858.652373] sched: Unexpected reschedule of offline CPU#0! [ 7858.652373] WARNING: CPU: 5 PID: 4746 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3f/0x50 [ 7858.652373] Modules linked in: xt_owner iptable_security nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat br_netfilter xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables aufs ip6table_filter ip6_tables iptable_filter overlay openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack nls_iso8859_1 mlx4_en sb_edac mlx4_core devlink kvm_intel kvm irqbypass input_leds joydev mac_hid intel_rapl_perf hv_balloon pci_hyperv serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov [ 7858.652373] async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc hid_hyperv aesni_intel hv_utils hv_storvsc ptp hid hyperv_keyboard scsi_transport_fc hv_netvsc pps_core aes_x86_64 hyperv_fb crypto_simd glue_helper cryptd psmouse pata_acpi i2c_piix4 floppy hv_vmbus [ 7858.652373] CPU: 5 PID: 4746 Comm: hello Tainted: G W 4.15.18+test #1 [ 7858.652373] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [ 7858.652373] RIP: 0010:native_smp_send_reschedule+0x3f/0x50 [ 7858.652373] RSP: 0018:ffff960e7fd43c28 EFLAGS: 00010086 [ 7858.652373] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006 [ 7858.652373] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff960e7fd56490 [ 7858.652373] RBP: ffff960e7fd43c28 R08: 0000000000000000 R09: 00000000000002a5 [ 7858.652373] R10: 0000000000100000 R11: 0000000000000038 R12: ffff960e55de16c0 [ 7858.652373] R13: ffff960e7fc22880 R14: ffff960e7fd43ce0 R15: ffff960e7fc22880 [ 7858.652373] FS: 0000000000000000(0000) GS:ffff960e7fd40000(0063) knlGS:00000000f7f040c0 [ 7858.652373] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 7858.652373] CR2: 000000000000e1f8 CR3: 000000086b978002 CR4: 00000000003626e0 [ 7858.652373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7858.652373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7858.652373] Call Trace: [ 7858.652373] <IRQ> [ 7858.652373] resched_curr+0x6c/0xd0 [ 7858.652373] check_preempt_curr+0x54/0x90 [ 7858.652373] ttwu_do_wakeup+0x1e/0x150 [ 7858.652373] ttwu_do_activate+0x77/0x80 [ 7858.652373] try_to_wake_up+0x1cc/0x4a0 [ 7858.652373] default_wake_function+0x12/0x20 [ 7858.652373] autoremove_wake_function+0x12/0x40 [ 7858.652373] __wake_up_common+0x8c/0x130 [ 7858.652373] __wake_up_common_lock+0x80/0xc0 [ 7858.652373] __wake_up+0x13/0x20 [ 7858.652373] wake_up_klogd_work_func+0x40/0x60 [ 7858.652373] irq_work_run_list+0x55/0x80 [ 7858.652373] ? tick_sched_do_timer+0x40/0x40 [ 7858.652373] irq_work_tick+0x40/0x50 [ 7858.652373] update_process_times+0x42/0x60 [ 7858.652373] tick_sched_handle+0x2a/0x60 [ 7858.652373] tick_sched_timer+0x39/0x80 [ 7858.652373] __hrtimer_run_queues+0xe7/0x230 [ 7858.652373] hrtimer_interrupt+0xb1/0x200 [ 7858.652373] vmbus_isr+0x16c/0x2a0 [hv_vmbus] [ 7858.652373] hyperv_vector_handler+0x3f/0x6e [ 7858.652373] hyperv_callback_vector+0x84/0x90 [ 7858.652373] </IRQ> [ 7858.652373] <#DF> [ 7858.652373] RIP: 0010:panic+0x1fe/0x244 [ 7858.652373] RSP: 0018:fffffe00000e3e90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff0c [ 7858.652373] RAX: 0000000000000034 RBX: fffffe00000e3f00 RCX: 0000000000000006 [ 7858.652373] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff960e7fd56490 [ 7858.652373] RBP: fffffe00000e3f08 R08: 0000000000000000 R09: 0000000000000268 [ 7858.652373] R10: 0000000000000000 R11: 0000000000000038 R12: 0000000000000000 [ 7858.652373] R13: 0000000000000000 R14: 000000086b979802 R15: 0000000000000000 [ 7858.652373] df_debug+0x2d/0x30 [ 7858.652373] do_double_fault+0x9a/0x130 [ 7858.652373] double_fault+0x1e/0x30 [ 7858.652373] RIP: 0010:0x1a80 [ 7858.652373] RSP: 0018:000000000000e200 EFLAGS: 00010086 [ 7858.652373] RAX: 00000000000000c5 RBX: 0000000000000001 RCX: 00000000ffc98f2c [ 7858.652373] RDX: 00000000000007d4 RSI: 00000000f7ef2d80 RDI: 00000000f7ef2000 [ 7858.652373] RBP: 00000000ffc98ed8 R08: 0000000000000000 R09: 0000000000000000 [ 7858.652373] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 7858.652373] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7858.652373] </#DF> [ 7858.652373] Code: 92 c0 84 c0 74 17 48 8b 05 7f 71 15 01 be fd 00 00 00 48 8b 40 30 e8 31 b6 ba 00 5d c3 89 fe 48 c7 c7 10 e3 0b 91 e8 b1 49 03 00 <0f> 0b 5d c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 [ 7858.652373] ---[ end trace 21e7449a655e2327 ]--- [ 7867.526414] hyperv_fb: Unable to send packet via vmbus [ 7867.526415] hyperv_fb: Unable to send packet via vmbus [ 7867.526415] hyperv_fb: Unable to send packet via vmbus [ 7867.526416] hyperv_fb: Unable to send packet via vmbus [ 7867.526416] hyperv_fb: Unable to send packet via vmbus [ 7867.526417] hyperv_fb: Unable to send packet via vmbus [ 7867.526417] hyperv_fb: Unable to send packet via vmbus [ 7867.526417] hyperv_fb: Unable to send packet via vmbus [ 7867.526418] hyperv_fb: Unable to send packet via vmbus On Thursday, December 10, 2020, 1:50:48 PM PST, Marcelo Cerri <1904...@bugs.launchpad.net> wrote: Thanks for reporting the issue, Kaveh. Do you have more information about the image you are using to create this Azure host? By your last comment, it doesn't seem the host is running one of the Ubuntu kernels. -- You received this bug notification because you are subscribed to the bug report. https://bugs.launchpad.net/bugs/1904632 Title: Ubuntu 18.04 Azure VM host kernel panic Status in linux-azure package in Ubuntu: New Bug description: Running a container on an DV3 Standard_D8_v3 Azure host, as the container comes up, the Azure host VM kernel panics per the logs below. Isolated the issue to a process in the container which uses the virtual NICs available on the Azure host. The container also is running Ubuntu 18.04 based packages. The problem happens every single time the container is started, unless its NIC access process is not started. Has this sort of kernel panic on Azure been seen and what is the root cause and remedy please. Also the kernel logs on the Azure host show it vulnerable to the following CVE. There are other VMs and containers that can run on the Azure host without a kernel panic on it, but providing this info in case there is some tie-in to the panic. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3646 Kernel panic from the Azure Host console: Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.13.33_e857c609-bc35-4b66-9a8b-e86fd8707e82.scope 2020-11-17T00:50:11.537914Z INFO MonitorHandler ExtHandler Stopped tracking cgroup: Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.13.33, path: /sys/fs/cgroup/memory/system.slice/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.13.33_e857c609-bc35-4b66-9a8b-e86fd8707e82.scope 2020-11-17T00:50:23.291433Z INFO ExtHandler ExtHandler Checking for agent updates (family: Prod) 2020-11-17T00:51:11.677191Z INFO ExtHandler ExtHandler [HEARTBEAT] Agent WALinuxAgent-2.2.52 is running as the goal state agent [DEBUG HeartbeatCounter: 7;HeartbeatId: 8A2DD5B7-02E5-46E2-9EDB-F8CCBA274479;DroppedPackets: 0;UpdateGSErrors: 0;AutoUpdate: 1] [11218.537937] PANIC: double fault, error_code: 0x0 [11218.541423] Kernel panic - not syncing: Machine halted. [11218.541423] CPU: 0 PID: 9281 Comm: vmxt Not tainted 4.15.18+test #1 [11218.541423] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [11218.541423] Call Trace: [11218.541423] <#DF> [11218.541423] dump_stack+0x63/0x8b [11218.541423] panic+0xe4/0x244 [11218.541423] df_debug+0x2d/0x30 [11218.541423] do_double_fault+0x9a/0x130 [11218.541423] double_fault+0x1e/0x30 [11218.541423] RIP: 0010:0x1a80 [11218.541423] RSP: 0018:0000000000002200 EFLAGS: 00010096 [11218.541423] RAX: 0000000000000102 RBX: 00000000f7a40768 RCX: 000000000000002f [11218.541423] RDX: 00000000f7ee9970 RSI: 00000000f7a40700 RDI: 00000000f7c3a000 [11218.541423] RBP: 00000000fffd6430 R08: 0000000000000000 R09: 0000000000000000 [11218.541423] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11218.541423] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11218.541423] </#DF> [11218.541423] Kernel Offset: 0x2a400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [11218.541423] ---[ end Kernel panic - not syncing: Machine halted. [11218.636804] ------------[ cut here ]------------ [11218.640802] sched: Unexpected reschedule of offline CPU#2! [11218.640802] WARNING: CPU: 0 PID: 9281 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3f/0x50 [11218.640802] Modules linked in: xt_nat xt_u32 vxlan ip6_udp_tunnel udp_tunnel veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter aufs xt_owner iptable_security xt_conntrack overlay openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack nls_iso8859_1 joydev input_leds mac_hid kvm_intel hv_balloon kvm serio_raw irqbypass intel_rapl_perf sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov [11218.640802] async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul hid_hyperv ghash_clmulni_intel hv_utils hv_storvsc pcbc ptp hv_netvsc hid pps_core scsi_transport_fc hyperv_keyboard aesni_intel aes_x86_64 crypto_simd hyperv_fb floppy glue_helper cryptd psmouse hv_vmbus i2c_piix4 pata_acpi [11218.640802] CPU: 0 PID: 9281 Comm: vmxt Not tainted 4.15.18+test #1 [11218.640802] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [11218.640802] RIP: 0010:native_smp_send_reschedule+0x3f/0x50 [11218.640802] RSP: 0018:ffff9446bfc03e08 EFLAGS: 00010082 [11218.640802] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000006 [11218.640802] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff9446bfc16490 [11218.640802] RBP: ffff9446bfc03e08 R08: 0000000000000000 R09: 0000000000001480 [11218.640802] R10: 0000000000000549 R11: 0000000000000038 R12: ffff9446bfca2880 [11218.640802] R13: 0000000000000000 R14: 000000010029a6b8 R15: ffff9446bfc1cd28 [11218.640802] FS: 0000000000000000(0000) GS:ffff9446bfc00000(0063) knlGS:00000000f7a40700 [11218.640802] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [11218.640802] CR2: 00000000000021f8 CR3: 000000084c576004 CR4: 00000000003626f0 [11218.640802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11218.640802] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11218.640802] Call Trace: [11218.640802] <IRQ> [11218.640802] trigger_load_balance+0x12a/0x230 [11218.640802] scheduler_tick+0xae/0xd0 [11218.640802] ? tick_sched_do_timer+0x40/0x40 [11218.640802] update_process_times+0x47/0x60 [11218.640802] tick_sched_handle+0x2a/0x60 [11218.640802] tick_sched_timer+0x39/0x80 [11218.640802] __hrtimer_run_queues+0xe7/0x230 [11218.640802] hrtimer_interrupt+0xb1/0x200 [11218.640802] vmbus_isr+0x16c/0x2a0 [hv_vmbus] [11218.640802] hyperv_vector_handler+0x3f/0x6e [11218.640802] hyperv_callback_vector+0x84/0x90 [11218.640802] </IRQ> [11218.640802] <#DF> [11218.640802] RIP: 0010:panic+0x1fe/0x244 [11218.640802] RSP: 0018:fffffe0000007e90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff0c [11218.640802] RAX: 0000000000000034 RBX: fffffe0000007f00 RCX: 0000000000000006 [11218.640802] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9446bfc16490 [11218.640802] RBP: fffffe0000007f08 R08: 0000000000000000 R09: 000000000000147e [11218.640802] R10: 0000000000000000 R11: 0000000000000038 R12: 0000000000000000 [11218.640802] R13: 0000000000000000 R14: 000000084c577804 R15: 0000000000000000 [11218.640802] df_debug+0x2d/0x30 [11218.640802] do_double_fault+0x9a/0x130 [11218.640802] double_fault+0x1e/0x30 [11218.640802] RIP: 0010:0x1a80 [11218.640802] RSP: 0018:0000000000002200 EFLAGS: 00010096 [11218.640802] RAX: 0000000000000102 RBX: 00000000f7a40768 RCX: 000000000000002f [11218.640802] RDX: 00000000f7ee9970 RSI: 00000000f7a40700 RDI: 00000000f7c3a000 [11218.640802] RBP: 00000000fffd6430 R08: 0000000000000000 R09: 0000000000000000 [11218.640802] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11218.640802] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11218.640802] </#DF> [11218.640802] Code: 92 c0 84 c0 74 17 48 8b 05 7f 71 15 01 be fd 00 00 00 48 8b 40 30 e8 31 b6 ba 00 5d c3 89 fe 48 c7 c7 10 e3 4b ac e8 b1 49 03 00 <0f> 0b 5d c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 [11218.640802] ---[ end trace 857d64f92b00ceb2 ]--- [11221.669166] hyperv_fb: Unable to send packet via vmbus [11221.669167] hyperv_fb: Unable to send packet via vmbus [11221.669167] hyperv_fb: Unable to send packet via vmbus [11221.669167] hyperv_fb: Unable to send packet via vmbu To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1904632/+subscriptions ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-3646 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1904632 Title: Ubuntu 18.04 Azure VM host kernel panic Status in linux-azure package in Ubuntu: New Bug description: Running a container on an DV3 Standard_D8_v3 Azure host, as the container comes up, the Azure host VM kernel panics per the logs below. Isolated the issue to a process in the container which uses the virtual NICs available on the Azure host. The container also is running Ubuntu 18.04 based packages. The problem happens every single time the container is started, unless its NIC access process is not started. Has this sort of kernel panic on Azure been seen and what is the root cause and remedy please. Also the kernel logs on the Azure host show it vulnerable to the following CVE. There are other VMs and containers that can run on the Azure host without a kernel panic on it, but providing this info in case there is some tie-in to the panic. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3646 Kernel panic from the Azure Host console: Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.13.33_e857c609-bc35-4b66-9a8b-e86fd8707e82.scope 2020-11-17T00:50:11.537914Z INFO MonitorHandler ExtHandler Stopped tracking cgroup: Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.13.33, path: /sys/fs/cgroup/memory/system.slice/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.13.33_e857c609-bc35-4b66-9a8b-e86fd8707e82.scope 2020-11-17T00:50:23.291433Z INFO ExtHandler ExtHandler Checking for agent updates (family: Prod) 2020-11-17T00:51:11.677191Z INFO ExtHandler ExtHandler [HEARTBEAT] Agent WALinuxAgent-2.2.52 is running as the goal state agent [DEBUG HeartbeatCounter: 7;HeartbeatId: 8A2DD5B7-02E5-46E2-9EDB-F8CCBA274479;DroppedPackets: 0;UpdateGSErrors: 0;AutoUpdate: 1] [11218.537937] PANIC: double fault, error_code: 0x0 [11218.541423] Kernel panic - not syncing: Machine halted. [11218.541423] CPU: 0 PID: 9281 Comm: vmxt Not tainted 4.15.18+test #1 [11218.541423] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [11218.541423] Call Trace: [11218.541423] <#DF> [11218.541423] dump_stack+0x63/0x8b [11218.541423] panic+0xe4/0x244 [11218.541423] df_debug+0x2d/0x30 [11218.541423] do_double_fault+0x9a/0x130 [11218.541423] double_fault+0x1e/0x30 [11218.541423] RIP: 0010:0x1a80 [11218.541423] RSP: 0018:0000000000002200 EFLAGS: 00010096 [11218.541423] RAX: 0000000000000102 RBX: 00000000f7a40768 RCX: 000000000000002f [11218.541423] RDX: 00000000f7ee9970 RSI: 00000000f7a40700 RDI: 00000000f7c3a000 [11218.541423] RBP: 00000000fffd6430 R08: 0000000000000000 R09: 0000000000000000 [11218.541423] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11218.541423] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11218.541423] </#DF> [11218.541423] Kernel Offset: 0x2a400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [11218.541423] ---[ end Kernel panic - not syncing: Machine halted. [11218.636804] ------------[ cut here ]------------ [11218.640802] sched: Unexpected reschedule of offline CPU#2! [11218.640802] WARNING: CPU: 0 PID: 9281 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3f/0x50 [11218.640802] Modules linked in: xt_nat xt_u32 vxlan ip6_udp_tunnel udp_tunnel veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter aufs xt_owner iptable_security xt_conntrack overlay openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack nls_iso8859_1 joydev input_leds mac_hid kvm_intel hv_balloon kvm serio_raw irqbypass intel_rapl_perf sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov [11218.640802] async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul hid_hyperv ghash_clmulni_intel hv_utils hv_storvsc pcbc ptp hv_netvsc hid pps_core scsi_transport_fc hyperv_keyboard aesni_intel aes_x86_64 crypto_simd hyperv_fb floppy glue_helper cryptd psmouse hv_vmbus i2c_piix4 pata_acpi [11218.640802] CPU: 0 PID: 9281 Comm: vmxt Not tainted 4.15.18+test #1 [11218.640802] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [11218.640802] RIP: 0010:native_smp_send_reschedule+0x3f/0x50 [11218.640802] RSP: 0018:ffff9446bfc03e08 EFLAGS: 00010082 [11218.640802] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000006 [11218.640802] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff9446bfc16490 [11218.640802] RBP: ffff9446bfc03e08 R08: 0000000000000000 R09: 0000000000001480 [11218.640802] R10: 0000000000000549 R11: 0000000000000038 R12: ffff9446bfca2880 [11218.640802] R13: 0000000000000000 R14: 000000010029a6b8 R15: ffff9446bfc1cd28 [11218.640802] FS: 0000000000000000(0000) GS:ffff9446bfc00000(0063) knlGS:00000000f7a40700 [11218.640802] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [11218.640802] CR2: 00000000000021f8 CR3: 000000084c576004 CR4: 00000000003626f0 [11218.640802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11218.640802] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11218.640802] Call Trace: [11218.640802] <IRQ> [11218.640802] trigger_load_balance+0x12a/0x230 [11218.640802] scheduler_tick+0xae/0xd0 [11218.640802] ? tick_sched_do_timer+0x40/0x40 [11218.640802] update_process_times+0x47/0x60 [11218.640802] tick_sched_handle+0x2a/0x60 [11218.640802] tick_sched_timer+0x39/0x80 [11218.640802] __hrtimer_run_queues+0xe7/0x230 [11218.640802] hrtimer_interrupt+0xb1/0x200 [11218.640802] vmbus_isr+0x16c/0x2a0 [hv_vmbus] [11218.640802] hyperv_vector_handler+0x3f/0x6e [11218.640802] hyperv_callback_vector+0x84/0x90 [11218.640802] </IRQ> [11218.640802] <#DF> [11218.640802] RIP: 0010:panic+0x1fe/0x244 [11218.640802] RSP: 0018:fffffe0000007e90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff0c [11218.640802] RAX: 0000000000000034 RBX: fffffe0000007f00 RCX: 0000000000000006 [11218.640802] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9446bfc16490 [11218.640802] RBP: fffffe0000007f08 R08: 0000000000000000 R09: 000000000000147e [11218.640802] R10: 0000000000000000 R11: 0000000000000038 R12: 0000000000000000 [11218.640802] R13: 0000000000000000 R14: 000000084c577804 R15: 0000000000000000 [11218.640802] df_debug+0x2d/0x30 [11218.640802] do_double_fault+0x9a/0x130 [11218.640802] double_fault+0x1e/0x30 [11218.640802] RIP: 0010:0x1a80 [11218.640802] RSP: 0018:0000000000002200 EFLAGS: 00010096 [11218.640802] RAX: 0000000000000102 RBX: 00000000f7a40768 RCX: 000000000000002f [11218.640802] RDX: 00000000f7ee9970 RSI: 00000000f7a40700 RDI: 00000000f7c3a000 [11218.640802] RBP: 00000000fffd6430 R08: 0000000000000000 R09: 0000000000000000 [11218.640802] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11218.640802] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11218.640802] </#DF> [11218.640802] Code: 92 c0 84 c0 74 17 48 8b 05 7f 71 15 01 be fd 00 00 00 48 8b 40 30 e8 31 b6 ba 00 5d c3 89 fe 48 c7 c7 10 e3 4b ac e8 b1 49 03 00 <0f> 0b 5d c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 [11218.640802] ---[ end trace 857d64f92b00ceb2 ]--- [11221.669166] hyperv_fb: Unable to send packet via vmbus [11221.669167] hyperv_fb: Unable to send packet via vmbus [11221.669167] hyperv_fb: Unable to send packet via vmbus [11221.669167] hyperv_fb: Unable to send packet via vmbu To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1904632/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
