I have followed the steps outlined in the Test Plan for testing the bionic series with the fix from my ppa https://launchpad.net/~hypothetical- lemon/+archive/ubuntu/lp-1978333?field.series_filter=
verified the bug exists in Bionic and has been resolved with the removal of the parameter a crash dump was created and the vm was rebooted without manual intervention. ** Description changed: [Impact] Azure VM instances hit I/O error on boot causing kernel crash [Test Plan] - Create VM on Azure + Create Ubuntu Marketplace VM on Azure - Install crash dump utilies (from guide: + ``` + ssh -i .ssh/id_rsa ubuntu@ipaddr<azureVM> + ``` + + Install crash dump utilities (from guide: https://ubuntu.com/server/docs/kernel-crash-dump) - sudo apt install linux-crashdump + apt-get install kdump-tools Say (y) to all questions during install - Reboot the VM + kdump-config show + *shows the vm is not yet ready to kdump - As root on the VM after reboot: - kdump-config show - + root@bionic3: kdump-config show + * no crashkernel= parameter in the kernel cmdline DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash - crashkernel addr: 0x - /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1085-azure - kdump initrd: - /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1085-azure + crashkernel addr: + /var/lib/kdump/vmlinuz + kdump initrd: + /var/lib/kdump/initrd.img + current state: Not ready to kdump + + kexec command: + no kexec command recorded + + + Reboot the VM + + sudo su + + As root on the VM after reboot: + + kdump-config show + + kdump-config show + DUMP_MODE: kdump + USE_KDUMP: 1 + KDUMP_SYSCTL: kernel.panic_on_oops=1 + KDUMP_COREDIR: /var/crash + crashkernel addr: 0x32000000 + /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure + kdump initrd: + /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure current state: ready to kdump kexec command: - /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1085-azure root=UUID=a117a689-aa1d-4cdc-8a30-7b9fbf174437 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz + /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz + #verify kdump is on cat /proc/cmdline ... crashkernel=512M-:192M dmesg | grep -i crash [ 0.071660] kexec: Reserving the low 1M of memory for crashkernel [ 0.269823] Reserving 192MB of memory at 640MB for crashkernel (System RAM: 4095MB) cat /proc/sys/kernel/sysrq # make sure this value is greater than 0 #set it to 1 sudo sysctl -w kernel.sysrq=1 + the directory of /var/crash should have no crashes yet as well. + Outcome with "ata_piix.prefer_ms_hyperv=0" in kexec command: # perform crash sudo su echo c > /proc/sysrq-trigger + After a couple of minuties + Open new terminal and try to ssh to azure VM, + It does not succeed + + Force a reboot of the VM through the portal or serial console #kdump doesnt work and hangs indefinitely #force reboot VM from Azure console - Outcome after removing "ata_piix.prefer_ms_hyperv=0" in kexec command: - # perform crash - sudo su + # verify package version of kdump-tool + # verify parameter is not listed in the kdump-config show output + kdump-config unload + kdump-config load + kdump-config show + + kdump-config show + DUMP_MODE: kdump + USE_KDUMP: 1 + KDUMP_SYSCTL: kernel.panic_on_oops=1 + KDUMP_COREDIR: /var/crash + crashkernel addr: 0x32000000 + /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure + kdump initrd: + /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure + current state: ready to kdump + + kexec command: + /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz + + + #trigger another crash echo c > /proc/sysrq-trigger - #kdump works, VM eventually (several minutes) reboots - # Login to VM and check that there is a core file in /var/crash: - ls /var/crash + # open new terminal and ssh back into vm + + cd /var/crash + + #verify a dump was created + linux-image-5.4.0-1086-azure-202208041658.crash [Where Problems Could Occur] This change modifies the debian/rules. The package could fail to build properly if mistyped. [Other] Back-porting a fix from upstream to remove "ata_piix.prefer_ms_hyperv=0" parameter. target series - Bionic, Focal, Jammy upstream patch https://salsa.debian.org/debian/kdump- tools/-/commit/b1bac9396ddbbce3817c34be3161630698e4a503 *Note: There are two source packages needed changes, kdump-tools for Impish -> Kinetic and makedumpfile for series Focal -> Bionic ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic ** Tags removed: verification-needed-impish -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1978333 Title: Remove "ata_piix.prefer_ms_hyperv=0" parameter Status in kdump-tools package in Ubuntu: Fix Committed Status in makedumpfile package in Ubuntu: Invalid Status in kdump-tools source package in Bionic: Invalid Status in makedumpfile source package in Bionic: New Status in kdump-tools source package in Focal: Invalid Status in makedumpfile source package in Focal: New Status in kdump-tools source package in Impish: Invalid Status in makedumpfile source package in Impish: Invalid Status in kdump-tools source package in Jammy: Incomplete Status in makedumpfile source package in Jammy: Invalid Status in kdump-tools source package in Kinetic: Fix Committed Status in makedumpfile source package in Kinetic: Invalid Bug description: [Impact] Azure VM instances hit I/O error on boot causing kernel crash [Test Plan] Create Ubuntu Marketplace VM on Azure ``` ssh -i .ssh/id_rsa ubuntu@ipaddr<azureVM> ``` Install crash dump utilities (from guide: https://ubuntu.com/server/docs/kernel-crash-dump) apt-get install kdump-tools Say (y) to all questions during install kdump-config show *shows the vm is not yet ready to kdump root@bionic3: kdump-config show * no crashkernel= parameter in the kernel cmdline DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: /var/lib/kdump/vmlinuz kdump initrd: /var/lib/kdump/initrd.img current state: Not ready to kdump kexec command: no kexec command recorded Reboot the VM sudo su As root on the VM after reboot: kdump-config show kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: 0x32000000 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure current state: ready to kdump kexec command: /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz #verify kdump is on cat /proc/cmdline ... crashkernel=512M-:192M dmesg | grep -i crash [ 0.071660] kexec: Reserving the low 1M of memory for crashkernel [ 0.269823] Reserving 192MB of memory at 640MB for crashkernel (System RAM: 4095MB) cat /proc/sys/kernel/sysrq # make sure this value is greater than 0 #set it to 1 sudo sysctl -w kernel.sysrq=1 the directory of /var/crash should have no crashes yet as well. Outcome with "ata_piix.prefer_ms_hyperv=0" in kexec command: # perform crash sudo su echo c > /proc/sysrq-trigger After a couple of minuties Open new terminal and try to ssh to azure VM, It does not succeed Force a reboot of the VM through the portal or serial console #kdump doesnt work and hangs indefinitely #force reboot VM from Azure console # verify package version of kdump-tool # verify parameter is not listed in the kdump-config show output kdump-config unload kdump-config load kdump-config show kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: 0x32000000 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure current state: ready to kdump kexec command: /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz #trigger another crash echo c > /proc/sysrq-trigger # open new terminal and ssh back into vm cd /var/crash #verify a dump was created linux-image-5.4.0-1086-azure-202208041658.crash [Where Problems Could Occur] This change modifies the debian/rules. The package could fail to build properly if mistyped. [Other] Back-porting a fix from upstream to remove "ata_piix.prefer_ms_hyperv=0" parameter. target series - Bionic, Focal, Jammy upstream patch https://salsa.debian.org/debian/kdump- tools/-/commit/b1bac9396ddbbce3817c34be3161630698e4a503 *Note: There are two source packages needed changes, kdump-tools for Impish -> Kinetic and makedumpfile for series Focal -> Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/kdump-tools/+bug/1978333/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
