------- Comment From [email protected] 2025-07-17 04:22 EDT------- (In reply to comment #16)
That's on me. I will surely do that Frank. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to crash in Ubuntu. https://bugs.launchpad.net/bugs/2117099 Title: "invalid kernel virtual address" error while analysing virsh dump Status in The Ubuntu-power-systems project: Incomplete Status in crash package in Ubuntu: Incomplete Bug description: == Kowshik Jois B S == ---Problem Description--- crash: invalid kernel virtual address: fbc10030fba10030 type: "first vmlist addr" error is observed while trying to analyse the dump collected with "virsh dump" using the crash tool. Machine Type = P10 Rainier ---Debugger--- A debugger is not configured ---Steps to Reproduce--- 1. collect the dump of a KVM guest on Ubuntu 25.04 ==> virsh dump ub2504guest1 myDump --memory-only --format=kdump-zlib 2. Try to analyze the collected dump using the crash tool. ==> crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic ./myDump ---uname output--- Linux 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le ppc64le ppc64le GNU/Linux Userspace rpm: crash The userspace tool has the following bit modes: 64-bit Userspace tool common name: crash Userspace tool obtained from project website: na output: # crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic myDump crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... crash: invalid kernel virtual address: fbc10030fba10030 type: "first vmlist addr" Errors like the one above typically occur when the kernel and memory source do not match. These are the files being used: KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic DUMPFILE: myDump HOST ENV: # uname -a Linux 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le ppc64le ppc64le GNU/Linux # qemu-system-ppc64 --version QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5) Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers root@:~# libvirtd --version libvirtd (libvirt) 11.0.0 # apt list --installed | grep crash WARNING: apt does not have a stable CLI interface. Use with caution in scripts. crash/plucky,now 8.0.6-1ubuntu1 ppc64el [installed] linux-crashdump/plucky,now 6.14.0-15.15 ppc64el [installed] # apt list --installed | grep dbgsym WARNING: apt does not have a stable CLI interface. Use with caution in scripts. linux-image-6.14.0-15-generic-dbgsym/now 6.14.0-15.15 ppc64el [installed,local] Guest ENV: # uname -a Linux ub2504guest1 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le ppc64le ppc64le GNU/Linux This issue is seen only with the virsh dump. KDUMP/FADUMP collected can be analysed with the tool normally. # crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic /var/crash/202505061152/dump.202505061152 crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic DUMPFILE: /var/crash/202505061152/dump.202505061152 [PARTIAL DUMP] CPUS: 16 DATE: Thu Jan 1 00:00:00 UTC 1970 UPTIME: 20 days, 16:15:55 LOAD AVERAGE: 0.06, 0.04, 0.01 TASKS: 347 NODENAME: RELEASE: 6.14.0-15-generic VERSION: #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 MACHINE: ppc64le (2500 Mhz) MEMORY: 25 GB PANIC: "Kernel panic - not syncing: sysrq triggered crash" PID: 78468 COMMAND: "bash" TASK: c0000000af9c0000 [THREAD_INFO: c0000000af9c0000] CPU: 11 STATE: TASK_RUNNING (PANIC) == Barnali Guha Thakurata == Hi Aditya and Kowshik, I am now able to read the dump with the above patch: root@:~/barnali/crash# crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic /root/myDump crash 9.0.0 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. WARNING: vmcoreinfo is empty, will read from symbols GNU gdb (GDB) 16.2 Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic DUMPFILE: /root/myDump [PARTIAL DUMP] CPUS: 4 DATE: Thu Jan 1 00:00:00 UTC 1970 UPTIME: 00:20:56 LOAD AVERAGE: 0.00, 0.01, 0.00 TASKS: 161 NODENAME: RELEASE: 6.14.0-15-generic VERSION: #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 MACHINE: ppc64le (2500 Mhz) MEMORY: 5 GB PANIC: "" PID: 0 COMMAND: "swapper/0" TASK: c000000003c0a380 (1 of 4) [THREAD_INFO: c000000003c0a380] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found == Barnali Guha Thakurata == The patch is pending for review in mailing list. The patch: https://lists.crash- utility.osci.io/archives/list/[email protected] utility.osci.io/thread/K3KA76AX6AIDB3ZKRQAEMNIWUTULZQ2S/ Hi Kowshik, Could you please test the above patch? == Kowshik Jois B S == I verified this bug on the recent Ubuntu 25.04 and below are the findings. Host ENV: root@:~# uname -a Linux 6.14.0-23-generic #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 ppc64le ppc64le ppc64le GNU/Linux root@:~# qemu-system-ppc64 --version QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5) Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers root@:~# libvirtd --version libvirtd (libvirt) 11.0.0 Guest ENV: root@:~# uname -a Linux 6.14.0-23-generic #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 ppc64le ppc64le ppc64le GNU/Linux LPAR is in KVM capable mode and in the guest's xml, domain type is set to kvm on a P10 machine. With the distro provided crash tool, I tried 'virsh dump' analysis. I am able to create 'virsh dump' and analyze the same with distro provided crash tool. root@:~# virsh dump Ubuntu KVMFormat --memory-only --format=kdump-zlib Domain 'Ubuntu' dumped to KVMFormat root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic ./KVMFormat crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... please wait... (gathering module symbol data) WARNING: cannot access vmalloc'd module memory KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic DUMPFILE: ./KVMFormat [PARTIAL DUMP] CPUS: 4 DATE: Thu Jan 1 00:00:00 UTC 1970 UPTIME: 02:00:28 LOAD AVERAGE: 0.00, 0.00, 0.00 TASKS: 154 NODENAME: RELEASE: 6.14.0-23-generic VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 MACHINE: ppc64le (3450 Mhz) MEMORY: 5 GB PANIC: "" PID: 0 COMMAND: "swapper/0" TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found crash> Exactly the same behavior is observed even when the dump is collected without mentioning the format like below: root@:~# virsh dump Ubuntu KVMNoFormat --memory-only Domain 'Ubuntu' dumped to KVMNoFormat Now I changed the domain type to 'qemu' in the guest xml to make the guest running in non-KVM mode and tried the same experiments with the distro provided crash. First I tried by specifying the format and I was able to collect and analyze the virsh dump. root@:~# virsh dump Ubuntu NonKVMFormat --memory-only --format=kdump- zlib Domain 'Ubuntu' dumped to NonKVMFormat root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic ./NonKVMFormat crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... please wait... (gathering module symbol data) WARNING: cannot access vmalloc'd module memory KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic DUMPFILE: ./NonKVMFormat [PARTIAL DUMP] CPUS: 4 DATE: Thu Jan 1 00:00:00 UTC 1970 UPTIME: 00:01:57 LOAD AVERAGE: 1.40, 0.83, 0.32 TASKS: 164 NODENAME: RELEASE: 6.14.0-23-generic VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 MACHINE: ppc64le (1000 Mhz) MEMORY: 5 GB PANIC: "" PID: 0 COMMAND: "swapper/0" TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found crash> Similar results observed even when tried without mentioning the format. No issues found in this case as well. In Summary, the crash tool is working fine in all the scenarios with the distro provided crash tool itself. Looks like the issue is fixed by distro in recent versions. Hence, the patch is not required. I think we can close this bug but before it makes sense to try the similar things when the LPAR is not in KVM capable mode. which I will try once and update the findings in the next comment. == Kowshik Jois B S == I changed the LPAR configuration by switching the KVM capable attribute in the HMC. Guest's xml has domain type set to qemu. I took the dump of the guest by specifying the format and I could reproduce the issue with distro provided crash tool. root@:~# virsh dump Ubuntu NonKVMFormat1 --memory-only --format=kdump- zlib Domain 'Ubuntu' dumped to NonKVMFormat1 root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic ./NonKVMFormat1 crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... crash: invalid kernel virtual address: 3ce2ffefe8410020 type: "first vmlist addr" Errors like the one above typically occur when the kernel and memory source do not match. These are the files being used: KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic Same error is observed even when the dump is collected without specifying the format with the distro provided kernel. Then, I cloned the crash tool from https://github.com/crash-utility/crash.git and applied the patch given in Comment 6. With the patched version of crash, I am not seeing the issue getting reproduced anymore. root@:~/crash# ./crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic ../NonKVMFormat1 crash 9.0.0++ Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. WARNING: vmcoreinfo is empty, will read from symbols GNU gdb (GDB) 16.2 Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic DUMPFILE: ../NonKVMFormat1 [PARTIAL DUMP] CPUS: 4 DATE: Thu Jan 1 00:00:00 UTC 1970 UPTIME: 00:01:12 LOAD AVERAGE: 2.73, 0.91, 0.32 TASKS: 163 NODENAME: RELEASE: 6.14.0-23-generic VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 MACHINE: ppc64le (1000 Mhz) MEMORY: 5 GB PANIC: "" PID: 0 COMMAND: "swapper/0" TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found crash> In summary, the issue can be seen only if the LPAR is not running in the KVM Capable mode and the proposed patch is fixing the issue. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/2117099/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
