Public bug reported:
== Kowshik Jois B S ==
---Problem Description---
crash: invalid kernel virtual address: fbc10030fba10030 type: "first vmlist
addr"
error is observed while trying to analyse the dump collected with "virsh dump"
using the crash tool.
Machine Type = P10 Rainier
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. collect the dump of a KVM guest on Ubuntu 25.04
==> virsh dump ub2504guest1 myDump --memory-only --format=kdump-zlib
2. Try to analyze the collected dump using the crash tool.
==> crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic ./myDump
---uname output---
Linux 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
Userspace rpm: crash
The userspace tool has the following bit modes: 64-bit
Userspace tool common name: crash
Userspace tool obtained from project website: na
output:
# crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic myDump
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
crash: invalid kernel virtual address: fbc10030fba10030 type: "first
vmlist addr"
Errors like the one above typically occur when the kernel and memory source
do not match. These are the files being used:
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
DUMPFILE: myDump
HOST ENV:
# uname -a
Linux 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
# qemu-system-ppc64 --version
QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
root@:~# libvirtd --version
libvirtd (libvirt) 11.0.0
# apt list --installed | grep crash
WARNING: apt does not have a stable CLI interface. Use with caution in
scripts.
crash/plucky,now 8.0.6-1ubuntu1 ppc64el [installed]
linux-crashdump/plucky,now 6.14.0-15.15 ppc64el [installed]
# apt list --installed | grep dbgsym
WARNING: apt does not have a stable CLI interface. Use with caution in
scripts.
linux-image-6.14.0-15-generic-dbgsym/now 6.14.0-15.15 ppc64el
[installed,local]
Guest ENV:
# uname -a
Linux ub2504guest1 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC
2025 ppc64le ppc64le ppc64le GNU/Linux
This issue is seen only with the virsh dump. KDUMP/FADUMP collected can
be analysed with the tool normally.
# crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
/var/crash/202505061152/dump.202505061152
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
DUMPFILE: /var/crash/202505061152/dump.202505061152 [PARTIAL DUMP]
CPUS: 16
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 20 days, 16:15:55
LOAD AVERAGE: 0.06, 0.04, 0.01
TASKS: 347
NODENAME:
RELEASE: 6.14.0-15-generic
VERSION: #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025
MACHINE: ppc64le (2500 Mhz)
MEMORY: 25 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 78468
COMMAND: "bash"
TASK: c0000000af9c0000 [THREAD_INFO: c0000000af9c0000]
CPU: 11
STATE: TASK_RUNNING (PANIC)
== Barnali Guha Thakurata ==
Hi Aditya and Kowshik, I am now able to read the dump with the above patch:
root@:~/barnali/crash# crash
/usr/lib/debug/boot/vmlinux-6.14.0-15-generic /root/myDump
crash 9.0.0
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
WARNING: vmcoreinfo is empty, will read from symbols
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
DUMPFILE: /root/myDump [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 00:20:56
LOAD AVERAGE: 0.00, 0.01, 0.00
TASKS: 161
NODENAME:
RELEASE: 6.14.0-15-generic
VERSION: #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025
MACHINE: ppc64le (2500 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a380 (1 of 4) [THREAD_INFO: c000000003c0a380]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
== Barnali Guha Thakurata ==
The patch is pending for review in mailing list.
The patch: https://lists.crash-
utility.osci.io/archives/list/[email protected]
utility.osci.io/thread/K3KA76AX6AIDB3ZKRQAEMNIWUTULZQ2S/
Hi Kowshik, Could you please test the above patch?
== Kowshik Jois B S ==
I verified this bug on the recent Ubuntu 25.04 and below are the findings.
Host ENV:
root@:~# uname -a
Linux 6.14.0-23-generic #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
root@:~# qemu-system-ppc64 --version
QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
root@:~# libvirtd --version
libvirtd (libvirt) 11.0.0
Guest ENV:
root@:~# uname -a
Linux 6.14.0-23-generic #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
LPAR is in KVM capable mode and in the guest's xml, domain type is set
to kvm on a P10 machine. With the distro provided crash tool, I tried
'virsh dump' analysis.
I am able to create 'virsh dump' and analyze the same with distro
provided crash tool.
root@:~# virsh dump Ubuntu KVMFormat --memory-only --format=kdump-zlib
Domain 'Ubuntu' dumped to KVMFormat
root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic ./KVMFormat
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
please wait... (gathering module symbol data)
WARNING: cannot access vmalloc'd module memory
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
DUMPFILE: ./KVMFormat [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 02:00:28
LOAD AVERAGE: 0.00, 0.00, 0.00
TASKS: 154
NODENAME:
RELEASE: 6.14.0-23-generic
VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025
MACHINE: ppc64le (3450 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash>
Exactly the same behavior is observed even when the dump is collected without
mentioning the format like below:
root@:~# virsh dump Ubuntu KVMNoFormat --memory-only
Domain 'Ubuntu' dumped to KVMNoFormat
Now I changed the domain type to 'qemu' in the guest xml to make the
guest running in non-KVM mode and tried the same experiments with the
distro provided crash.
First I tried by specifying the format and I was able to collect and
analyze the virsh dump.
root@:~# virsh dump Ubuntu NonKVMFormat --memory-only --format=kdump-
zlib
Domain 'Ubuntu' dumped to NonKVMFormat
root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
./NonKVMFormat
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
please wait... (gathering module symbol data)
WARNING: cannot access vmalloc'd module memory
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
DUMPFILE: ./NonKVMFormat [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 00:01:57
LOAD AVERAGE: 1.40, 0.83, 0.32
TASKS: 164
NODENAME:
RELEASE: 6.14.0-23-generic
VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025
MACHINE: ppc64le (1000 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash>
Similar results observed even when tried without mentioning the format. No
issues found in this case as well.
In Summary, the crash tool is working fine in all the scenarios with the
distro provided crash tool itself. Looks like the issue is fixed by
distro in recent versions. Hence, the patch is not required.
I think we can close this bug but before it makes sense to try the
similar things when the LPAR is not in KVM capable mode. which I will
try once and update the findings in the next comment.
== Kowshik Jois B S ==
I changed the LPAR configuration by switching the KVM capable attribute in the
HMC.
Guest's xml has domain type set to qemu. I took the dump of the guest by
specifying the format and I could reproduce the issue with distro
provided crash tool.
root@:~# virsh dump Ubuntu NonKVMFormat1 --memory-only --format=kdump-
zlib
Domain 'Ubuntu' dumped to NonKVMFormat1
root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
./NonKVMFormat1
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
crash: invalid kernel virtual address: 3ce2ffefe8410020 type: "first
vmlist addr"
Errors like the one above typically occur when the kernel and memory source
do not match. These are the files being used:
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
Same error is observed even when the dump is collected without specifying the
format with the distro provided kernel.
Then, I cloned the crash tool from https://github.com/crash-utility/crash.git
and applied the patch given in Comment 6.
With the patched version of crash, I am not seeing the issue getting
reproduced anymore.
root@:~/crash# ./crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
../NonKVMFormat1
crash 9.0.0++
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
WARNING: vmcoreinfo is empty, will read from symbols
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
DUMPFILE: ../NonKVMFormat1 [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 00:01:12
LOAD AVERAGE: 2.73, 0.91, 0.32
TASKS: 163
NODENAME:
RELEASE: 6.14.0-23-generic
VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025
MACHINE: ppc64le (1000 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash>
In summary, the issue can be seen only if the LPAR is not running in the
KVM Capable mode and the proposed patch is fixing the issue.
** Affects: crash (Ubuntu)
Importance: Undecided
Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
Status: New
** Tags: architecture-ppc64le bugnameltc-212620 severity-high
targetmilestone-inin2504
** Tags added: architecture-ppc64le bugnameltc-212620 severity-high
targetmilestone-inin2504
** Changed in: ubuntu
Assignee: (unassigned) => Ubuntu on IBM Power Systems Bug Triage
(ubuntu-power-triage)
** Package changed: ubuntu => crash (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to crash in Ubuntu.
https://bugs.launchpad.net/bugs/2117099
Title:
"invalid kernel virtual address" error while analysing virsh dump
Status in crash package in Ubuntu:
New
Bug description:
== Kowshik Jois B S ==
---Problem Description---
crash: invalid kernel virtual address: fbc10030fba10030 type: "first vmlist
addr"
error is observed while trying to analyse the dump collected with "virsh
dump" using the crash tool.
Machine Type = P10 Rainier
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. collect the dump of a KVM guest on Ubuntu 25.04
==> virsh dump ub2504guest1 myDump --memory-only --format=kdump-zlib
2. Try to analyze the collected dump using the crash tool.
==> crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic ./myDump
---uname output---
Linux 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
Userspace rpm: crash
The userspace tool has the following bit modes: 64-bit
Userspace tool common name: crash
Userspace tool obtained from project website: na
output:
# crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic myDump
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
crash: invalid kernel virtual address: fbc10030fba10030 type: "first
vmlist addr"
Errors like the one above typically occur when the kernel and memory source
do not match. These are the files being used:
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
DUMPFILE: myDump
HOST ENV:
# uname -a
Linux 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
# qemu-system-ppc64 --version
QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
root@:~# libvirtd --version
libvirtd (libvirt) 11.0.0
# apt list --installed | grep crash
WARNING: apt does not have a stable CLI interface. Use with caution in
scripts.
crash/plucky,now 8.0.6-1ubuntu1 ppc64el [installed]
linux-crashdump/plucky,now 6.14.0-15.15 ppc64el [installed]
# apt list --installed | grep dbgsym
WARNING: apt does not have a stable CLI interface. Use with caution in
scripts.
linux-image-6.14.0-15-generic-dbgsym/now 6.14.0-15.15 ppc64el
[installed,local]
Guest ENV:
# uname -a
Linux ub2504guest1 6.14.0-15-generic #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC
2025 ppc64le ppc64le ppc64le GNU/Linux
This issue is seen only with the virsh dump. KDUMP/FADUMP collected
can be analysed with the tool normally.
# crash /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
/var/crash/202505061152/dump.202505061152
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
DUMPFILE: /var/crash/202505061152/dump.202505061152 [PARTIAL DUMP]
CPUS: 16
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 20 days, 16:15:55
LOAD AVERAGE: 0.06, 0.04, 0.01
TASKS: 347
NODENAME:
RELEASE: 6.14.0-15-generic
VERSION: #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025
MACHINE: ppc64le (2500 Mhz)
MEMORY: 25 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 78468
COMMAND: "bash"
TASK: c0000000af9c0000 [THREAD_INFO: c0000000af9c0000]
CPU: 11
STATE: TASK_RUNNING (PANIC)
== Barnali Guha Thakurata ==
Hi Aditya and Kowshik, I am now able to read the dump with the above patch:
root@:~/barnali/crash# crash
/usr/lib/debug/boot/vmlinux-6.14.0-15-generic /root/myDump
crash 9.0.0
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
WARNING: vmcoreinfo is empty, will read from symbols
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-15-generic
DUMPFILE: /root/myDump [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 00:20:56
LOAD AVERAGE: 0.00, 0.01, 0.00
TASKS: 161
NODENAME:
RELEASE: 6.14.0-15-generic
VERSION: #15-Ubuntu SMP Sun Apr 6 14:52:42 UTC 2025
MACHINE: ppc64le (2500 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a380 (1 of 4) [THREAD_INFO: c000000003c0a380]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
== Barnali Guha Thakurata ==
The patch is pending for review in mailing list.
The patch: https://lists.crash-
utility.osci.io/archives/list/[email protected]
utility.osci.io/thread/K3KA76AX6AIDB3ZKRQAEMNIWUTULZQ2S/
Hi Kowshik, Could you please test the above patch?
== Kowshik Jois B S ==
I verified this bug on the recent Ubuntu 25.04 and below are the findings.
Host ENV:
root@:~# uname -a
Linux 6.14.0-23-generic #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
root@:~# qemu-system-ppc64 --version
QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
root@:~# libvirtd --version
libvirtd (libvirt) 11.0.0
Guest ENV:
root@:~# uname -a
Linux 6.14.0-23-generic #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025 ppc64le
ppc64le ppc64le GNU/Linux
LPAR is in KVM capable mode and in the guest's xml, domain type is set
to kvm on a P10 machine. With the distro provided crash tool, I tried
'virsh dump' analysis.
I am able to create 'virsh dump' and analyze the same with distro
provided crash tool.
root@:~# virsh dump Ubuntu KVMFormat --memory-only --format=kdump-zlib
Domain 'Ubuntu' dumped to KVMFormat
root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
./KVMFormat
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
please wait... (gathering module symbol data)
WARNING: cannot access vmalloc'd module memory
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
DUMPFILE: ./KVMFormat [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 02:00:28
LOAD AVERAGE: 0.00, 0.00, 0.00
TASKS: 154
NODENAME:
RELEASE: 6.14.0-23-generic
VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025
MACHINE: ppc64le (3450 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash>
Exactly the same behavior is observed even when the dump is collected without
mentioning the format like below:
root@:~# virsh dump Ubuntu KVMNoFormat --memory-only
Domain 'Ubuntu' dumped to KVMNoFormat
Now I changed the domain type to 'qemu' in the guest xml to make the
guest running in non-KVM mode and tried the same experiments with the
distro provided crash.
First I tried by specifying the format and I was able to collect and
analyze the virsh dump.
root@:~# virsh dump Ubuntu NonKVMFormat --memory-only --format=kdump-
zlib
Domain 'Ubuntu' dumped to NonKVMFormat
root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
./NonKVMFormat
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
please wait... (gathering module symbol data)
WARNING: cannot access vmalloc'd module memory
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
DUMPFILE: ./NonKVMFormat [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 00:01:57
LOAD AVERAGE: 1.40, 0.83, 0.32
TASKS: 164
NODENAME:
RELEASE: 6.14.0-23-generic
VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025
MACHINE: ppc64le (1000 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash>
Similar results observed even when tried without mentioning the format. No
issues found in this case as well.
In Summary, the crash tool is working fine in all the scenarios with
the distro provided crash tool itself. Looks like the issue is fixed
by distro in recent versions. Hence, the patch is not required.
I think we can close this bug but before it makes sense to try the
similar things when the LPAR is not in KVM capable mode. which I will
try once and update the findings in the next comment.
== Kowshik Jois B S ==
I changed the LPAR configuration by switching the KVM capable attribute in
the HMC.
Guest's xml has domain type set to qemu. I took the dump of the guest
by specifying the format and I could reproduce the issue with distro
provided crash tool.
root@:~# virsh dump Ubuntu NonKVMFormat1 --memory-only --format=kdump-
zlib
Domain 'Ubuntu' dumped to NonKVMFormat1
root@:~# crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
./NonKVMFormat1
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
crash: invalid kernel virtual address: 3ce2ffefe8410020 type: "first
vmlist addr"
Errors like the one above typically occur when the kernel and memory source
do not match. These are the files being used:
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
Same error is observed even when the dump is collected without specifying the
format with the distro provided kernel.
Then, I cloned the crash tool from https://github.com/crash-utility/crash.git
and applied the patch given in Comment 6.
With the patched version of crash, I am not seeing the issue getting
reproduced anymore.
root@:~/crash# ./crash /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
../NonKVMFormat1
crash 9.0.0++
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
WARNING: vmcoreinfo is empty, will read from symbols
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: /usr/lib/debug/boot/vmlinux-6.14.0-23-generic
DUMPFILE: ../NonKVMFormat1 [PARTIAL DUMP]
CPUS: 4
DATE: Thu Jan 1 00:00:00 UTC 1970
UPTIME: 00:01:12
LOAD AVERAGE: 2.73, 0.91, 0.32
TASKS: 163
NODENAME:
RELEASE: 6.14.0-23-generic
VERSION: #23-Ubuntu SMP Fri Jun 13 22:20:18 UTC 2025
MACHINE: ppc64le (1000 Mhz)
MEMORY: 5 GB
PANIC: ""
PID: 0
COMMAND: "swapper/0"
TASK: c000000003c0a700 (1 of 4) [THREAD_INFO: c000000003c0a700]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash>
In summary, the issue can be seen only if the LPAR is not running in
the KVM Capable mode and the proposed patch is fixing the issue.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/crash/+bug/2117099/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
