Assigning to kernel team initially, although this may be a qemu/libvirt issue.
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1810392 Title: [Ubuntu 1810] Migrating Power9 guest back to source causes kernel Oops inside guest Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Bug description: ---Problem Description--- Migrating back Power9 to source cause Kernel Oops inside guest Contact Information = Balamuruhan S / balamuruh...@in.ibm.com ---uname output--- source and target host Kernel: 4.18.0-8-generic Guest Kernel: 4.18.0-8-generic Machine Type = Boston ---Debugger--- A debugger is not configured ---Steps to Reproduce--- Due to Bug 172310, Turn off the firewalld in source and target host machines. 1. Define a healthy guest and placed image in NFS share. 2. Mounted the image location in destination and start the guest to perform migration. 3. Perform migration from source to target, 08:07:54 INFO | Migrating VM avocado-vt-vm1-migration from qemu:///system to qemu+ssh://9.40.194.13/system 08:07:54 DEBUG| Running virsh command: migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.194.13/system --timeout 60 08:07:54 INFO | Running '/usr/bin/virsh -c 'qemu:///system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.194.13/system --timeout 60' 08:08:21 DEBUG| [stdout] 08:08:21 INFO | Command '/usr/bin/virsh -c 'qemu:///system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.194.13/system --timeout 60' finished with 0 after 26.9465551376s 08:08:21 DEBUG| status: 0 08:08:21 DEBUG| stdout: 08:08:21 DEBUG| stderr: 08:08:21 INFO | Migration exit status: 0 4. check VM uptime after successful migration, 08:07:44 INFO | Check VM uptime before migration: 57.66 5. Migration is successful from source to destination, now perform migration back to source 08:08:49 INFO | Migrating VM avocado-vt-vm1-migration from qemu+ssh://9.40.194.13/system to qemu:///system 08:08:49 DEBUG| Running virsh command: migrate --live --domain avocado-vt-vm1-migration --desturi qemu:///system --timeout 60 08:08:49 INFO | Running '/usr/bin/virsh -c 'qemu+ssh://9.40.194.13/system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu:///system --timeout 60' 08:09:22 DEBUG| [stdout] 08:09:22 INFO | Command '/usr/bin/virsh -c 'qemu+ssh://9.40.194.13/system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu:///system --timeout 60' finished with 0 after 32.8640539646s 08:09:22 DEBUG| status: 0 08:09:22 DEBUG| stdout: 08:09:22 DEBUG| stderr: 08:09:22 INFO | Migration exit status: 0 6. check uptime of VM after migration and it is observed that VM have gone for reboot 08:10:22 INFO | Check VM uptime after migrating back to source: 31.09 5. By checking the VM console Call Trace is observed as mentioned below The time stamp during the call trace observed inside guest matches with the timestamp during which migration is triggered. Stack trace output: no Attachment: 1. VM sosreport 2. source and target sosreport 3. VM console log with call trace 4. Guest xml 5. Test job log Oops output: 2018-10-26 08:08:26: root@ubuntu1810:~# 2018-10-26 08:09:11: [ 144.512549] Unable to handle kernel paging request for data at address 0x71ff9672fa0e0989 2018-10-26 08:09:11: [ 144.512655] Faulting instruction address: 0xc0000000003af7cc 2018-10-26 08:09:11: [ 144.512741] Oops: Kernel access of bad area, sig: 11 [#1] 2018-10-26 08:09:11: [ 144.512789] LE SMP NR_CPUS=2048 NUMA pSeries 2018-10-26 08:09:11: [ 144.512859] Modules linked in: iscsi_target_mod target_core_mod xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bpfilter kvm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto crct10dif_vpmsum sch_fq_codel nfsd ib_iser auth_rpcgss rdma_cm nfs_acl iw_cm lockd ib_cm grace ib_core sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear virtio_net net_failover crc32c_vpmsum virtio_blk virtio_scsi failover 2018-10-26 08:09:11: [ 144.513801] CPU: 5 PID: 1072 Comm: systemd-network Not tainted 4.18.0-8-generic #9-Ubuntu 2018-10-26 08:09:11: [ 144.513872] NIP: c0000000003af7cc LR: c0000000003af7b8 CTR: c000000000c9ea80 2018-10-26 08:09:11: [ 144.513943] REGS: c0000000f19ab910 TRAP: 0380 Not tainted (4.18.0-8-generic) 2018-10-26 08:09:11: [ 144.514025] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24002824 XER: 00000000 2018-10-26 08:09:11: [ 144.514102] CFAR: c0000000000174a0 IRQMASK: 0 2018-10-26 08:09:11: [ 144.514102] GPR00: c0000000003af670 c0000000f19abb90 c00000000178c800 0000000000000000 2018-10-26 08:09:11: [ 144.514102] GPR04: 0000000000000000 0000000000000007 0000000000000000 0000000000000006 2018-10-26 08:09:11: [ 144.514102] GPR08: b1ff967250115c09 0000000000000000 0000000000000000 c000000000d71000 2018-10-26 08:09:11: [ 144.514102] GPR12: c000000000c9ea80 c00000000fff8a00 0000000000000000 0000000000000000 2018-10-26 08:09:11: [ 144.514102] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 2018-10-26 08:09:11: [ 144.514102] GPR20: 0000000000000000 0000000000000000 0000000000000000 c0000000f19abde0 2018-10-26 08:09:11: [ 144.514102] GPR24: b1ff967250115c09 c0000000aa1f5580 71ff9672fa0e0989 c0000000fd017380 2018-10-26 08:09:11: [ 144.514102] GPR28: c000000000b5d188 00000000006000c0 c0000000aa1f5580 c0000000058a7c80 2018-10-26 08:09:11: [ 144.515013] NIP [c0000000003af7cc] kmem_cache_alloc+0x30c/0x360 2018-10-26 08:09:11: [ 144.515102] LR [c0000000003af7b8] kmem_cache_alloc+0x2f8/0x360 2018-10-26 08:09:11: [ 144.515186] Call Trace: 2018-10-26 08:09:11: [ 144.515224] [c0000000f19abb90] [c0000000003af670] kmem_cache_alloc+0x1b0/0x360 (unreliable) 2018-10-26 08:09:11: [ 144.515337] [c0000000f19abc00] [c000000000b5d188] sk_prot_alloc+0x58/0x200 2018-10-26 08:09:11: [ 144.515426] [c0000000f19abc50] [c000000000b5d3e4] sk_alloc+0x54/0x220 2018-10-26 08:09:11: [ 144.515514] [c0000000f19abca0] [c000000000c9e8f0] unix_create1+0xe0/0x270 2018-10-26 08:09:11: [ 144.515601] [c0000000f19abcf0] [c000000000c9eb98] unix_create+0x118/0x150 2018-10-26 08:09:11: [ 144.515689] [c0000000f19abd10] [c000000000b58234] __sock_create+0x134/0x2c0 2018-10-26 08:09:11: [ 144.515775] [c0000000f19abdc0] [c000000000b59f90] __sys_socket+0x60/0x160 2018-10-26 08:09:11: [ 144.515860] [c0000000f19abe10] [c000000000b5a0bc] sys_socket+0x2c/0x40 2018-10-26 08:09:11: [ 144.515948] [c0000000f19abe30] [c00000000000b288] system_call+0x5c/0x70 2018-10-26 08:09:11: [ 144.516032] Instruction dump: 2018-10-26 08:09:11: [ 144.516088] 409efe1c 7c98c278 38a50001 7f3ac278 fb490000 f8aa0000 4bc67cc5 60000000 2018-10-26 08:09:11: [ 144.516195] 7fb9c040 419e0028 813f0020 e91f0138 <7d5a482a> 7d3a4a14 7d4a4278 7fa95040 2018-10-26 08:09:11: [ 144.516314] ---[ end trace ea4650f5afbe0331 ]--- Userspace tool common name: Qemu, Libvirt Userspace rpm: Qemu: # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu4 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu3 all PXE boot firmware - Compat EFI ROM images for qemu ii qemu-block-extra:ppc64el 1:2.12+dfsg-3ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.12+dfsg-3ubuntu7 ppc64el QEMU Full virtualization on x86 hardware ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system-common 1:2.12+dfsg-3ubuntu7 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-data 1:2.12+dfsg-3ubuntu7 all QEMU full system emulation (data files) ii qemu-system-gui 1:2.12+dfsg-3ubuntu7 ppc64el QEMU full system emulation binaries (user interface and audio support) ii qemu-system-ppc 1:2.12+dfsg-3ubuntu7 ppc64el QEMU full system emulation binaries (ppc) ii qemu-utils 1:2.12+dfsg-3ubuntu7 ppc64el QEMU utilities Libvirt: # dpkg -l | grep libvirt ii gir1.2-libvirt-glib-1.0:ppc64el 1.0.0-1 ppc64el GObject introspection files for the libvirt-glib library ii gir1.2-libvirt-sandbox-1.0 0.5.1+git20160404-1 ppc64el GObject introspection files for the libvirt-sandbox library ii libvirt-clients 4.6.0-2ubuntu3 ppc64el Programs for the libvirt library ii libvirt-daemon 4.6.0-2ubuntu3 ppc64el Virtualization daemon ii libvirt-daemon-driver-storage-gluster 4.6.0-2ubuntu3 ppc64el Virtualization daemon glusterfs storage driver ii libvirt-daemon-driver-storage-rbd 4.6.0-2ubuntu3 ppc64el Virtualization daemon RBD storage driver ii libvirt-daemon-driver-storage-sheepdog 4.6.0-2ubuntu3 ppc64el Virtualization daemon Sheedog storage driver ii libvirt-daemon-driver-storage-zfs 4.6.0-2ubuntu3 ppc64el Virtualization daemon ZFS storage driver ii libvirt-daemon-system 4.6.0-2ubuntu3 ppc64el Libvirt daemon configuration files ii libvirt-dbus 1.2.0-1 ppc64el libvirt D-Bus API bindings ii libvirt-dev:ppc64el 4.6.0-2ubuntu3 ppc64el development files for the libvirt library ii libvirt-doc 4.6.0-2ubuntu3 all documentation for the libvirt library ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library ii libvirt-glib-1.0-dev:ppc64el 1.0.0-1 ppc64el Development files for the libvirt-glib library ii libvirt-ocaml 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (runtime) ii libvirt-ocaml-dev 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (development files) ii libvirt-sandbox-1.0-5 0.5.1+git20160404-1 ppc64el Application sandbox toolkit shared library ii libvirt-sandbox-1.0-dev 0.5.1+git20160404-1 ppc64el Development files for libvirt-sandbox library ii libvirt-sanlock 4.6.0-2ubuntu3 ppc64el Sanlock plugin for virtlockd ii libvirt-wireshark 4.6.0-2ubuntu3 ppc64el Wireshark dissector for the libvirt protocol ii libvirt0:ppc64el 4.6.0-2ubuntu3 ppc64el library for interfacing with different virtualization systems ii libvirtodbc0 6.1.6+repack-0ubuntu9 ppc64el high-performance database - ODBC libraries ii libvirtualpg-dev:ppc64el 2.0.0~rc0-1 ppc64el VirtualPG development files ii libvirtualpg0:ppc64el 2.0.0~rc0-1 ppc64el VirtualPG shared library ii python-libvirt 4.6.0-1 ppc64el libvirt Python bindings To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1810392/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
