** Description changed: + [Impact] + + * Users migrating between 16.04 on power8 and 18.04 on poer9 can not + migrate back (the very same guest) + + * IBM identified a set of qemu fixes that will make the migration work + as it is not impossible but mostly due to a misadvertisement in + features in the p8 compat mode. + + [Test Case] + + * Get two hosts: + - Ubuntu 16.04 on power8 + - Ubuntu 18.04 on power9 + * on 16.04 spawn a guest and set power 8 compat in libvirt + <cpu mode='host-model'> + <model>power8</model> + </cpu> + * Migrate the guest to the 18.04 host + * Migrate the guest back to the 16.04 host + + [Regression Potential] + + * The changes are ppc64 only and identified/suggested by IBM so any + fallout should be limited to ppc64 which is their domain + * Furthermore it only affects pre 2.8 guest types, so anything spawned + on 18.04 is unaffected by the changes (as it will default to 2.11) + * The remaining regression "potential" would be around guests migrated + from 16.04 to 18.04 that after the update behave slightly different + (more correct, but different) + + [Other Info] + + * Fixes are only needed on the newer qemu to handle older guests (pre + 2.8 and xenial is 2.5) with power8 compat in a way that they can + be migrated back. + + == Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 == ---Problem Description--- Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails # virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose - root@9.47.70.201's password: + root@9.47.70.201's password: Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu' 2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument - - This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4). - - Machine Type = Boston - + + This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and + QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu + 18.04) is not compatible with older qemu in (Ubuntu 16.04.4). + + Machine Type = Boston + ---Steps to Reproduce--- 1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22, <cpu mode='host-model'> - <model>power8</model> + <model>power8</model> </cpu> 2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration 3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected, # virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose Migration: [100 %] 4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken, # virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose - root@9.47.70.201's password: + root@9.47.70.201's password: Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu' 2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument ---uname output--- Ubuntu 18.04 P9 Host: Kernel # uname -a Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Qemu: # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities Libvirt: # dpkg -l | grep libvirt ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library Ubuntu 16.04.4 P8 Host: Kernel: # uname -a Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Qemu: # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc) ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities Libvirt: # dpkg -l | grep libvirt ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings Ubuntu 16.04.4 P8 Guest: Kernel: # uname -a Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux - Attachment: 1. Libvirtd debug logs: source and destination host 2. Sosreport: source host, destination host and Guest ---Debugger--- A debugger is not configured - - Userspace tool common name: qemu - - Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7 - - The userspace tool has the following bit modes: ppc64le - - Userspace tool obtained from project website: na - + + Userspace tool common name: qemu + + Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7 + + The userspace tool has the following bit modes: ppc64le + + Userspace tool obtained from project website: na == Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 == Backward migration had three distinct issues. Fixes are now upstream: https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7 https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6 https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well. == Comment: #6 - Balamuruhan S <balamuruh...@in.ibm.com> - 2018-06-08 07:49:51 == Tested with latest qemu build available for Ubuntu 18.04, Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not included in Qemu 1:2.11+dfsg-1ubuntu7.2 Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host - # virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose + # virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose Migration: [100 %] Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host # virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose - root@9.47.70.201's password: + root@9.47.70.201's password: Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu' 2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument P9 Ubuntu 18.04: Qemu # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc) ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities P8 Ubuntu 16.04.4: Qemu # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc) ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities - == Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 == According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018. According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018. greg@wslp001:~$ lsb_release -d Description: Ubuntu 18.04.1 LTS greg@wslp001:~$ qemu-system-ppc64 --version QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1809083 Title: [Ubuntu 18.04] Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host is broken To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1809083/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
