Package: libvirt-daemon
Version: 7.0.0-3
Severity: important
File: /usr/sbin/libvirtd
Dear Maintainer,
* What led up to the situation?
Attempting live migration of VM between hosts, either from virt-manager on
a seperate workstation or from the host itself via terminal.
Example - virsh migrate --live web02 qemu+ssh://hypervisor01:64228/system
* What exactly did you do (or not do) that was effective (or ineffective)?
Investigated the following logs.
/var/log/syslog
Apr 18 09:25:45 hypervisor04 libvirtd[542]: Cannot start job (query, none,
none) for domain email01; current job is (none, none, migration in) owned by (0
<null>, 0 <null>, 0 remoteDispatchDomainMigratePrepare3Params (flags=0x19)) for
(0s, 0s, 305s)
/var/log/kern.log
Apr 17 22:45:45 hypervisor05 kernel: [ 804.114785] Internal error: Oops:
96000004 [#1] SMP
Apr 17 22:57:49 hypervisor05 kernel: [ 206.952482] Internal error: Oops:
96000004 [#1] SMP
Apr 18 01:06:11 hypervisor05 kernel: [ 463.133575] Internal error: Oops:
96000004 [#1] SMP
Apr 18 11:12:39 hypervisor05 kernel: [36851.073954] Internal error: Oops:
96000004 [#2] SMP
Apr 18 11:29:19 hypervisor05 kernel: [37850.896463] Internal error: Oops:
96000004 [#3] SMP
error in dmesg
[ 324.673078] audit: type=1400 audit(1650240228.486:23):
apparmor="STATUS" operation="profile_replace" profile="unconfined"
name="libvirt-72799f47-939c-4415-92c3-73ec371425fd" pid=979
comm="apparmor_parser"
[ 324.768028] audit: type=1400 audit(1650240228.582:24):
apparmor="DENIED" operation="capable" profile="libvirtd" pid=542
comm="rpc-worker" capability=39 capname="bpf"
[ 324.774241] audit: type=1400 audit(1650240228.586:25):
apparmor="DENIED" operation="capable" profile="libvirtd" pid=542
comm="rpc-worker" capability=38 capname="perfmon"
[ 326.770324] audit: type=1400 audit(1650240230.582:26):
apparmor="DENIED" operation="capable" profile="libvirtd" pid=542
comm="rpc-worker" capability=39 capname="bpf"
I added
capability bpf,
capability perfmon,
to /etc/apparmor.d/usr.sbin.libvirtd which resolved the DENIED errors but
did not resolve the live migration failures.
* What was the outcome of this action?
The following errors were produced.
kernel:\[37850.896463\] Internal error: Oops: 96000004 \[#3\] SMP
Message from syslogd@hypervisor05 at Apr 18 11:29:19 ...
kernel:\[37851.195226\] Code: 910003fd f9000bf3 2a0003f3 97ff7164
(b95ed801)
The VM that was submitted for migration ends up hung in a paused state.
The only way to recover it is force power off on the VM, then 'sudo systemctl
restart libvirtd.service'. The VM can then be powered on again normally.
* What outcome did you expect instead?
Live migration to complete successfully which has been the case on eariler
kernel versions. However at this time I do not know which kernel versions
worked other than the one it shipped with which was as follows.
linux-image-5.10.0-8-arm64 5.10.46-5 arm64 Linux 5.10 for 64-bit ARMv8
machines (signed)
-- System Information:
Debian Release: 11.3
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: arm64 (aarch64)
Kernel: Linux 5.10.0-13-arm64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_CRAP
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages libvirt-daemon depends on:
ii libblkid1 2.36.1-8+deb11u1
ii libc6 2.31-13+deb11u3
ii libdevmapper1.02.1 2:1.02.175-2.1
ii libgcc-s1 10.2.1-6
ii libglib2.0-0 2.66.8-1
ii libnetcf1 1:0.2.8-1.1
ii libparted2 3.4-1
ii libpcap0.8 1.10.0-2
ii libpciaccess0 0.16-1
ii libselinux1 3.1-3
ii libudev1 247.3-7
ii libvirt-daemon-driver-qemu 7.0.0-3
ii libvirt0 7.0.0-3
ii libxml2 2.9.10+dfsg-6.7+deb11u1
Versions of packages libvirt-daemon recommends:
ii libvirt-daemon-driver-lxc 7.0.0-3
pn libvirt-daemon-driver-vbox <none>
ii libvirt-daemon-driver-xen 7.0.0-3
ii libxml2-utils 2.9.10+dfsg-6.7+deb11u1
ii netcat-openbsd 1.217-3
ii qemu-system 1:5.2+dfsg-11+deb11u1
ii qemu-system-arm [qemu-kvm] 1:5.2+dfsg-11+deb11u1
Versions of packages libvirt-daemon suggests:
pn libvirt-daemon-driver-storage-gluster <none>
pn libvirt-daemon-driver-storage-iscsi-direct <none>
pn libvirt-daemon-driver-storage-rbd <none>
pn libvirt-daemon-driver-storage-zfs <none>
ii libvirt-daemon-system 7.0.0-3
pn numad <none>
-- no debconf information
