TL:DR: GNUTLS is liable to crash QEMU when live migration is run with TLS enabled and a return path channel is present, if approx 64 GB of data is transferred. This is easily triggered in a 16 GB VM with 4 CPUs, by running 'stress-ng --vm 4 --vm-bytes 80%' to prevent convergance until 64 GB of RAM has been copied. Then triggering post-copy switchover, or removing the stress workload to allow completion, will crash it.
The only live migration scenario that should avoid this danger is multifd, since the high volume data transfers are handled in dedicated TCP connections which are unidirectional. The main bi-directionl TCP connection is only for co-ordination purposes This patch implements a workaround that will prevent future QEMU versions from triggering the crash. The only way to avoid the crash with *existing* running QEMU processes is to change the TLS cipher priority string to avoid use of AES with TLS 1.3. This can be done with the 'priority' field in the 'tls-creds-x509' object.eg -object tls-creds-x509,id=tls0,priority=NORMAL:-AES-256-GCM:-AES-128-GCM:-AES-128-CCM which should force the use of CHACHA20-POLY1305 which does not require TLS re-keying after 16 million sent records (64 GB of migration data). https://gitlab.com/qemu-project/qemu/-/issues/1937 On RHEL/Fedora distros you can also use the system wide crypto priorities to override this from the migration *target* host by creating /etc/crypto-policies/local.d/gnutls-qemu.config containing QEMU=NONE:+ECDHE-RSA:+ECDHE-ECDSA:+RSA:+DHE-RSA:+GROUP-X25519:+GROUP-X448:+GROUP-SECP256R1:+GROUP-SECP384R1:+GROUP-SECP521R1:+GROUP-FF and running 'update-crypto-policies'. I recommend the QEMU level 'tls-creds-x509' workaround though, which new libvirt patches can soon do: https://lists.libvirt.org/archives/list/de...@lists.libvirt.org/thread/LX5KMIUFZSP5DPUXKJDFYBZI5TIE3E5N/ Daniel P. Berrangé (4): crypto: implement workaround for GNUTLS thread safety problems io: add support for activating TLS thread safety workaround migration: activate TLS thread safety workaround crypto: add tracing & warning about GNUTLS countermeasures crypto/tlssession.c | 99 +++++++++++++++++++++++++++++++++-- crypto/trace-events | 2 + include/crypto/tlssession.h | 14 +++++ include/io/channel.h | 1 + io/channel-tls.c | 5 ++ meson.build | 9 ++++ meson_options.txt | 2 + migration/tls.c | 9 ++++ scripts/meson-buildoptions.sh | 5 ++ 9 files changed, 143 insertions(+), 3 deletions(-) -- 2.50.1
