Hi Zhang Chen, >From your tracing log, it seems to be hanged in colo_flush_ram_cache()? Does it come across a dead loop there ? I'll test it by using the new qemu.
Thanks, Hailiang From: Zhang, Chen [mailto:[email protected]] Sent: Friday, May 15, 2020 11:16 AM To: Zhanghailiang <[email protected]>; Dr . David Alan Gilbert <[email protected]>; qemu-devel <[email protected]>; Li Zhijian <[email protected]> Cc: Jason Wang <[email protected]>; Lukas Straub <[email protected]> Subject: About migration/colo issue Hi Hailiang/Dave. I found a urgent problem in current upstream code, COLO will stuck on secondary checkpoint and later. The guest will stuck by this issue. I have bisect upstream code, this issue caused by Hailiang's optimize patch: >From 0393031a16735835a441b6d6e0495a1bd14adb90 Mon Sep 17 00:00:00 2001 From: zhanghailiang <[email protected]<mailto:[email protected]>> Date: Mon, 24 Feb 2020 14:54:10 +0800 Subject: [PATCH] COLO: Optimize memory back-up process This patch will reduce the downtime of VM for the initial process, Previously, we copied all these memory in preparing stage of COLO while we need to stop VM, which is a time-consuming process. Here we optimize it by a trick, back-up every page while in migration process while COLO is enabled, though it affects the speed of the migration, but it obviously reduce the downtime of back-up all SVM'S memory in COLO preparing stage. Signed-off-by: zhanghailiang <[email protected]<mailto:[email protected]>> Message-Id: <[email protected]<mailto:[email protected]>> Signed-off-by: Dr. David Alan Gilbert <[email protected]<mailto:[email protected]>> minor typo fixes Hailiang, do you have time to look into it? The detail log: Primary node: [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'checkpoint-ready' message {"timestamp": {"seconds": 1589511271, "microseconds": 917406}, "event": "RESUME"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'stop' => 'run' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'checkpoint-request' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'checkpoint-reply' message {"timestamp": {"seconds": 1589511291, "microseconds": 244096}, "event": "STOP"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'run' => 'stop' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'vmstate-send' message [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'vmstate-size' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'vmstate-received' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'vmstate-loaded' message {"timestamp": {"seconds": 1589511309, "microseconds": 31862}, "event": "RESUME"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'stop' => 'run' {"timestamp": {"seconds": 1589511311, "microseconds": 111617}, "event": "VNC_CONNECTED", "data": {"server": {"auth": "none", "family": "ipv4", "service": "5907", "host": "0.0.0.0", "websocket": false}, "client": {"family": "ipv4", "service": "51564", "host": "10.239.13.19", "websocket": false}}} {"timestamp": {"seconds": 1589511311, "microseconds": 116197}, "event": "VNC_INITIALIZED", "data": {"server": {"auth": "none", "family": "ipv4", "service": "5907", "host": "0.0.0.0", "websocket": false}, "client": {"family": "ipv4", "service": "51564", "host": "10.239.13.19", "websocket": false}}} [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'checkpoint-request' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'checkpoint-reply' message {"timestamp": {"seconds": 1589511311, "microseconds": 351439}, "event": "STOP"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'run' => 'stop' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'vmstate-send' message [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'vmstate-size' message Secondary node: {"timestamp": {"seconds": 1589510920, "microseconds": 778207}, "event": "RESUME"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'stop' => 'run' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'checkpoint-ready' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'checkpoint-request' message {"timestamp": {"seconds": 1589510940, "microseconds": 105712}, "event": "STOP"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'run' => 'stop' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'checkpoint-reply' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'vmstate-send' message [email protected]:colo_flush_ram_cache_begin<mailto:[email protected]:colo_flush_ram_cache_begin> dirty_pages 2456 [email protected]:colo_flush_ram_cache_end<mailto:[email protected]:colo_flush_ram_cache_end> [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'vmstate-size' message [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'vmstate-received' message qemu-system-x86_64: warning: TSC frequency mismatch between VM (2792980 kHz) and host (2925999 kHz), and TSC scaling unavailable {"timestamp": {"seconds": 1589510957, "microseconds": 754184}, "event": "RESUME"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'stop' => 'run' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'vmstate-loaded' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'checkpoint-request' message {"timestamp": {"seconds": 1589510960, "microseconds": 106148}, "event": "STOP"} [email protected]:colo_vm_state_change<mailto:[email protected]:colo_vm_state_change> Change 'run' => 'stop' [email protected]:colo_send_message<mailto:[email protected]:colo_send_message> Send 'checkpoint-reply' message [email protected]:colo_receive_message<mailto:[email protected]:colo_receive_message> Receive 'vmstate-send' message [email protected]:colo_flush_ram_cache_begin<mailto:[email protected]:colo_flush_ram_cache_begin> dirty_pages 25
