Hi Hailiang/Dave. I found a urgent problem in current upstream code, COLO will stuck on secondary checkpoint and later. The guest will stuck by this issue. I have bisect upstream code, this issue caused by Hailiang's optimize patch:
>From 0393031a16735835a441b6d6e0495a1bd14adb90 Mon Sep 17 00:00:00 2001 From: zhanghailiang <[email protected]> Date: Mon, 24 Feb 2020 14:54:10 +0800 Subject: [PATCH] COLO: Optimize memory back-up process This patch will reduce the downtime of VM for the initial process, Previously, we copied all these memory in preparing stage of COLO while we need to stop VM, which is a time-consuming process. Here we optimize it by a trick, back-up every page while in migration process while COLO is enabled, though it affects the speed of the migration, but it obviously reduce the downtime of back-up all SVM'S memory in COLO preparing stage. Signed-off-by: zhanghailiang <[email protected]> Message-Id: <[email protected]> Signed-off-by: Dr. David Alan Gilbert <[email protected]> minor typo fixes Hailiang, do you have time to look into it? The detail log: Primary node: [email protected]:colo_receive_message Receive 'checkpoint-ready' message {"timestamp": {"seconds": 1589511271, "microseconds": 917406}, "event": "RESUME"} [email protected]:colo_vm_state_change Change 'stop' => 'run' [email protected]:colo_send_message Send 'checkpoint-request' message [email protected]:colo_receive_message Receive 'checkpoint-reply' message {"timestamp": {"seconds": 1589511291, "microseconds": 244096}, "event": "STOP"} [email protected]:colo_vm_state_change Change 'run' => 'stop' [email protected]:colo_send_message Send 'vmstate-send' message [email protected]:colo_send_message Send 'vmstate-size' message [email protected]:colo_receive_message Receive 'vmstate-received' message [email protected]:colo_receive_message Receive 'vmstate-loaded' message {"timestamp": {"seconds": 1589511309, "microseconds": 31862}, "event": "RESUME"} [email protected]:colo_vm_state_change Change 'stop' => 'run' {"timestamp": {"seconds": 1589511311, "microseconds": 111617}, "event": "VNC_CONNECTED", "data": {"server": {"auth": "none", "family": "ipv4", "service": "5907", "host": "0.0.0.0", "websocket": false}, "client": {"family": "ipv4", "service": "51564", "host": "10.239.13.19", "websocket": false}}} {"timestamp": {"seconds": 1589511311, "microseconds": 116197}, "event": "VNC_INITIALIZED", "data": {"server": {"auth": "none", "family": "ipv4", "service": "5907", "host": "0.0.0.0", "websocket": false}, "client": {"family": "ipv4", "service": "51564", "host": "10.239.13.19", "websocket": false}}} [email protected]:colo_send_message Send 'checkpoint-request' message [email protected]:colo_receive_message Receive 'checkpoint-reply' message {"timestamp": {"seconds": 1589511311, "microseconds": 351439}, "event": "STOP"} [email protected]:colo_vm_state_change Change 'run' => 'stop' [email protected]:colo_send_message Send 'vmstate-send' message [email protected]:colo_send_message Send 'vmstate-size' message Secondary node: {"timestamp": {"seconds": 1589510920, "microseconds": 778207}, "event": "RESUME"} [email protected]:colo_vm_state_change Change 'stop' => 'run' [email protected]:colo_send_message Send 'checkpoint-ready' message [email protected]:colo_receive_message Receive 'checkpoint-request' message {"timestamp": {"seconds": 1589510940, "microseconds": 105712}, "event": "STOP"} [email protected]:colo_vm_state_change Change 'run' => 'stop' [email protected]:colo_send_message Send 'checkpoint-reply' message [email protected]:colo_receive_message Receive 'vmstate-send' message [email protected]:colo_flush_ram_cache_begin dirty_pages 2456 [email protected]:colo_flush_ram_cache_end [email protected]:colo_receive_message Receive 'vmstate-size' message [email protected]:colo_send_message Send 'vmstate-received' message qemu-system-x86_64: warning: TSC frequency mismatch between VM (2792980 kHz) and host (2925999 kHz), and TSC scaling unavailable {"timestamp": {"seconds": 1589510957, "microseconds": 754184}, "event": "RESUME"} [email protected]:colo_vm_state_change Change 'stop' => 'run' [email protected]:colo_send_message Send 'vmstate-loaded' message [email protected]:colo_receive_message Receive 'checkpoint-request' message {"timestamp": {"seconds": 1589510960, "microseconds": 106148}, "event": "STOP"} [email protected]:colo_vm_state_change Change 'run' => 'stop' [email protected]:colo_send_message Send 'checkpoint-reply' message [email protected]:colo_receive_message Receive 'vmstate-send' message [email protected]:colo_flush_ram_cache_begin dirty_pages 25
