Public bug reported: [Impact]
chronyc waitsync hangs with the error message '506 Cannot talk to daemon' if chronyd happens to be unavailable while chronyc is polling it. By default, chronyc communicates with chronyd over a Unix socket, IPv4 or IPv6 loopback addresses. This can be overridden with the -h option for chronyc, and bindcmdaddress for chronyd. chronyc attempts to connect to one of these addresses on startup. It also attempts another address when a request to chronyd receives no response. However, after all addresses have been attempted, chronyc never returns to the beginning of the address list. This causes waitsync to keep reporting '506 Cannot talk to daemon' even after chronyd becomes available again. The execution path is detailed in [Other Info]. This bug was identified by Jan Mikkelsen in the following thread on the chrony-users mailing list: https://www.mail-archive.com/chrony- [email protected]/msg02780.html The fix was committed in the upstream and released in chrony 4.3: https://gitlab.com/chrony/chrony/-/commit/759580aa6f32fcc591ff357c12b54c22a8e03b91 I confirmed that this commit fixes the bug. I also ran git bisect with an automated script and confirmed that it is the first commit to fix the bug between chrony 4.2 and 4.3. The bug is present in Bionic (chrony 3.2), Focal (chrony 3.5), and Jammy (chrony 4.2). Noble (chrony 4.5) and Resolute (chrony 4.8) are not affected. I also confirmed that the relevant diff from 759580a is absent in Jammy and present in Noble. Apart from the upstream mailing list, I found no matching report in the Debian BTS or Launchpad. [Test Plan] I have reproduced the bug on Bionic, Focal and Jammy with the following steps: Launch and configure a Jammy VM: multipass launch 22.04 --name chrony-jammy multipass shell chrony-jammy sudo apt update && sudo apt install -y chrony strace chronyc --version # 4.2 on Jammy / 4.5 on Noble Stop chronyd temporarily: sudo systemctl reset-failed chrony sudo systemctl stop chrony sudo systemctl status chrony # Stopped Start waitsync with strace. The max-tries argument is set to zero to reproduce the indefinite hang. # man chronyc | grep waitsync # waitsync [max-tries [max-correction [max-skew [interval]]]] sudo strace -e trace=connect,socket,write chronyc waitsync 0 0.1 0.0 0.1 The unpatched chronyc attempts the Unix socket, IPv4 and IPv6 loopback addresses. After these attempts, it repeatedly prints '506 Cannot talk to daemon' at every interval. Now open another terminal and enter the same VM: multipass shell chrony-jammy Start chronyd and wait for it to become synchronized: sudo systemctl start chrony sudo systemctl status chrony sleep 10 && chronyc tracking # Confirm leap status is back to normal Return to the first terminal, and confirm that waitsync continues printing '506 Cannot talk to daemon' with no new connect() attempts, even though chronyd is back online again. To fix this, I patched chrony 4.2 with 759580a, installed it in another Jammy VM, and ran the same reproduction steps as above. This time waitsync exits with status zero as soon as chronyd starts and becomes synchronised. I also ran autopkgtest against the patched Jammy package and verified that all tests passed. [Where problems could occur] I believe backporting 759580a has minimal impact. The commit 759580a adds the following lines in open_io(): /* Start from the first address if called again */ address_index = 0; The change only resets address_index after all connection attempts fail. The impact is limited to chronyc, and no changes are made to chronyd. The patch applies cleanly to Jammy without merge conflicts. The same change has been present in Noble and later releases. The patched Jammy package builds successfully, fixes the bug, and passes autopkgtest. [Other Info] The default server address list used by chronyc is: 1. /run/chrony/chronyd.sock 2. 127.0.0.1 3. ::1 open_io() stores the current position in this list in a static variable called address_index. When chronyd is stopped, the initial call to open_io() in main() cannot connect to the Unix socket because /run/chrony/chronyd.sock is removed. So open_io() then advances address_index to 127.0.0.1. Connecting to the UDP socket for 127.0.0.1 succeeds without chronyd listening, which allows the initial open_io() call in main() to succeed. submit_request() however receives no response from chronyd. request_reply() calls open_io() again to attempt another address as a fallback. Because the socket for 127.0.0.1 is already open, open_io() calls close_io(), which closes the socket, sets sock_fd to -1, and advances address_index to ::1 Connecting to the socket for ::1 also succeeds, but the next submit_request() again receives no response. open_io() is called again as a fallback, which closes the socket, sets sock_fd to -1, and advances address_index beyond the end of the address list. At this point: - sock_fd == -1 - address_index == 3 Every call to submit_request() now fails because sock_fd == -1, which triggers a call to open_io(). But every call to open_io() also fails without any connection attempt, because the address list is exhausted. The outer polling loop in process_cmd_waitsync() therefore continues printing '506 Cannot talk to daemon'. Since the reproducer sets max- tries to zero, it hangs forever until chronyc is killed externally. On a different note, if the Unix socket is specified as the only address as below: sudo chronyc -h /run/chrony/chronyd.sock waitsync 0 0.1 0.0 0.1 The command does not hang and returns immediately with the error message 'Could not open connection to daemon'. The initial open_io() in main() with the default address list only succeeds because connecting to the UDP sockets succeeds, not the Unix socket. ** Affects: chrony (Ubuntu) Importance: Medium Assignee: Taichi Maeda (taichimaeda1) Status: Fix Released ** Affects: chrony (Ubuntu Jammy) Importance: Medium Assignee: Taichi Maeda (taichimaeda1) Status: In Progress ** Changed in: chrony (Ubuntu) Assignee: (unassigned) => Taichi Maeda (taichimaeda1) ** Changed in: chrony (Ubuntu) Assignee: Taichi Maeda (taichimaeda1) => (unassigned) ** Changed in: chrony (Ubuntu) Assignee: (unassigned) => Taichi Maeda (taichimaeda1) ** Also affects: chrony (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: chrony (Ubuntu Jammy) Status: New => In Progress ** Changed in: chrony (Ubuntu Jammy) Importance: Undecided => Medium ** Changed in: chrony (Ubuntu Jammy) Assignee: (unassigned) => Taichi Maeda (taichimaeda1) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2165114 Title: chronyc hangs when chronyd goes down temporarily To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2165114/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
