jianjindream commented on issue #20339: URL: https://github.com/apache/druid/issues/20339#issuecomment-5949326353
I investigated this on `master` at `3e9365196f9893ff9b1cefcfbcb57d49e4080b4a` and reproduced the stale-holder initialization wait. Releasing the syncer's initialization latch on stop, while tracking successful initialization separately, addresses the wait. However, skipping the removed holder during startup exposes a task-recovery ordering that needs clarification: 1. Startup is waiting on holder A before its first sync. 2. Discovery removes A and adds replacement B for the same host. 3. B's first sync remains pending, but startup completes after skipping A. 4. A task marked RUNNING in storage has not yet been reported by B. TaskQueue recovery submits it to another initialized worker C. 5. When B finally reports the original running task, the runner retains C's ownership and sends a shutdown request to B. The replacement is correctly excluded from eligible workers until initialized; that does not prevent reassignment to C. In a control case where B completes fullSync before task recovery, the original ownership is restored without assignment or shutdown requests. This is a controlled local test using the production discovery, syncer, TaskQueue recovery and assignment paths, with mocked HTTP transport and storage. TaskQueue management is invoked directly; no live cluster was tested. What startup synchronization contract should this fix preserve? In particular, should startup follow the current holder for an initially discovered host and wait for its first fullSync? How should that handle a remove/readd gap or repeated replacement within a bounded wait? I have paused the production changes rather than introduce an unconfirmed recovery policy. I can share the focused reproduction and candidate diff if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
