FrankChen021 commented on PR #20299: URL: https://github.com/apache/druid/pull/20299#issuecomment-6170077587
The stall comes from the work done *under* `statusLock`, not the lock itself. Most of that work is `WorkerHolder.toImmutable()` → `ImmutableWorkerInfo.fromWorkerAnnouncements()`, which walks every task announcement on every worker — repeated for every pending task. Instead of caching a snapshot in the pending loop, `WorkerHolder` could compute its `ImmutableWorkerInfo` when announcements change (at the end of `fullSync` / `deltaSync`, which run outside `statusLock`), store it in an `AtomicReference`, and have `toImmutable()` just return it. The pending loop then stays unchanged and always sees current capacity, but snapshot building becomes O(workers) instead of O(workers × tasksPerWorker). This removes both the stale-snapshot concern and the extra refresh cost, and also speeds up other `toImmutable()` callers. `lastCompletedTaskTime` / `blacklistedUntil` changes would need to refresh the cached value too (or be attached at read time). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
