yashmayya opened a new pull request, #16008: URL: https://github.com/apache/pinot/pull/16008
- Builds on top of the refactors in https://github.com/apache/pinot/pull/15990. - Initially, the main purpose of the rebalance stats in ZK was to support automatic retries of failed and stuck rebalances through the `RebalanceChecker` periodic controller job. Since `SegmentRelocator` is itself a periodic controller job, it was deemed unnecessary to track these rebalances in ZK. - However, with https://github.com/apache/pinot/issues/15683 / https://github.com/apache/pinot/pull/15990, we're also going to be using the ZK stats to prevent multiple concurrent rebalances for the same table and this should extend to rebalances triggered by the `SegmentRelocator` as well. - This patch enables ZK-based progress tracking for `SegmentRelocator` rebalances, and also makes some changes to the rebalance context to ensure that `RebalanceChecker` ignores rebalance jobs initiated by the `SegmentRelocator`. - Since `SegmentRelocator` is a periodic controller job (1 hour frequency by default), this could lead to a number of rebalance jobs being stored in ZK, so we're planning to also improve the cleanup logic for controller job ZK metadata (https://github.com/apache/pinot/issues/16005). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
