somandal opened a new issue, #15683: URL: https://github.com/apache/pinot/issues/15683
Today we allow running multiple TableRebalancer jobs for the same table at the same time. TableRebalance is idempotent, but ideally we should not allow this behavior. There are some periodic tasks such as the `SegmentRelocator` task that creates rebalance jobs at a fixed frequency. It becomes hard to track progress and make sense of the expected behavior when multiple rebalance jobs run. We should instead: - Create a TableRebalance manager class to oversee the creation and management of rebalance jobs on tables - Have all calls to create rebalance jobs go through the above manager class, include periodic tasks like SegmentRelocator - Track ongoing rebalances and reject rebalance jobs for tables already ongoing rebalance - We could potentially also have a thread pool mechanism to limit the number of jobs spawned at a time - Enforce that all jobs enable progress stats tracking so that their status can be stored in ZK Some additional considerations - We will need to ensure that we can handle scenarios where a controller dies that had ongoing rebalance jobs. These will have a status in ZK, but when a new controller (or the old controller on start-up) identifies this scenario, it should gracefully handle it (i.e. in this scenario it should start a new job even though on in ZK exists with IN_PROGRESS status) - Today we don't clean up ZK job status ZNodes. Thus they can grow indefinitely and we may start hitting the ZNode size limitations. We should periodically clean up older job statuses cc @Jackie-Jiang @npawar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
