somandal opened a new pull request, #14943: URL: https://github.com/apache/pinot/pull/14943
Rebuilding the StarTree indexes can be more resource intensive than rebuilding other indexes. This PR adds a specific StarTree index rebuild throttler as a safety net to limit the parallel StarTree index rebuilds that can occur in parallel. This change is related to https://github.com/apache/pinot/pull/14894 and utilizes the same ZK cluster config listener mechanism for config updates. This PR adds the following: - Adds a CLUSTER configuration to control the max startree preprocess parallelism allowed - Adds a `SegmentStarTreePreprocessThrottler` class to control how many startree index rebuild operations can happen in parallel. This is plugged into the `ImmutableSegmentLoader` and `SegmentPreProcesser` on code paths where startree index rebuild throttling is desirable I had considered adding the above to the existing `SegmentPreprocessThrottler` class itself, but wanted to avoid confusion regarding which semaphore should be used where in the code. Using a different class allows for better isolation of concerns. cc @Jackie-Jiang @klsince @npawar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
