shuturmurgh opened a new issue, #18485: URL: https://github.com/apache/pinot/issues/18485
Controller currently has no guardrail over ingestion of out-of-retention segments. We've encountered this issue in production due to incorrectly configured ingestion jobs. Currently only Retention Manager handles their clean up. Adding this validation to Controller segment upload APIs can solve following issues. 1. Load on Servers for loading invalid segments 2. Retention Manager and Servers can run into race condition leaving out-of-retention segments temporarily in ERROR state (depicted in diagram below) <img width="768" height="512" alt="Image" src="https://github.com/user-attachments/assets/dd8016db-445a-421a-8bf0-bff0c37447f6"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
