itschrispeck opened a new issue, #11276: URL: https://github.com/apache/pinot/issues/11276
Storage quota is currently available for offline tables, done during segment upload. For realtime tables we'd like to enable tail deletion of old segments once a table uses more than a set quota. For our use case this is preferable to setting ingestion rate limits as fresh data has more value than historic data. This helps when multiple tables share the same tenant and ingestion rate can be highly unpredictable. We'd prefer to add this as a non-blocking/asynchronous process. Strict adherence to the quota is not necessarily required and a segment commit should not be blocked if the new segment would go over the configured quota. Instead, we'd prefer a lazy tail deletion of old segments when the quota is breached in order to not impact segment commit duration/p99 freshness. I did not find similar issues open and would be open to adding this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
