pri1712 opened a new pull request, #19011: URL: https://github.com/apache/pinot/pull/19011
This PR aims to introduce: [16689](https://github.com/apache/pinot/issues/16689) **Summary:** Introduces a new query option (`skipOutOfRetentionValues=true`) to automatically exclude out-of-retention records from the result set at query time. **Motivation:** Pinot currently drops segments only when the segment's maximum time value falls out of the retention window. When small segment merging or compaction is enabled, segments often span multiple days. This results in queries returning expired data because the segment is kept alive by its newer records. This PR allows users to dynamically filter out that expired data without altering segment boundaries. **Changes:** - Added SKIP_OUT_OF_RETENTION_VALUES to `QueryOptionKey`. - Modified `BaseBrokerRequestHandler` to intercept the query prior to execution. - If the option is enabled, the broker fetches the table's retention config and DateTimeFormatSpec, computes the exact cutoff timestamp in the column's native storage format, and dynamically injects a >= condition into the Calcite AST (SqlNode) WHERE clause. - For now it **does not work for multi table queries (JOINS)**, this is because each table may have different semantics, leading to extremely complex queries. If needed this can be picked up as a separate issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
