Jackie-Jiang opened a new pull request #7158: URL: https://github.com/apache/incubator-pinot/pull/7158
## Description Currently within `SegmentProcessorFramework`, time handling is modeled as separate general filter/transform/partition function within the processor config, which is quite hard to configure and much less efficient. Also, record filter/transform should be configured in the table config and handled within the `RecordTransformer` instead of in the framework, or it can cause inconsistency. This PR extracts the time handling logic for SegmentProcessorFramework and support: - Time filtering: filter time within a time range - Time rounding: Round time according to the round time bucket - Time partitioning: Partition time according to the partition time bucket Currently only `EPOCH` mode is supported for time handling. Will add `DATE_TIME` mode in the future to support round/partition time on calendar date boundary with timezone support. ## Release Notes In `RealtimeToOfflineSegmentsTask` config: - `timeColumnTransformFunction` is removed (backward-incompatible, but rollup is not supported anyway) - Deprecate `collectorType` and replace it with `mergeType` - Add `roundBucketTimePeriod` and `partitionBucketTimePeriod` to config the time bucket for round and partition -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
