Jackie-Jiang opened a new pull request #7092:
URL: https://github.com/apache/incubator-pinot/pull/7092
## Description
Enhance `SegmentProcessorFramework` with the following improvements:
- Refactor reducer to:
- Support CONCAT/ROLLUP/DEDUP merge type (using off-heap memory)
- Support null values
- Generate generic row file as intermediate file
- Minimize the intermediate file generated (0 for CONCAT, 1 for
ROLLUP/DEDUP)
- Add `GenericRowFileRecordReader` which can directly read generic row file
for a range of records to avoid creating extra intermediate files when
generating segments
- Simplify the `MergeRollupTaskExecutor` to directly use the
`SegmentProcessorFramework`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
