yuanbenson opened a new issue, #9268: URL: https://github.com/apache/pinot/issues/9268
Consistent data push protocol APIs are available via controller REST APIs such as `startReplaceSegments`, `endReplaceSegments`, and `revertReplaceSegments`. However, previously, ingestion jobs are not wired to use this feature. Introduce a new boolean `consistentDataPush` in `TableConfig->ingestionConfig->batchIngestionConfig` that when enabled, supports batch ingestion in REFRESH mode to run in consistent data push mode. _Consistent push goal: supports atomic switching (on broker level) between data snapshots and eliminate the time period where the query is getting computed from inconsistent data mixed from existing and new data. Moreover, we aim to provide an easy way to rollback to the previous data in case of the bad data push._ See https://github.com/apache/pinot/issues/7813 for more details. Some tasks breakdown associated with this issue: 1. Improve test coverage for pinot-batch-ingestion-standalone jobs to cover `SegmentMetadataPushJobRunner`, `SegmentTarPushJobRunner` and `SegmentUriPushJobRunner`. 2. Refactor the common logics out of all pushJobRunner(s) into a new abstract class `BaseSegmentPushJobRunner`. 3. Main change on enabling consistent data push on ingestion jobs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
