zhtaoxiang commented on code in PR #9890:
URL: https://github.com/apache/pinot/pull/9890#discussion_r1040029118
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##########
@@ -61,13 +63,26 @@
/**
* A {@link PinotTaskGenerator} implementation for generating tasks of type
{@link MergeRollupTask}
*
- * TODO: Add the support for realtime table
+ * Assumptions:
+ * - When the MergeRollupTask starts the first time, records older than the
bufferTimeMs have already been ingested.
Review Comment:
Good point!
If I understand correctly, we are using `maxEndTimeMs - bufferTimeMs`. See
the method `getValidBucketEndTimeMsForSegment` for details.
I will update the comment.
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##########
@@ -428,20 +446,43 @@ public List<PinotTaskConfig>
generateTasks(List<TableConfig> tableConfigs) {
return pinotTaskConfigs;
}
+ @VisibleForTesting
+ static List<SegmentZKMetadata> filterSegmentsBasedOnStatus(TableType
tableType, List<SegmentZKMetadata> allSegments) {
+ if (tableType == TableType.REALTIME) {
+ // For realtime table, filter out
+ // 1. in-progress segments
+ // 2. sealed segments with start time later than the earliest start time
of all in progress segments
Review Comment:
maybe `filter out` is confusing here? Let me change it to `don't process`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
