jackjlli commented on a change in pull request #7827:
URL: https://github.com/apache/pinot/pull/7827#discussion_r756603260
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -252,15 +252,15 @@ public String getTaskType() {
long bucketEndMs = bucketStartMs + bucketMs;
// Create delay metrics even if there's no task scheduled, this helps
the case that the controller is restarted
// but the metrics are not available until the controller schedules a
valid task
- long maxEndTimeMs = Long.MIN_VALUE;
+ long maxBucketEndTimeMs = Long.MIN_VALUE;
for (SegmentZKMetadata preSelectedSegment : preSelectedSegments) {
- long currentEndTimeMs = preSelectedSegment.getEndTimeMs();
- // Compute maxEndTimeMs among segments that are valid for merge
- if (currentEndTimeMs < System.currentTimeMillis() - bufferMs) {
- maxEndTimeMs = Math.max(maxEndTimeMs, currentEndTimeMs);
+ // Compute maxBucketEndTimeMs among segments that are valid for merge
+ long currentMaxBucketEndTimeMs =
getMaxValidBucketEndTimeMsForSegment(preSelectedSegment, bucketMs, bufferMs);
+ if (currentMaxBucketEndTimeMs != Long.MIN_VALUE) {
Review comment:
This if block may not be needed.
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -439,6 +439,18 @@ private boolean validate(TableConfig tableConfig, String
taskType) {
return true;
}
+ /**
+ * Get the max valid merge bucket end time for the segment, LONG.MIN_VALUE
if the segment is not ready to merge
+ */
+ private long getMaxValidBucketEndTimeMsForSegment(SegmentZKMetadata
segmentZKMetadata, long bucketMs, long bufferMs) {
+ if (segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs >
System.currentTimeMillis() - bufferMs) {
Review comment:
Could you add more description on how the max valid bucket end time is
calculated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
