lnbest0707-uber commented on code in PR #15234:
URL: https://github.com/apache/pinot/pull/15234#discussion_r1999700724
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1093,6 +1093,35 @@ public void segmentStoppedConsuming(LLCSegmentName
llcSegmentName, String instan
}
}
+ /**
+ * An instance is reporting that it cannot build segment due to
non-recoverable error, usually due to size too large.
+ * Reduce the segment "segment.flush.threshold.size" to its half value.
+ */
+ public void reduceSegmentSizeAndReset(LLCSegmentName llcSegmentName, int
prevNumRows) {
+ Preconditions.checkState(!_isStopping, "Segment manager is stopping");
+ // reduce the segment size to its half
+ String segmentName = llcSegmentName.getSegmentName();
+ String realtimeTableName =
TableNameBuilder.REALTIME.tableNameWithType(llcSegmentName.getTableName());
+
+ Stat stat = new Stat();
+ SegmentZKMetadata prevSegmentZKMetadata =
getSegmentZKMetadata(realtimeTableName, segmentName, stat);
+ Preconditions.checkState(prevSegmentZKMetadata.getStatus() ==
Status.IN_PROGRESS,
+ "Segment status for segment: %s should be IN_PROGRESS, found: %s",
segmentName,
+ prevSegmentZKMetadata.getStatus());
+
+ int prevTargetNumRows =
prevSegmentZKMetadata.getSizeThresholdToFlushSegment();
+ int newNumRows = Math.min(prevNumRows / 2, prevTargetNumRows / 2);
Review Comment:
Not always. The endCriteria check is between consuming each "batch" of
messages. So it is possible that after consuming a batch, the numRowsIndexed
goes slightly over threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
