Jackie-Jiang commented on code in PR #16034:
URL: https://github.com/apache/pinot/pull/16034#discussion_r2132932089
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -830,12 +834,61 @@ public void replaceConsumingSegment(String segmentName)
throws Exception {
_logger.info("Replacing CONSUMING segment: {} with the one sealed
locally", segmentName);
File indexDir = new File(_indexDir, segmentName);
+
+ // Fix creation time consistency before loading the locally built segment
(for upsert tables)
+ if (isUpsertEnabled()) {
+ fixCreationTimeConsistencyForLocalSegment(segmentName, indexDir);
+ }
+
// Get a new index loading config with latest table config and schema to
load the segment
IndexLoadingConfig indexLoadingConfig = fetchIndexLoadingConfig();
addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig,
_segmentOperationsThrottler));
_logger.info("Replaced CONSUMING segment: {}", segmentName);
}
+ /**
+ * Fixes creation time consistency for locally built segments in upsert
tables.
+ * This ensures that slower replicas have consistent creation times for
proper upsert comparison.
+ */
+ private void fixCreationTimeConsistencyForLocalSegment(String segmentName,
File indexDir) {
+ try {
+ SegmentZKMetadata zkMetadata = fetchZKMetadata(segmentName);
+ long segmentCreationTimeFromZK = zkMetadata.getCreationTime();
Review Comment:
ZK creation time is not the real segment creation time, and I'd suggest not
modifying the local segment creation time since it is useful info. One solution
is to add a separate field ZK creation time into `SegmentMetadataImpl` and you
can set it when loading the segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
