deemoliu commented on code in PR #10915:
URL: https://github.com/apache/pinot/pull/10915#discussion_r1237428186
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java:
##########
@@ -181,6 +210,29 @@ protected void removeSegment(IndexSegment segment,
MutableRoaringBitmap validDoc
}
}
+ /**
+ * When TTL is enabled for upsert, this function is used to remove expired
keys from the primary key indexes.
+ * This function will be called before new consuming segment start to
consume.
+ *
+ * @param watermark The watermark is the time used to clean up the metadata
in the previous round
Review Comment:
The watermark is the time used to clean up the metadata in the previous
round.
it's millisecond unixtimestamp in long value
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java:
##########
@@ -243,6 +295,47 @@ protected GenericRow doUpdateRecord(GenericRow record,
RecordInfo recordInfo) {
}
}
+ protected long loadWatermark() {
+ File watermarkFile = getWatermarkFile();
+ if (watermarkFile.exists()) {
+ try {
+ byte[] bytes = FileUtils.readFileToByteArray(watermarkFile);
+ long watermark = ByteBuffer.wrap(bytes).getLong();
+ _logger.info("Loaded watermark {} from file for table: {}
partition_id: {}", watermark, _tableNameWithType,
+ _partitionId);
+ return watermark;
+ } catch (Exception e) {
+ _logger.warn("Caught exception while deleting watermark file: {},
skipping",
Review Comment:
gotcha, thanks for reviewing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
