chenboat commented on code in PR #10047:
URL: https://github.com/apache/pinot/pull/10047#discussion_r1197274563
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -526,6 +529,32 @@ private void handleUpsert(ImmutableSegment
immutableSegment) {
_tableNameWithType));
PartitionUpsertMetadataManager partitionUpsertMetadataManager =
_tableUpsertMetadataManager.getOrCreatePartitionManager(partitionId);
+ return partitionUpsertMetadataManager;
+ }
+
+ /**
+ * Handle Upsert TTL during segment state change from CONSUMING to ONLINE.
+ *
+ * When a new segments commit, we will remove primary key indexes from heap
that has timestamp < (eventTime - TTL).
+ * For all segments sealed before (eventTime-TTL), their validDocIds won't
get updated. We do an one-time persistance.
+ */
+ protected void handleUpsertTTL(String segmentName, SegmentZKMetadata
segmentZKMetadata,
+ UpsertTTLConfig upsertTTLConfig) {
+ PartitionUpsertMetadataManager partitionUpsertMetadataManager =
+ getPartitionUpsertMetadataManagerBySegment(segmentName);
+
+ assert segmentZKMetadata != null;
+ long endTimeInMillis = segmentZKMetadata.getEndTimeMs();
+ long expiredTimestamp = endTimeInMillis - upsertTTLConfig.getTtlInMs();
+
+ partitionUpsertMetadataManager.removeExpiredPrimaryKeys(expiredTimestamp);
+
partitionUpsertMetadataManager.persistSnapshotForStableSegment(expiredTimestamp);
Review Comment:
Should we use persistSnapshotForStableSegments because usually there are
more than 1 stable segments per partition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
