chenboat commented on code in PR #10047:
URL: https://github.com/apache/pinot/pull/10047#discussion_r1106520979
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java:
##########
@@ -181,6 +184,32 @@ protected void removeSegment(IndexSegment segment,
MutableRoaringBitmap validDoc
}
}
+ /**
+ * When TTL is enabled for upsert, this function is used to remove expired
keys from the primary key indexes.
+ *
+ * When committing consuming segment, we replaced the consuming segment with
an immutable segments.
Review Comment:
Why do we want to do the cleanup of out-of-ttl keys during segment commit
phase? This means the clean up is blocking and potentially slow down the
ingestion. An alternative is to do it lazily as a background scheduled job. If
I understand correctly, TTL for upsert table is an optimization, we do not need
it synchronously. Here is the design doc about the choices
https://docs.google.com/document/d/1AhkZPkf4St3hj96IXyNs6NOiItxm3LXw3NAgNbG3oRo/edit?pli=1#heading=h.5ol3ncuq1lx2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
