deemoliu commented on PR #10047: URL: https://github.com/apache/pinot/pull/10047#issuecomment-1442335171
> > Hi @Jackie-Jiang I checked this part, the function will be called during LLRealtimeSegmentDataManager.commitSegment(String, boolean). Since it's triggered when commit Segment, the new segment on the same partition should not started consuming. Can you help me understand why the new consuming segment might already be created? > > Do you foresee any risk on the following operations during segment commit? > > removeExpiredPrimaryKeys(segment, expiredTimestamp); persistSnapshotForStableSegment(expiredTimestamp); > > Only the committing server will call `commitSegment()`, but we need all servers to clean up the expired keys after finishing the current consuming segment and before starting the new consuming segment. Hi @Jackie-Jiang, thanks for the commit. Can you help understand this part? In my understanding, each partitionManager has its own PrimaryKeyToRecordLocation map. When committing a new segment in one partition, do we need other partitionManagers to clean up the expired keys? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
