deemoliu commented on PR #10047: URL: https://github.com/apache/pinot/pull/10047#issuecomment-1449056733
> > Hi @Jackie-Jiang, thanks for the commit. Can you help understand this part? In my understanding, each partitionManager has its own PrimaryKeyToRecordLocation map. When committing a new segment in one partition, do we need other partitionManagers to clean up the expired keys? > > No, each partition can be handled independently. Discussed with Jackie, and found the following issue. There are multiple servers consuming the same segment, only one of them will commit the segment, and the other servers will download the committed segment. With the current approach, the servers not committing the segment won't clean up the expired keys, which will cause inconsistency. Basically we want to make sure all servers clean up the expired keys after finishing the current consuming segment but before starting the next consuming segment [12:17](https://apache-pinot.slack.com/archives/D01CCL54XCM/p1677183435873319) We want to block the consumption until the keys are cleaned up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
