Jackie-Jiang commented on PR #10928: URL: https://github.com/apache/pinot/pull/10928#issuecomment-1599349616
> @Jackie-Jiang the snapshot operation in prod can take till around 30 seconds as per my observation. > Are we fine with spending that much extra time between each segment commit? > There will be a momentary consumption lag while snapshot is being taken although I think pinot should be able to easily catch up. > One optimisation we can do is only take snapshot of the segment which were **changed** from the last commit. @KKcorps Good point. It will definitely increase the delay before a new consuming segment starts consuming events, but might not be the whole 30 seconds because we only persist snapshot for one partition, and it doesn't involve the segment destroy time. I added a TODO to consider optimizing it by only persist the changed snapshot, but want to see how long it usually takes to persist all the snapshot as a baseline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
