mcvsubbu commented on issue #10147: URL: https://github.com/apache/pinot/issues/10147#issuecomment-2471187891
> @mcvsubbu > > 1. In ZooKeeper, segment metadata will use **COMMITTING** as the status for segments being built, distinguishing them from **DONE** segments that are already in deep store. Segment ZK metadata will act as the source of truth. > > 2. @Jackie-Jiang supports the new approach of marking the IdealState as ONLINE. > 2.1. As before, the external view will help us know the current state of the system and the segment will still be CONSUMING in EV Consider the scenario when there are two consuming segments on the same stream partition, and a pinot server is restarted. It will get state transitions for two segments (in any order). I believe we have protection against starting two consumers on the same partition, so only one of them can start to consume. Even if that protection is not there, the server may not have capacity to handle two consumers for one partition (and this may happen for several partitions on the consumer, since partitions generally complete around the same time). Worse, if the server that is restarted is the one that is committing (server-2 in your diagram), I am not sure what the recovery path is. Can you draw ladder diagrams for these scenarios so that we can understand the proposal better? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
