mcvsubbu commented on issue #5559: URL: https://github.com/apache/incubator-pinot/issues/5559#issuecomment-649634794
@kishoreg Maybe you forget, :) but I know how hard it was to detect (let alone debug) the phenomena of two consuming segments for the same partition in the same replica. Yes, this is a corner case, but when an issue is reported, we need to debug it. If it was hard when we had all the machines and logs at hand and could look at zookpeer logs at will, imagine doing it remotely when some open source user reports a problem when we can't even GUESS that there are parallel consuming segments. We added plenty of comments and put the logic in one class so that we don't stumble on such things again. https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L665 https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L715 https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L754 We make a strong assumption here that there are only two threads in the entire cluster that create segments. By starting a new validation manager, we violate that assumption. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
