mcvsubbu commented on issue #10147: URL: https://github.com/apache/pinot/issues/10147#issuecomment-1405890147
Changing the idealstate is a much more complex solution, and will have the number of rows to consume as an unresolved point. Today, we compute the new number of rows based on the old number of rows consumed, at the time the commit happens. Further, the computation is done on the controller (for a good reason, and it should be there) and not on the server. Our proposal for pauseless consumption is a much simpler one, no fundamental changes, and we can fine tune it as needed. Completely local to the server. We can keep the "daughter" segment inside the `LLRealtimeSegmentDataManager` object. Whenever queries come in, the other segment is also consulted to return the results. So, the view from rest of pinot will be that a segment in the CONSUMING state has not paused consumption during segment creation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
