npawar commented on issue #6302: URL: https://github.com/apache/pinot/issues/6302#issuecomment-892878224
> > The segment will complete when the time constraint is reached. Since there is no more new data, the last completed segment will be moved, and the next consuming segment will be empty. Am I right? Not quite. The movement job waits for the complete window. So if you've set window as 1h, it will wait until the completed segments have crossed the entire 1 hour. If there's a consuming segment immediately after, it will wait. There's no way to know if the next consuming segment is going to get events for the incomplete window or not. This was designed with the assumption that in production you'll have a continuous stream of events and none of these manual and edge cases. My preference would be to design a pause/resume, instead of relying on the realtimeToOfflineJob for this. Or else, we'll be adding too many if-else to handle all these edge cases in the minion job. Plus, they would have to wait for the time threshold, or adjust it manually just for making this work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
