Jackie-Jiang commented on issue #6679: URL: https://github.com/apache/incubator-pinot/issues/6679#issuecomment-877850622
@mcvsubbu In AUTO mode, we create the new consuming segment in the same way it is created during regular consumption: pick the last offset of the previous completed segment if exists, or follow the offset rule within the table config if not. We need other mode because in certain cases the default way won't skip the bad offset (second scenario). Let's say there is a bad offset two days ago, and the consumption stopped because of that. User might want to set the offset to: EARLIEST: Bad offset is already expired within the stream, and user wants to minimize the data loss LATEST: Start from the latest offset so that the bad offset is guaranteed to be skipped CUSTOMIZED: Bad offset might not be expired yet. Manually put an offset to skip the bad offset but not losing all the recent data This design should be able to handle all the requirements -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
