npawar commented on issue #6679: URL: https://github.com/apache/incubator-pinot/issues/6679#issuecomment-876693015
+1 to this `automatic fix always pick the latest offset if there are completed segments, or earliest available offset if not`. One thing to watch out for here is in Kinesis, it is valid to have no CONSUMING segment. We shouldn't end up creating a CONSUMING segment for those, we might have to add endOfPartition into segment metadata. This issue and the suggestion was created before we got the Kinesis refactoring. I'm not sure about the second part of providing the ability to put an offset, which they will use when they want to skip some offsets. This means that we are declaring that if someone wants to skip some offsets, they have to delete the consuming segments. They may have to delete some completed segments too, if the records they want to skip are already in the completed seg). Do we really want users to be deleting consuming segments whenever they want to skip offsets? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
