mcvsubbu commented on issue #6679: URL: https://github.com/apache/incubator-pinot/issues/6679#issuecomment-877523700
@Jackie-Jiang I don't understand the proposal clearly. Table config already has ways to specify earliest/latest/1d/3d etc. Let us see what are the cases we need to handle: - User has deleted the consuming segment (or segments) by mistake and wants to restore whatever was there - User has deleted the consuming segments (or segments) because some partition-groups had a bad offset. User has the right offset to re-start consumption from. - User has deleted all consuming segments because they changed something in the underlying stream (e.g. changed the kafka topic, or something like that) In all of these cases, we need the following arguments - A partition group ID (or, some way to say "all partition groups") - A directive to pick the best offset (i.e. pick the next offset from prev segment if that offset is still in the stream. Otherwise, pick the earliest offset in the stream, while indicating loss of data) - A directive to say that the offsets in the previous segments need to be forgotten, and we just need to pick whatever offset is available now in the partition-group(s), even if it compares lower than the previous segment's offset. In this case, the table config will indicate whether the offset to be chosen is "earliest" or "latest" or some other offset directive. This is same as table creation mode. Are you in agreement with this, or am I missing some things? Once we have identified the requirements, we can design an API that meets all of that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
