lnbest0707-uber commented on PR #15234: URL: https://github.com/apache/pinot/pull/15234#issuecomment-2737702374
> > Regarding to pauseless consumption, this issue could also arise and be even more severe(no way to recover). In my opinion, we should introduce similar thing in the /reingest failure recovery loop introduced in #14920. > > For pauseless consumption, it will be hard to reingest the segment with a different size threshold because start/end offset are already determined. We need to detect the issue earlier, i.e. before segment build True.. The current limitation is on the segment sequence number. Once the segment offset is decided, when we reingest it, it is hard to split it into 2 as the next continuous sequence number is already taken. Probably some tricks need to be introduced there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
