lnbest0707-uber commented on PR #15234: URL: https://github.com/apache/pinot/pull/15234#issuecomment-2735020908
> This won't work for pauseless consumption because segment build happens after commit. cc @KKcorps @9aman > > Ideally we should try to follow the mechanism introduced in #14479 to early commit the segment when it cannot consume more rows I have a follow up PR for early commit in https://github.com/apache/pinot/pull/15120. However, the threshold is not easy to set. Concerns are listed in that PR. This retry mechanism (enabled by the table config) could be preferred in some cases. E.g. for large tables, false alarm on early commits is not always acceptable(creating too many segments). Regarding to pauseless consumption, this issue could also arise and be even more severe(no way to recover). In my opinion, we should introduce similar thing in the `/reingest` failure recovery loop introduced in https://github.com/apache/pinot/pull/14920. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
