ege-st commented on PR #12157: URL: https://github.com/apache/pinot/pull/12157#issuecomment-1861814766
> > We should be able to emit metric within `PartitionGroupConsumer.fetchMessages()` when the start offset is not available (e.g. the asked offset range is not fully returned). > > @Jackie-Jiang Are you suggesting to run this check in in `RealtimeSegmentDataManager::consumeLoop` effectively instead of when `PartitionGroupConsumer::start` is called? Will that cause too many checks? > > One doubt I have is whether the consumer may fall behind even after the consumer was started. In that case it is better to move the check to `consumeLoop` or `fetchMessages`. I don't think there will be a noticeable performance impact if the check is done in `fetchMessages` or `consumeLoop` and executed for each message batch. The performance cost should be relatively small compared to everything else that we do in the `consumeLoop`. But it would be awesome to do a performance test just to make sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
