mcvsubbu commented on PR #9244: URL: https://github.com/apache/pinot/pull/9244#issuecomment-1222550398
> The issue with offset based status checker is that we don't know how many events from the stream we're behind. That's because the StreamPartitionMsgOffset interface only tells us if we're caught up or we're behind, but it it doesn't tell us by how many events we are behind. That was the reason we couldn't define a threshold for catchup when offset based status checker was added. > > Anyways, the current PR has duplicate code between OffsetBasedStatusChecker and FreshnessBasedStatusChecker. I think by making a small change to the OffsetBasedStatusChecker, we can achieve the goal of this PR. We can pass in a config for freshness threshold to OffsetBasedStatusChecker and where we get the stream latest offset at startup, if the config is provided, we can get stream latest offset at the current moment and check the freshness then. Good point. Can we enhance the stream interface to decide the lag and then emit a metric ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
