jadami10 commented on PR #9244: URL: https://github.com/apache/pinot/pull/9244#issuecomment-1223418588
> The issue with offset based status checker is that we don't know how many events from the stream we're behind. That's because the StreamPartitionMsgOffset interface only tells us if we're caught up or we're behind, but it it doesn't tell us by how many events we are behind. That was the reason we couldn't define a threshold for catchup when offset based status checker was added. Even if we had this, I'm not sure it would help. I feel most folks set their SLAs based on time freshness, not "number of events". So even if we knew we were 10,000 events behind, that might be fine if you have 1,000 events per second but not fine if you have 10 events per seconds. > Anyways, the current PR has duplicate code between OffsetBasedStatusChecker and FreshnessBasedStatusChecker. I think by making a small change to the OffsetBasedStatusChecker, we can achieve the goal of this PR. We can pass in a config for freshness threshold to OffsetBasedStatusChecker and where we get the stream latest offset at startup, if the config is provided, we can get stream latest offset at the current moment and check the freshness then. Fair feedback. I opted to use a parent class to keep the functionality separate rather than try to merge them. I feel like this is a better approach at the moment to keep things more easily testable and singularly responsible. > Good point. Can we enhance the stream interface to decide the lag and then emit a metric ? I'd really prefer not to make such a big change in this PR if it's not necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
