navina opened a new pull request #8280: URL: https://github.com/apache/pinot/pull/8280
A common user-request is to understand if there is any lag in the consuming segment of a realtime table. While lag calculation varies significantly across streaming sources, this PR proposes an approach to surface this information in a generic manner. The PR demonstrates offset-lag computation for Kafka. It can be extended to show more useful lag information : such as last consumed message timestamp (event time), ingestion rate etc. It can also be extended to other streaming systems that may support different lag calculation mechanisms. **Open questions for discussion:** 1. Does lag information require a new end-point or should we add to the existing `/consumingSegmentsInfo` ? 2. Currently, the `/consumingSegmentsInfo` only returns consumer lag information for active consuming segments. Would it be useful to fetch lag information even when the segment is not actively consuming (say, on a paused table?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
