tibrewalpratik17 opened a new issue, #13277: URL: https://github.com/apache/pinot/issues/13277
We are exploring ways to set alerts on our realtime table that if consumption stops for any partition in any replica, we can be notified. Consumption can stop due to various reasons: - Scenario 1: The upstream Kafka partition itself does not have data (this is a legit scenario for no messages consumed) - Scenario 2: Consuming segment in ERROR / OFFLINE state in any one replica - Scenario 3: Consuming segment in ERROR / OFFLINE state in ALL replicas The issue we are facing is differentiating between scenario 1 and scenario 3 intelligently. Till now, we have explored these 2 metrics: - realtimerowsconsumed - llcpartitionconsuming The above metrics specially `llcpartitionconsuming` works great in detecting scenario 2. But in other scenarios (1 and 3), metrics stops getting published rather than a clear differentiation that CONSUMING segment is ONLINE but not consuming (something like publishing 0 realtimerowsconsumed). We also checked `missingConsumingSegmentTotalCount` but it checks whether CONSUMING segment is present in IDEAL state but not the ExternalView of the table. Note: There is a not-so-straight-forward way of doing this is using one of the above `realtimerowsconsumed` / `llcpartitionconsuming` metric and having a dependent alert on upstream kafka topic message rate. But can we simplify this somehow from Pinot itself using some form of `callbackGauge` in consuming thread to ping that it is online but not consuming any messages? Looking forward to suggestions from community on how they are tracking stopped ingestion in their clusters. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
