Brian Let me answer you in bullet points: 1. I have tried the expression with both, ie. > 1 and also > 100. Both don't fire. 2. I am getting other Alerts through Alertmanager for example, UP/down of instance. So its not the Alertmanager.
Expression shows non-empty results in PromQL Query interface but still it doesn't fire. J On Thu, Aug 22, 2024 at 10:20 AM 'Brian Candler' via Prometheus Users < [email protected]> wrote: > Your test example in PromQL browser has: > confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} > 1 > and the values were 2 or 3; but the alerting expression has > confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} > 100 > So clearly it's not going to trigger under that condition, when the lags > are less than 100. > > If that's not the probelm, then you need to determine: is the rule not > firing? Or is Alertmanager not sending an alert? > > To do this, check in the Prometheus web interface under the Alerts tab. Is > there a firing alert there? If yes, then you focus your investigation on > the alertmanager side (e.g. check alertmanager logs). If no, then drill > further into the expression, although if the same expression shows a > non-empty result in the PromQL query interface, then it certainly should be > able to fire an alert. > > On Wednesday 21 August 2024 at 21:24:58 UTC+1 Jay wrote: > >> Here is the text: >> >> groups: >> - name: confluent-rules >> rules: >> >> - alert: Dev-NotEqualtoBoolZero >> expr: >> confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} >> > 100 >> labels: >> severity: critical >> annotations: >> description: "The consumer lags for Dev client`" >> On Wed, Aug 21, 2024 at 1:51 PM Daz Wilkin <[email protected]> wrote: >> >>> Please include the rule. >>> >>> You've shown that the query returns results which is necessary but >>> insufficient. >>> >>> On Wednesday, August 21, 2024 at 8:19:34 AM UTC-7 Jay P wrote: >>> >>>> I am not new to Prometheus however, I wrote the following rules which >>>> never fires. (Alertmanger and all other settings are fine since I get >>>> Alarms for other rules except this one) >>>> >>>> Attached here it he screenshot and i am copy pasting here as well. >>>> >>>> confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} > 1 >>>> >>>> Results: >>>> confluent_kafka_server_consumer_lag_offsets{consumer_group_id="XXX", >>>> instance="api.telemetry.confluent.cloud:443", job="confluent-cloud", >>>> kafka_id="XXX", topic="XXX"} >>>> 2 >>>> confluent_kafka_server_consumer_lag_offsets{consumer_group_id="XXX", >>>> instance="api.telemetry.confluent.cloud:443", job="confluent-cloud", >>>> kafka_id="XXX", topic="XXX"} >>>> 3 >>>> >>>> Any help is greatly appreciated. Thank you >>>> >>> -- >>> >> You received this message because you are subscribed to a topic in the >>> Google Groups "Prometheus Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/prometheus-users/pBEqCDIUFug/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/e8007886-5a85-4fdb-938e-36373409498cn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/e8007886-5a85-4fdb-938e-36373409498cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "Prometheus Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/prometheus-users/pBEqCDIUFug/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/8be525b8-e216-4cc4-8f05-c126bf42fc35n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/8be525b8-e216-4cc4-8f05-c126bf42fc35n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAN3oSLTuLZFHwey3EmkQOQWoKnbgKZn%3D9_cpNAj5C-yQNhWNkQ%40mail.gmail.com.
