I am encountering challenges with configuring Prometheus and Alertmanager for my application's alarm system. Below are the configurations I am currently using:
*prometheus.yml:* Scrape Interval: 1h *rules.yml:* groups: - name: recording-rule interval: 1h rules: - record: myRecord expr: expression….. (calculating ratio by dividing two metric > than value) - name: alerting-rule interval: 4h rules: - alert: myAlert expr: max_over_time(myRecord[4h]) labels: severity: warning annotations: summary: “summary” *alertmanager.yml:* group_by: ['alertname'] group_wait: 30s group_interval: 5m repeat_interval: 4h *Issues:* - *Inconsistent Alerting:* The similarity in scrape interval and recording rule evaluation interval (both set to 1 hour) leads to instances where Prometheus scrapes data before the recording rule evaluation. Consequently, during the recording rule evaluation, there may be no value in the metric, resulting in the recording rule failing to trigger an alert despite the condition being satisfied. - *Discrepancy in Firing Alerts:* The number of firing alerts in Prometheus varies significantly from the number of alerts received by Alertmanager, causing inconsistency and confusion in alert handling. - *Uncertainty in Alert Evaluation Timing:* The alerting rule seems to be evaluated inconsistently, sometimes triggering alerts shortly after service restart, while other times with delays beyond the expected 4-hour interval. *Request for Assistance:* I am seeking guidance on configuring Prometheus and Alertmanager to achieve the following: - Ensuring the alerting expression is evaluated every 4 hours, checking for the maximum of the recording metric over the specified interval. - The recording rule is evaluated every 1 hour to maintain accuracy in alert triggering. - I would appreciate any insights or recommendations on addressing these challenges and achieving the desired configuration for our use case. *Thanks in advance.* -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a1758877-002e-4a36-a4db-96072c3b5abdn%40googlegroups.com.
