Actually, I initially thought of this as support, then now I know its discussion forum. And I am seeking for help to resolve this strange behavior of prometheus alertmanager.
On Saturday 6 April 2024 at 22:39:25 UTC+5:30 Ben Kochie wrote: > This is not a support team, this is a users discussion list. > > If you require professional support, please see our support page: > https://prometheus.io/support-training/ > > On Sat, Apr 6, 2024 at 7:07 PM Baburao Ganpatrao <[email protected]> > wrote: > >> Dear Support Team, >> >> I am currently experiencing some discrepancies and unexpected behavior >> with the configuration of Prometheus AlertManager. Despite configuring >> alerts to fire every 4 hours, I am encountering issues with the actual >> number of alerts sent and received by AlertManager. There seems to be a >> significant disparity between the expected and observed counts. >> >> Furthermore, even when no conditions are satisfied for triggering alerts, >> Prometheus continues to send alerts every 4 to 5 hours, and AlertManager >> subsequently forwards the payload to the configured webhook. I have noticed >> this behavior persisting even after a prolonged period of no data changes – >> in one instance, alerts were still being received 20 hours after data had >> ceased to change. >> >> Below are the relevant configuration files: prometheus.yml: Scrape >> Interval: 2m >> >> *rules.yml:* groups: >> - name: recording-rule >> interval: 1h >> rules: >> - record: myRecord >> expr: [expression calculating ratio by dividing two metrics, >> greater than a specified value] >> - name: alerting-rule >> interval: 4h >> rules: >> - alert: myAlert >> expr: max_over_time(myRecord[4h]) >> labels: >> severity: warning >> annotations: >> summary: "Summary" >> * alertmanager.yml:* group_by: ['alertname'] >> group_wait: 30s >> group_interval: 1h >> repeat_interval: 4h >> I would appreciate your assistance in resolving the following issues: >> >> 1. Identifying and rectifying the discrepancies in the number of >> alerts between Prometheus and AlertManager. >> 2. Investigating and addressing the persistence of alerts being fired >> by Prometheus and forwarded by AlertManager, even in the absence of new >> data or changes. >> 3. Resolving the issue of receiving alerts at regular intervals, >> despite a prolonged period of no new data. >> >> *It's worth noting that a substantial amount of data is being processed >> for alerts, more than 25,000 unique metrics.* >> >> Your prompt attention to these matters would be greatly appreciated. >> >> Thank you for your assistance. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2ff8af79-ebd0-4cab-a058-366ce160820fn%40googlegroups.com.
