This is not a support team, this is a users discussion list. If you require professional support, please see our support page: https://prometheus.io/support-training/
On Sat, Apr 6, 2024 at 7:07 PM Baburao Ganpatrao < [email protected]> wrote: > Dear Support Team, > > I am currently experiencing some discrepancies and unexpected behavior > with the configuration of Prometheus AlertManager. Despite configuring > alerts to fire every 4 hours, I am encountering issues with the actual > number of alerts sent and received by AlertManager. There seems to be a > significant disparity between the expected and observed counts. > > Furthermore, even when no conditions are satisfied for triggering alerts, > Prometheus continues to send alerts every 4 to 5 hours, and AlertManager > subsequently forwards the payload to the configured webhook. I have noticed > this behavior persisting even after a prolonged period of no data changes – > in one instance, alerts were still being received 20 hours after data had > ceased to change. > > Below are the relevant configuration files: prometheus.yml: Scrape > Interval: 2m > > *rules.yml:* groups: > - name: recording-rule > interval: 1h > rules: > - record: myRecord > expr: [expression calculating ratio by dividing two metrics, > greater than a specified value] > - name: alerting-rule > interval: 4h > rules: > - alert: myAlert > expr: max_over_time(myRecord[4h]) > labels: > severity: warning > annotations: > summary: "Summary" > * alertmanager.yml:* group_by: ['alertname'] > group_wait: 30s > group_interval: 1h > repeat_interval: 4h > I would appreciate your assistance in resolving the following issues: > > 1. Identifying and rectifying the discrepancies in the number of > alerts between Prometheus and AlertManager. > 2. Investigating and addressing the persistence of alerts being fired > by Prometheus and forwarded by AlertManager, even in the absence of new > data or changes. > 3. Resolving the issue of receiving alerts at regular intervals, > despite a prolonged period of no new data. > > *It's worth noting that a substantial amount of data is being processed > for alerts, more than 25,000 unique metrics.* > > Your prompt attention to these matters would be greatly appreciated. > > Thank you for your assistance. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmr2P2ABJmpvgdSP6X21OU1ywKeiKF2%3DWw2P_jVJXB33CA%40mail.gmail.com.
