Please remove me from this email list On Sat, Apr 6, 2024 at 12:15 PM Baburao Ganpatrao < [email protected]> wrote:
> Actually, I initially thought of this as support, then now I know its > discussion forum. And I am seeking for help to resolve this strange > behavior of prometheus alertmanager. > > On Saturday 6 April 2024 at 22:39:25 UTC+5:30 Ben Kochie wrote: > >> This is not a support team, this is a users discussion list. >> >> If you require professional support, please see our support page: >> https://prometheus.io/support-training/ >> >> On Sat, Apr 6, 2024 at 7:07 PM Baburao Ganpatrao <[email protected]> >> wrote: >> >>> Dear Support Team, >>> >>> I am currently experiencing some discrepancies and unexpected behavior >>> with the configuration of Prometheus AlertManager. Despite configuring >>> alerts to fire every 4 hours, I am encountering issues with the actual >>> number of alerts sent and received by AlertManager. There seems to be a >>> significant disparity between the expected and observed counts. >>> >>> Furthermore, even when no conditions are satisfied for triggering >>> alerts, Prometheus continues to send alerts every 4 to 5 hours, and >>> AlertManager subsequently forwards the payload to the configured webhook. I >>> have noticed this behavior persisting even after a prolonged period of no >>> data changes – in one instance, alerts were still being received 20 hours >>> after data had ceased to change. >>> >>> Below are the relevant configuration files: prometheus.yml: Scrape >>> Interval: 2m >>> >>> *rules.yml:* groups: >>> - name: recording-rule >>> interval: 1h >>> rules: >>> - record: myRecord >>> expr: [expression calculating ratio by dividing two metrics, >>> greater than a specified value] >>> - name: alerting-rule >>> interval: 4h >>> rules: >>> - alert: myAlert >>> expr: max_over_time(myRecord[4h]) >>> labels: >>> severity: warning >>> annotations: >>> summary: "Summary" >>> * alertmanager.yml:* group_by: ['alertname'] >>> group_wait: 30s >>> group_interval: 1h >>> repeat_interval: 4h >>> I would appreciate your assistance in resolving the following issues: >>> >>> 1. Identifying and rectifying the discrepancies in the number of >>> alerts between Prometheus and AlertManager. >>> 2. Investigating and addressing the persistence of alerts being >>> fired by Prometheus and forwarded by AlertManager, even in the absence of >>> new data or changes. >>> 3. Resolving the issue of receiving alerts at regular intervals, >>> despite a prolonged period of no new data. >>> >>> *It's worth noting that a substantial amount of data is being processed >>> for alerts, more than 25,000 unique metrics.* >>> >>> Your prompt attention to these matters would be greatly appreciated. >>> >>> Thank you for your assistance. >>> >> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/2ff8af79-ebd0-4cab-a058-366ce160820fn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/2ff8af79-ebd0-4cab-a058-366ce160820fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMazdJ8DdA5iGGBXDux-_C1Ww%2BSRJ8mpt_o_d8yJCaW3z74akQ%40mail.gmail.com.
