On Wednesday 23 October 2024 at 16:26:30 UTC+1 bashar madani wrote: The issue I’m facing is that Alertmanager keeps repeating the FIRING message even after the issue is resolved. I want to ensure that only the RESOLVED message is sent when the problem is fixed.
If you have a group of alerts, and some of them are resolved, then you'll get a new [FIRING] message with the smaller set of alerts. That's because, clearly, at least one is still firing. You'll only get [RESOLVED] when the last alert in the group has stopped firing. If you want, you can disabling grouping entirely and then each alert will individually generate its own mails (firing and resolved). But that could mean a lot more mails if there are lots of similar alerts which would normally be grouped. https://prometheus.io/docs/alerting/latest/configuration/#route # To aggregate by all possible labels use the special value '...' as the sole label name, for example: # group_by: ['...'] # This effectively disables aggregation entirely, passing through all # alerts as-is. This is unlikely to be what you want, unless you have # a very low alert volume or your upstream notification system performs # its own grouping. Does anyone have examples or best practices to share? Personally, I'd say the best practice with resolved messages is *not to send them at all* (send_resolved: false). For an explanation see: https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit If something was worth alerting on then it's worth investigating: even if the alert condition is no longer present, it clearly was earlier. Just saying "oh look, it's gone away, never mind" is not helping to understand or fix the problem (with the system and/or with the alert itself). Seriously: turning off resolved messages is great. At very least, it reduces your notification volume by 50%. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/5cdea308-750f-44e9-a6b4-e42b8894bf6bn%40googlegroups.com.
