Hi! > What we'd like for our purposes is some way to re-send all such suppressed alert notifications if the alert is still active when an inhibition or mute expires. So, for example, if we mute alert notifications out of hours and some alert triggers then, at the start of working hours we'd get an alert notification for that alert (or group of them, etc).
Alertmanager isn't able to re-send alerts when a mute time interval becomes inactive, because time intervals are not checked until it's time to send the notification again. > We don't want to do this by reducing our repeat_interval > (which is currently large). However, I think you can approximate this by having a short group_interval (for example, 5 minutes). Each time the group_interval elapses, Alertmanager will check if there are new alerts in the group for which a notification has not been sent. This will be true if an alert fired over night, and is still firing in working hours. However, if the alert resolved before the start of working hours, then a notification might not be sent as the alert is resolved. Best regards, George On Monday, March 18, 2024 at 6:01:50 PM UTC Chris Siebenmann wrote: > Alertmanager has supported inhibiting alert notifications for a long > time, and somewhat more recently it's added support for muting them > during specific time ranges (or only sending them in specific time > ranges). As far as I know, the behaviour (for both inhibited alerts and > muted alerts) is that the alert notification that would be sent is > entirely suppressed; Alertmanager basically 'sends' it into the void, > and afterwards will only re-send an (unchanged) alert notification at > your repeat_interval. > > What we'd like for our purposes is some way to re-send all such > suppressed alert notifications if the alert is still active when an > inhibition or mute expires. So, for example, if we mute alert > notifications out of hours and some alert triggers then, at the start of > working hours we'd get an alert notification for that alert (or group of > them, etc). We don't want to do this by reducing our repeat_interval > (which is currently large). > > (I am aware that we could build a dashboard of Prometheus alerts that > are currently active, but such a dashboard wouldn't contain our specific > and helpful alert message text and so on.) > > Is there any clever way to do this within the current features of > Alertmanager (and/or Prometheus)? > > (I've even looked at the Alertmanager API reference, but as far as I can > see there's no way to trigger the re-sending of an alert notification. I > suppose we could write a script that tried to add an artificial new > alert to every outstanding alert group that matched suitable criteria to > trigger an Alertmanager resend because the alert group has 'changed', > but that seems like a bad hack and potentially fragile.) > > Thanks in advance. > > - cks > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0264dec8-2e01-468a-9360-72be51dd3e6en%40googlegroups.com.
