I haven't tried using it yet but Labyrinth Labs has made a preprocessor that makes it possible to override alerts.
See https://www.lablabs.io/2020/04/19/how-we-solved-our-need-to-override-prometheus-alerts/ On Saturday, 2 May 2020 14:49:13 UTC+4, Julius Volz wrote: > > Alert names don't have to be unique, in fact it can be quite common to use > the same alert name multiple times (for different severity levels, for > example). > > One way to make per-team customizations to generic alerting rules is > outlined in > https://www.robustperception.io/using-time-series-as-alert-thresholds. > But in your case where a team may want to make completely generic rule > expression overrides (that aren't expressed as simple threshold values or > similar), you would probably have to solve it outside Prometheus somehow > with some templating system and render the final rule files accordingly. > > On Fri, May 1, 2020 at 10:42 PM Alexander Ryabov <[email protected] > <javascript:>> wrote: > >> We're using two groups of Alerts >> - common for all teams >> - per-team only >> >> In prometheus config it looks like this: >> >> rule_files: >> - /git-sync/common/*.yaml >> - /git-sync/rules/*.yaml >> >> And directory git-sync comes from team repo. In this repo folder /common/ >> comes from git submodule. >> This mostly works fine, but there are corner cases when team want to >> override some common Alert rule on their level. >> Currently it is not possible, consider this case: >> >> # /git-sync/common/prom.yaml >> groups: >> - name: prom >> rules: >> - alert: InstanceDown >> expr: up == 0 >> for: 5m >> labels: >> severity: critical >> annotations: >> summary: Instance scrape failed >> >> # /git-sync/rules/prom.yaml >> groups: >> - name: prom >> rules: >> - alert: InstanceDown >> expr: up{group!="maintenance"} == 0 >> for: 5m >> labels: >> severity: critical >> annotations: >> summary: Instance scrape failed >> >> This still loads as 2 different alerts in Prometheus, and first one >> continues to fire. >> Is it possible to override alerts? >> Shouldn't alert names be unique? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/d4532c99-835e-4c0c-b145-9e662a965f2f%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/d4532c99-835e-4c0c-b145-9e662a965f2f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/09ec7773-bb98-4a98-90c7-d0c323af3c1e%40googlegroups.com.
