Alert names don't have to be unique, in fact it can be quite common to use the same alert name multiple times (for different severity levels, for example).
One way to make per-team customizations to generic alerting rules is outlined in https://www.robustperception.io/using-time-series-as-alert-thresholds. But in your case where a team may want to make completely generic rule expression overrides (that aren't expressed as simple threshold values or similar), you would probably have to solve it outside Prometheus somehow with some templating system and render the final rule files accordingly. On Fri, May 1, 2020 at 10:42 PM Alexander Ryabov <[email protected]> wrote: > We're using two groups of Alerts > - common for all teams > - per-team only > > In prometheus config it looks like this: > > rule_files: > - /git-sync/common/*.yaml > - /git-sync/rules/*.yaml > > And directory git-sync comes from team repo. In this repo folder /common/ > comes from git submodule. > This mostly works fine, but there are corner cases when team want to > override some common Alert rule on their level. > Currently it is not possible, consider this case: > > # /git-sync/common/prom.yaml > groups: > - name: prom > rules: > - alert: InstanceDown > expr: up == 0 > for: 5m > labels: > severity: critical > annotations: > summary: Instance scrape failed > > # /git-sync/rules/prom.yaml > groups: > - name: prom > rules: > - alert: InstanceDown > expr: up{group!="maintenance"} == 0 > for: 5m > labels: > severity: critical > annotations: > summary: Instance scrape failed > > This still loads as 2 different alerts in Prometheus, and first one > continues to fire. > Is it possible to override alerts? > Shouldn't alert names be unique? > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/d4532c99-835e-4c0c-b145-9e662a965f2f%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/d4532c99-835e-4c0c-b145-9e662a965f2f%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6Yoxjvot2-x9A_C4jkN_bextUTCj-kMhG4d3z9E7960TkXQ%40mail.gmail.com.
