Yeah, everything in Prometheus and the Alertmanager revolves around alerts only being identified by their label sets and nothing else. For example, this is important for the Alertmanager to see alerts from multiple Prometheus servers as identical if they have the same label set, even if they began and were resolved at slightly different times.
An "alert" in that sense is different from an "incident" or particular time-based instance of an alert, which Prometheus does not explicitly model. The closest thing to that is the Alertmanager taking in varying alert states over time and turning them into discrete notifications while applying throttling and grouping mechanisms. Those can prevent some flapping on the notification front, and careful alerting rules (averaging over large enough durations, using "for" durations, etc.) can do their part as well. On Fri, Jun 23, 2023 at 9:34 AM Matthias Rampke <[email protected]> wrote: > For a very long time, Prometheus did not store apeet state across > restarts, so the alert startsAt would update even though the condition had > not changed. > > I don't think we ever considered this time to be very meaningful or > stable, partially due to the originally stateless implementation, but also > due to the HA synchronization issue you mentioned. > > Can you explain more what the scenario is where the current label-based > identity doesn't work? If I am reading it right, this is the first time > someone asks for the alerts to be more responsive to flapping, more > typically the desire is to reduce that, identifying successive alerts as > being the same thing even if the alert condition wasn't held for a > short period of time. > > /MR > > On Tue, 20 Jun 2023, 15:14 'George Robinson' via Prometheus Developers, < > [email protected]> wrote: > >> In prometheus/common the fingerprint of an alert is calculated as an >> fnv64a hash of it's labels. The labels are first sorted, and then the label >> name, separator, label value, and another separator for each label is added >> to the hash before the final sum is calculated.period of time >> >> I noticed that something missing from the fingerprint is the alert's >> StartsAt time. You could argue that an alert with labels a₁a₂aₙ that >> started at time t₁ and then resolved at time t₂ is a different alert than >> one also with the labels a₁a₂aₙ but started at time t₃ - and so these two >> alerts should have different fingerprints. >> >> The fact that the fingerprint is constant over its labels has proven >> interesting while debugging cases of flapping alerts in Alertmanager. >> >> However, while I would like to add StartsAt to the fingerprint, I am >> concerned that adding the StartsAt timestamp to the fingerprint will break >> Prometheus rules when run in HA as I do not believe the StartsAt time is >> synchronised across rulers. >> >> I was wondering if there is some historical context for this? Perhaps the >> reasons mentioned above, but there could be others that I am also unaware >> of? >> >> Best regards >> >> George >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/b722e00e-bfd8-4ff4-bbef-e5e0836280bbn%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-developers/b722e00e-bfd8-4ff4-bbef-e5e0836280bbn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gYviO9Ad%3DJXrHKHdZzMDgnu545rTEtPi_hnALvyWzaxAA%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gYviO9Ad%3DJXrHKHdZzMDgnu545rTEtPi_hnALvyWzaxAA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAObpH5xX_sN3cY6EpNZ8vVfK76z__C7bfL-VgPvv87uDe6HS6w%40mail.gmail.com.
