On Wednesday 23 October 2024 at 16:26:30 UTC+1 mohammad md wrote:
-
-
*annotations: summary: "High CPU usage on {{ $labels.Host }} for {{
$labels.Client }} ({{ $value }})" description: "CPU usage on {{
$labels.Host }} for {{ $labels.Client }} has exceeded 70% for 5 minutes."
resolved: "CPU usage on {{ $labels.Host }} for {{ $labels.Client }} is
back
to normal ({{ $value }})."*
I'm afraid that's not how alerting works in Prometheus.
In Prometheus, you write a PromQL expression which returns an instant
vector, containing zero or more values that generate alerts. For example:
expr: foo > 0.9
The PromQL expression "foo" returns a vector of all metrics with name
"foo". The expression "foo > 0.9" returns the same vector but filtered down
to only those timeseries whose value is > 0.9
If the vector is empty, there's no alert. If the vector is non-empty, then
one or more alerts are active.
So, suppose you have:
foo{instance="a"} 0.8
foo{instance="b"} 0.93
foo{instance="c"} 0.7
foo{instance="d"} 0.99
then the expression foo > 0.9 will return
[
foo{instance="b"} 0.93
foo{instance="d"} 0.99
]
and $value will be 0.93 or 0.99 respectively.
Now, let's say foo{instance="b"} drops to (say) 0.85, then the expression
value becomes
[
foo{instance="d"} 0.99
]
and when foo{instance="d"} becomes 0.6 then you'll get an empty vector
[
]
And at this point, the alert is resolved. But there's no concept of "a
normal value" for $value, because there is no $value at all, because the
vector is empty. All you get is the absence of an alert. This means the
"resolve" message, if it references $value at all, will show the last value
which generated an alert.
That's simply how it works. "foo > 0.9" is not a boolean test, it's a
filter, and $value will only show values which are passed through the
filter; all other values are dropped.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/9a269304-5eaa-462a-b34e-f1c192f87096n%40googlegroups.com.
