Hello Brian,
thanks for clarification. I investigated the issue further and found that
Grafana Dashboards is manipulationg the data and for that reason
"max_over_time" for the last 1h showed the correct peak and max_over_time
for the last 24hrs did not show that peak because of a too low set of Data
points. I increased it to 11.000 and then I was able to see the peak value
again as expected. As "min Step" I set 15s in Grafana.
However this only solved the first part of my main problem. Now I can
reliably query the peaks of a time range and do not miss the peak.
If I run "max_over_time{}[1m:15s]" it will show me the peak of every 1m
evaluating every 15s sample. That's ok.
second step would be to get the top3 max_over_time values of the selected
time_range. If I run "max_over_time{}[1m:15s]" for the last 3 days I will
get 3d x 24h x 60m "max_over_time" values. May goal was and that is why I
used "topk(3,)" (wrong) to get the top3 values of the last 24hrs evaluated
by "max_over_time{}[1m:15s]". But topk(3,) only show me the top3 values at
the same timestamp
third step would be to get the time of these top3 values calculated earlier.
the fourth step would be to get the top3 vaues of the month august and the
top3 values of the month july.
Maybe you can push mit into the righ direction.
My two prometheus are 2.54.1 and 2.53.1
Brian Candler schrieb am Samstag, 31. August 2024 um 10:16:01 UTC+2:
> Why are you doing a subquery there? max_over_time(metric[1h]) should give
> you the largest value at any time over that 1h period. The range vector
> includes all the points in that time period, without resampling.
>
> A subquery could be used if you needed to take an instant vector
> expression and turn it into a range vector by evaluating it at multiple
> time instants, e.g.
>
> max_over_time( (metric > 10 < 100)[1h:15s] )
>
> But for a simple vector expression, the range vector is better than the
> subquery as you get all the data points without resampling.
>
> You said before:
>
> > However first problem if the max value is e.g. 22 and it appears several
> times within the timerange I see this displayxed several times.
>
> That makes no sense. The result of max_over_time() is an *instant vector*.
> By definition, it only has one value for each unique set of labels. If you
> see multiple values of 22, then they are for separate timeseries, and each
> will be identified by its unique sets of labels.
>
> That's what max_over_time does: it works on a range vector of timeseries,
> and gives you the *single* maximum for *each* timeseries. If you pass it a
> range vector with 10 timeseries, you will get an instant vector with 10
> timeseries.
>
> > I would like to idealle see the most recent ones
>
> That also makes no sense. For each timeseries, you will get the maximum
> value of that timeseries across the whole time range, regardless of at what
> time it occurred, and regardless of the values of any other timeseries.
>
> topk(3, ...) then just picks whichever three timeseries have the highest
> maxima over the time period.
>
> > Why do I see a correct peak using
> > max_over_time(metric{}[1h:15s])
> >
> > but if I run this command the peak is lower than with the other command
> before?
> > max_over_time(metric{}[24h:15s])
>
> I'm not sure, but first, try comparing the range vector forms:
>
> max_over_time(metric{}[1h])
> max_over_time(metric{}[24h])
>
> If those work as expected, then there may be some issue with subqueries.
> That can be drilled down into by looking at the raw data. Try these queries
> in the PromQL browser, set to "table" rather than "graph" mode:
>
> metric{}[24h]
> metric{}[24h:15s]
>
> It will show the actual data that max_over_time() is working across. It
> might be some issue around resampling of the data, but I can't think off
> the top of my head what it could be.
>
> What version of prometheus are you running? It could be a bug with
> subqueries, which may or may not be fixed in later versions.
>
> Also, please remove Grafana from the equation. Enter your PromQL queries
> directly into the PromQL browser in Prometheus. There are lots of ways you
> can misconfigure Grafana or otherwise confuse matters, e.g. by asking it to
> sweep an instant vector query over a time range to form a graph.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/4ec69f1d-ac75-4fda-8330-313398474164n%40googlegroups.com.
