On 25/08/2023 04:37, sunil sahu wrote:
Hello All,
I need help in my below use case.
We have a centralized python application which continuously monitors
state for other applications by running tests. We creates
*GUAGE* metrics with a value (Ex. 100) and set to different value when
state changes (Ex. 200 or 300) using gauge's set function (python
prometheus_client instrumentation). This run frequently (1min, 5m etc)
depending on tests.
Our central application runs with multiples pods in load balancing. So
any pod can serve the request and set the metric value to new/existing
one.
We are scraping individual pods every minute using k8s discovery
(role: pods), now here any of the pod/instance can have latest/recent
updated value. N*ow my problem is how to get the metric with latest
value out of all instances' metrics.*
Aggregations on 'instance' label like sum/min or max won't be a ideal
choice here, may be `last` (which I don't find. I am aware of
last_over_time but that is kind of to fill the data-point gaps with
previous values)
Currently I am living with `last_over_time(max without(instance)
(metric_name))` but as I explained it is giving wrong results for
flaky tests.
I tried to explain as best as possible, still let me know if any
missing gaps to understand the scenario.
So are you saying that the multiple pods which are all getting scraped
might give totally different values for this metric, but only one of
those is correct?
If that is the case then you'd need to ensure the correct value is
returned by every pod, or you store that value elsewhere and scrape that
other system.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/37460f3a-1ef0-69b5-d3ad-02cf2f8e4154%40Jahingo.com.
