The pushgateway does use the prometheus pull mechanism. I'd really suggest
taking a look at it: https://github.com/prometheus/pushgateway
As for how to export a timestamp, The timestamp metric would just be a
sister metric to your main metric and you would update when you update the
main metric with the current epoch time. So if your metrics were:
my_gauge{....} <value>
my_guage_ts{...} <epoch_timestamp>
Then your query to return the most recent value would look something like:
my_guage and max(my_gauge_ts)
--Matt
On Tue, Sep 5, 2023, 12:40 sunil sahu <[email protected]> wrote:
> Thanks for quick response
> Yes you understood it correctly. Earlier we were using push mechanism only
> for statsd format metrics from our custom/on promise instaces. But now want
> to use promql format with pull mechanism.
>
> It would be great help islf you can share example for exporting
> timestamps with gauge metrics? Would it be 1 metric/pod or 1 for each
> metrics?
>
>
> Regards,
> Sunil
>
>
> On Tue, 5 Sept 2023, 9:44 am Matt Doughty, <[email protected]> wrote:
>
>> So if I'm understanding you correctly. You have multiple pods
>> exporting a gauge based on a remote request, and because it is load
>> balanced the request could go to one of many pods, and you are trying
>> to figure out which one is the most recent? This sounds like a
>> perfect use case for pushgateway as it will have the latest value, and
>> the metric will survive pod death.
>>
>> That said, if you are married to the current pattern you have set up,
>> the answer would be to export a corresponding timestamp metric so you
>> can select the most recently updated pod's gauge value. I hope this
>> helps.
>>
>> --Matt
>>
>> On Mon, Sep 4, 2023 at 3:23 PM Stuart Clark <[email protected]>
>> wrote:
>> >
>> > On 25/08/2023 04:37, sunil sahu wrote:
>> >
>> > Hello All,
>> >
>> > I need help in my below use case.
>> >
>> > We have a centralized python application which continuously monitors
>> state for other applications by running tests. We creates GUAGE metrics
>> with a value (Ex. 100) and set to different value when state changes (Ex.
>> 200 or 300) using gauge's set function (python prometheus_client
>> instrumentation). This run frequently (1min, 5m etc) depending on tests.
>> >
>> > Our central application runs with multiples pods in load balancing. So
>> any pod can serve the request and set the metric value to new/existing one.
>> >
>> > We are scraping individual pods every minute using k8s discovery (role:
>> pods), now here any of the pod/instance can have latest/recent updated
>> value. Now my problem is how to get the metric with latest value out of all
>> instances' metrics.
>> >
>> > Aggregations on 'instance' label like sum/min or max won't be a ideal
>> choice here, may be `last` (which I don't find. I am aware of
>> last_over_time but that is kind of to fill the data-point gaps with
>> previous values)
>> >
>> > Currently I am living with `last_over_time(max without(instance)
>> (metric_name))` but as I explained it is giving wrong results for flaky
>> tests.
>> >
>> >
>> > I tried to explain as best as possible, still let me know if any
>> missing gaps to understand the scenario.
>> >
>> > So are you saying that the multiple pods which are all getting scraped
>> might give totally different values for this metric, but only one of those
>> is correct?
>> >
>> > If that is the case then you'd need to ensure the correct value is
>> returned by every pod, or you store that value elsewhere and scrape that
>> other system.
>> >
>> > --
>> > Stuart Clark
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to [email protected].
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/37460f3a-1ef0-69b5-d3ad-02cf2f8e4154%40Jahingo.com
>> .
>>
>>
>>
>> --
>> --Matt
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/CAGyBzczvoh8GFLYrHH2y31swyUqQyNhDhb3298BM7x_1KHcq%2BA%40mail.gmail.com.
