On Tue, 18 May 2021 at 21:49, Николай Марков <[email protected]> wrote:
> I will try to describe the problem.
>
> We send our app-metrics to statsd and we have a statsd_exporter for
> prometheus.
> One of the metric we have is a counter which indicates a count of some
> "cron job" executions.
> And we have an alertmanager which checks the following query:
>
> sum(increase(extremist_material_notifier_executions_count
> {environment="production"}[1d2h])) < 1
>
> This worker may run on different servers and statsd automatically adds "
> instance" label for each metric so this metric may have a different "
> instance" label on each run - this is why we use "sum" in alert-query.
> Example of metrics:
>
> extremist_material_notifier_executions_count{environment="production",
> instance="tasks1"}
> extremist_material_notifier_executions_count{environment="production",
> instance="tasks2"}
> extremist_material_notifier_executions_count{environment="production",
> instance="tasks3"}
> extremist_material_notifier_executions_count{environment="production",
> instance="tasks4"}
>
> As you know statsd forgets its metrics on restarts so when it happens we
> have a "counter reset" and "increase"-function does not handle it well - it
> does not show increase when counter was e.g. 100 before the restart and
> disappears after statsd restart.
>
> So I would like to use "textfile collector" module to persist such metrics
> and expose all other metrics through "/metrics" endpoint as usual - this is
> why I would like to use different stores for different metrics.
>
Have you considered the pushgateway for this? Cluster-level cron jobs are
what it's designed for.
Brian
>
> среда, 12 мая 2021 г. в 23:14:29 UTC+3, [email protected]:
>
>> Hello,
>>
>> Do you have an example of a use case for this?
>>
>> In general, you wouldn't "care to lose" any metrics. On a server restart,
>> you'd lose all of them, which Prometheus would see as a reset, and deal
>> with it normally.
>> That is actually the best practice, and we recommend emptying the metrics
>> temp directory on startup, if you have a persistent file system, to ensure
>> metrics get reset.
>>
>> Sorry if I misunderstood your use case, however. An example would
>> probably help.
>>
>> Thank you!
>> Daniel
>>
>> On Saturday, May 8, 2021 at 6:21:50 PM UTC+1 [email protected] wrote:
>>
>>> *Problem*
>>>
>>> I would like to use different stores for different metrics.
>>> E.g. for metrics that I *don't scare* to lose I would like to use
>>> faster storage, and for metrics I *scare* to lose - more reliable
>>> storage like DirectFileStore.
>>> But seems like it is possible for now just to use a single store for all
>>> metrics, because there is a one "global" config setting
>>> `Prometheus::Client.config.data_store=`
>>>
>>> I'm ready to make a PR with such functionality. What do you think about
>>> such feature?
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Brian Brazil
www.robustperception.io
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLogaJiHUOfGAo%3D70MZOvURedMhRLvOsiWN5qRn2H8xJbQ%40mail.gmail.com.
