I will try to describe the problem.
We send our app-metrics to statsd and we have a statsd_exporter for
prometheus.
One of the metric we have is a counter which indicates a count of some
"cron job" executions.
And we have an alertmanager which checks the following query:
sum(increase(extremist_material_notifier_executions_count
{environment="production"}[1d2h])) < 1
This worker may run on different servers and statsd automatically adds "
instance" label for each metric so this metric may have a different "
instance" label on each run - this is why we use "sum" in alert-query.
Example of metrics:
extremist_material_notifier_executions_count{environment="production",
instance="tasks1"}
extremist_material_notifier_executions_count{environment="production",
instance="tasks2"}
extremist_material_notifier_executions_count{environment="production",
instance="tasks3"}
extremist_material_notifier_executions_count{environment="production",
instance="tasks4"}
As you know statsd forgets its metrics on restarts so when it happens we
have a "counter reset" and "increase"-function does not handle it well - it
does not show increase when counter was e.g. 100 before the restart and
disappears after statsd restart.
So I would like to use "textfile collector" module to persist such metrics
and expose all other metrics through "/metrics" endpoint as usual - this is
why I would like to use different stores for different metrics.
среда, 12 мая 2021 г. в 23:14:29 UTC+3, [email protected]:
> Hello,
>
> Do you have an example of a use case for this?
>
> In general, you wouldn't "care to lose" any metrics. On a server restart,
> you'd lose all of them, which Prometheus would see as a reset, and deal
> with it normally.
> That is actually the best practice, and we recommend emptying the metrics
> temp directory on startup, if you have a persistent file system, to ensure
> metrics get reset.
>
> Sorry if I misunderstood your use case, however. An example would probably
> help.
>
> Thank you!
> Daniel
>
> On Saturday, May 8, 2021 at 6:21:50 PM UTC+1 [email protected] wrote:
>
>> *Problem*
>>
>> I would like to use different stores for different metrics.
>> E.g. for metrics that I *don't scare* to lose I would like to use faster
>> storage, and for metrics I *scare* to lose - more reliable storage like
>> DirectFileStore.
>> But seems like it is possible for now just to use a single store for all
>> metrics, because there is a one "global" config setting
>> `Prometheus::Client.config.data_store=`
>>
>> I'm ready to make a PR with such functionality. What do you think about
>> such feature?
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com.
