> Have you considered the pushgateway for this? Cluster-level cron jobs are what it's designed for.
Yes, we do. But I hope that Prometheus ruby client will support text-file store first of all as it is a much simpler solution for us, as we already have a text-file collector module. I can open a PR with such a feature (per metric store support + text-file store). WDYT? On Wed, May 19, 2021 at 12:08 AM Brian Brazil < [email protected]> wrote: > On Tue, 18 May 2021 at 21:49, Николай Марков <[email protected]> wrote: > >> I will try to describe the problem. >> >> We send our app-metrics to statsd and we have a statsd_exporter for >> prometheus. >> One of the metric we have is a counter which indicates a count of some >> "cron job" executions. >> And we have an alertmanager which checks the following query: >> >> sum(increase(extremist_material_notifier_executions_count >> {environment="production"}[1d2h])) < 1 >> >> This worker may run on different servers and statsd automatically adds " >> instance" label for each metric so this metric may have a different " >> instance" label on each run - this is why we use "sum" in alert-query. >> Example of metrics: >> >> extremist_material_notifier_executions_count{environment="production", >> instance="tasks1"} >> extremist_material_notifier_executions_count{environment="production", >> instance="tasks2"} >> extremist_material_notifier_executions_count{environment="production", >> instance="tasks3"} >> extremist_material_notifier_executions_count{environment="production", >> instance="tasks4"} >> >> As you know statsd forgets its metrics on restarts so when it happens we >> have a "counter reset" and "increase"-function does not handle it well - it >> does not show increase when counter was e.g. 100 before the restart and >> disappears after statsd restart. >> >> So I would like to use "textfile collector" module to persist such >> metrics and expose all other metrics through "/metrics" endpoint as usual - >> this is why I would like to use different stores for different metrics. >> > > Have you considered the pushgateway for this? Cluster-level cron jobs are > what it's designed for. > > Brian > > >> >> среда, 12 мая 2021 г. в 23:14:29 UTC+3, [email protected]: >> >>> Hello, >>> >>> Do you have an example of a use case for this? >>> >>> In general, you wouldn't "care to lose" any metrics. On a server >>> restart, you'd lose all of them, which Prometheus would see as a reset, and >>> deal with it normally. >>> That is actually the best practice, and we recommend emptying the >>> metrics temp directory on startup, if you have a persistent file system, to >>> ensure metrics get reset. >>> >>> Sorry if I misunderstood your use case, however. An example would >>> probably help. >>> >>> Thank you! >>> Daniel >>> >>> On Saturday, May 8, 2021 at 6:21:50 PM UTC+1 [email protected] wrote: >>> >>>> *Problem* >>>> >>>> I would like to use different stores for different metrics. >>>> E.g. for metrics that I *don't scare* to lose I would like to use >>>> faster storage, and for metrics I *scare* to lose - more reliable >>>> storage like DirectFileStore. >>>> But seems like it is possible for now just to use a single store for >>>> all metrics, because there is a one "global" config setting >>>> `Prometheus::Client.config.data_store=` >>>> >>>> I'm ready to make a PR with such functionality. What do you think about >>>> such feature? >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- С уважением, Николай Марков. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAGyoySeZyRRg50_2E8vtzeE%3DoT31v3uOPUMVeyM_tZELP%2BVuNw%40mail.gmail.com.
