My personal opinion is that this use case is a bit too niche for adding to the client. I don't think it'll be very common for people to need different data stores for different metrics.
In terms of a text file store, you can make your own store and have it in your codebase, and set it in the Prometheus config, so we wouldn't need to add it to the client at all. And my suggestion is that you set all your metrics to use this same store. However, if you *must* have different stores, you *could* hack your way around this by setting the config to the store you need before declaring the metrics that need a different one, and then resetting it back. You can only do this if your code is not multi-threaded at the time of declaring those metrics, so you want to make sure to define them before worker threads start up if you have any. Again, that's not what I'd recommend as a general use case, it's definitely a hack that is not explicitly supported. I'd suggest using the same store for all of your metrics, but if you absolutely need to do this, that could work. Hope that helps, Daniel On Tuesday, May 18, 2021 at 10:34:54 PM UTC+1 [email protected] wrote: > > Have you considered the pushgateway for this? Cluster-level cron jobs > are what it's designed for. > > Yes, we do. > But I hope that Prometheus ruby client will support text-file store first > of all as it is a much simpler solution for us, as we already have a > text-file collector module. > > I can open a PR with such a feature (per metric store support + text-file > store). > WDYT? > > On Wed, May 19, 2021 at 12:08 AM Brian Brazil < > [email protected]> wrote: > >> On Tue, 18 May 2021 at 21:49, Николай Марков <[email protected]> wrote: >> >>> I will try to describe the problem. >>> >>> We send our app-metrics to statsd and we have a statsd_exporter for >>> prometheus. >>> One of the metric we have is a counter which indicates a count of some >>> "cron job" executions. >>> And we have an alertmanager which checks the following query: >>> >>> sum(increase(extremist_material_notifier_executions_count >>> {environment="production"}[1d2h])) < 1 >>> >>> This worker may run on different servers and statsd automatically adds " >>> instance" label for each metric so this metric may have a different " >>> instance" label on each run - this is why we use "sum" in alert-query. >>> Example of metrics: >>> >>> extremist_material_notifier_executions_count{environment="production", >>> instance="tasks1"} >>> extremist_material_notifier_executions_count{environment="production", >>> instance="tasks2"} >>> extremist_material_notifier_executions_count{environment="production", >>> instance="tasks3"} >>> extremist_material_notifier_executions_count{environment="production", >>> instance="tasks4"} >>> >>> As you know statsd forgets its metrics on restarts so when it happens we >>> have a "counter reset" and "increase"-function does not handle it well - it >>> does not show increase when counter was e.g. 100 before the restart and >>> disappears after statsd restart. >>> >>> So I would like to use "textfile collector" module to persist such >>> metrics and expose all other metrics through "/metrics" endpoint as usual - >>> this is why I would like to use different stores for different metrics. >>> >> >> Have you considered the pushgateway for this? Cluster-level cron jobs are >> what it's designed for. >> >> Brian >> >> >>> >>> среда, 12 мая 2021 г. в 23:14:29 UTC+3, [email protected]: >>> >>>> Hello, >>>> >>>> Do you have an example of a use case for this? >>>> >>>> In general, you wouldn't "care to lose" any metrics. On a server >>>> restart, you'd lose all of them, which Prometheus would see as a reset, >>>> and >>>> deal with it normally. >>>> That is actually the best practice, and we recommend emptying the >>>> metrics temp directory on startup, if you have a persistent file system, >>>> to >>>> ensure metrics get reset. >>>> >>>> Sorry if I misunderstood your use case, however. An example would >>>> probably help. >>>> >>>> Thank you! >>>> Daniel >>>> >>>> On Saturday, May 8, 2021 at 6:21:50 PM UTC+1 [email protected] wrote: >>>> >>>>> *Problem* >>>>> >>>>> I would like to use different stores for different metrics. >>>>> E.g. for metrics that I *don't scare* to lose I would like to use >>>>> faster storage, and for metrics I *scare* to lose - more reliable >>>>> storage like DirectFileStore. >>>>> But seems like it is possible for now just to use a single store for >>>>> all metrics, because there is a one "global" config setting >>>>> `Prometheus::Client.config.data_store=` >>>>> >>>>> I'm ready to make a PR with such functionality. What do you think >>>>> about such feature? >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Brian Brazil >> www.robustperception.io >> > > > -- > С уважением, > Николай Марков. > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/76716349-69de-4b99-bd77-5358cda51458n%40googlegroups.com.
