On Friday 29 March 2024 at 22:09:18 UTC Chris Siebenmann wrote: I believe that recording rules and alerting rules similarly may have their evaluation time happen at different offsets within their evaluation interval. This is done for the similar reason of spreading out the internal load of rule evaluations across time.
I think it's more accurate to say that *rule groups* are spread spread over their evaluation interval, and rules within the same rule group are evaluated sequentially <https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules>. This is how you can build rules that depend on each other, e.g. a recording rule followed by other rules that depend on its output; put them in the same rule group. As for scraping: you *can* change this staleness interval, using --query.lookback-delta, but it's strongly not recommended. Using the default of 5 mins, you should use a maximum scrape interval of 2 mins so that even if you miss one scrape for a random reason, you still have two points within the lookback-delta so that the timeseries does not go stale. There's no good reason to scrape at one hour intervals: * Prometheus is extremely efficient with its storage compression, especially when adjacent data points are equal, so scraping the same value every 2 minutes is going to use hardly any more storage than scraping it every hour. * If you're worried about load on the exporter because responding to a scrape is slow or expensive, then you should run the exporter every hour from a local cronjob, and write its output to a persistent location (e.g. to PushGateway or statsd_exporter, or simply write it to a file which can be picked up by node_exporter textfile-collector or even a vanilla HTTP server). You can then scrape this as often as you like. node_exporter textfile-collector exposes an extra metrics for the timestamp on each file, so you can alert in the case that the file isn't being updated. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/244eb39e-1ded-4161-80cf-b32deb9cd2c7n%40googlegroups.com.
