Thank you Chris, That information related to your setup 's resource usage was very helpful.
Hi Ben, The query i had shared is provided as part of the default exporter dashboard from grafana. It is supposed to give a measure of the % load on a server . [image: Untitled.png] [image: Untitled.png] Here data fill be read for 2 series and along with that there are irate, avg_over_time and a division operators. Scrape interval is 20s. So if i query data for last 3 years, can i assume that this might take ~ 40MiB of memory per series ? and these operators will not have any significant memory consumption? Regards Puneet On Wednesday 21 February 2024 at 12:18:30 UTC+5:30 Ben Kochie wrote: > Again, it depends, I don't know what you want to collect and store. It > could be enough, it could not be enough. Only you can capacity plan this > with the knowledge of your internal requirements. > > On Tue, Feb 20, 2024 at 8:09 PM Puneet Singh <[email protected]> wrote: > >> Hi Ben, >> Thank you for the response. So for scraping and storing/writing the data >> for this setup, prometheus would need ~30G memory . >> >> i have a query on 2 Data Query scenarios- >> scenario1: We run a query to get 3 years of load via grafana as - >> node_load15{instance="$node",job="$job"}[$__rate_interval] > > >> scenario2: we decide to query the system load worth 3 years for a single >> server from the grafana, which uses following query: >> avg_over_time(node_load15{instance="$node",job="$job"}[$__rate_interval]) >> * 100 / on(instance) group_left sum by >> (instance)(irate(node_cpu_seconds_total{instance="$node",job="$job"}[$__rate_interval])) >> >> this involves 2 series, plus 3 functions (avg_over_time,sum by , irate) >> and a division operation >> >> Is there a way to get a rough estimate of the amount of CPU and RAM >> required for the queries mentioned in scenario1 and scenario2? >> > > Again, it depends on how exactly many series and samples you need to load. > Figure maybe 20MiB of memory per series with 15s scrape intervals. > > That query is a bit nonsensical. Load average is not really a useful > metric to look at, and I don't understand why you're dividing it by CPU > seconds. > > For things that you want to graph over long periods of time, you can use > recording rules to generate pre-computed data that is easier to query. > > https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ > > >> >> Regards, >> Puneet >> >> On Tuesday 20 February 2024 at 23:17:19 UTC+5:30 Ben Kochie wrote: >> >>> Prometheus needs a minimum of about 4KiB per "active series". Retention >>> policy doesn't affect the memory usage very much. >>> >>> If you have 10,000 metrics per server and 300 servers, that's 3 million >>> series. >>> >>> 3 million * 4KiB = 11.4GiB of memory. >>> >>> Of course, you will also need some page cache and such. Usually 2x is >>> more than good enough, but it depends on your query load. So 20-30GiB of >>> memory should be enough. >>> >>> But of course, it highly depends on how many metrics per server you have. >>> >>> On Tue, Feb 20, 2024 at 6:24 PM Puneet Singh <[email protected]> >>> wrote: >>> >>>> HI All, >>>> I am planning to store 3 years of data from 300 server in a single >>>> prometheus server. The data will primarily consist of default exporter >>>> metrics and the server has 500G memory and 80 cores. >>>> >>>> I'd like to ensure that my solution is optimal in terms of resource >>>> utilization, query performance and scalability. Is there a general >>>> recommendation about the amount of resources i should be having on this >>>> setup? >>>> >>>> Regards, >>>> Puneet >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/2735babe-1905-45e0-90e7-2316c2f35f8bn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/2735babe-1905-45e0-90e7-2316c2f35f8bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/31c10035-a36d-473b-95cb-016cd5060dd3n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/31c10035-a36d-473b-95cb-016cd5060dd3n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0f45cf21-2620-466c-92c1-ddd79d55922cn%40googlegroups.com.
