Load average is not really "load" in the way you're thinking. That query is
not going to work the way you are thinking.
You probably want CPU utilization.
Something like: avg without (cpu,mode)
(1-rate(node_cpu_seconds_total{...}[$__rate_interval]))
Again, it's recommended to create recording rules for these kinds of
queries.
On Thu, Feb 22, 2024 at 8:44 PM Puneet Singh <[email protected]>
wrote:
> Thank you Chris,
> That information related to your setup 's resource usage was very helpful.
>
> Hi Ben,
> The query i had shared is provided as part of the default exporter
> dashboard from grafana.
> It is supposed to give a measure of the % load on a server .
> [image: Untitled.png]
> [image: Untitled.png]
> Here data fill be read for 2 series and along with that there are irate,
> avg_over_time and a division operators.
> Scrape interval is 20s. So if i query data for last 3 years, can i assume
> that this might take ~ 40MiB of memory per series ? and these operators
> will not have any significant memory consumption?
>
>
>
> Regards
> Puneet
>
> On Wednesday 21 February 2024 at 12:18:30 UTC+5:30 Ben Kochie wrote:
>
>> Again, it depends, I don't know what you want to collect and store. It
>> could be enough, it could not be enough. Only you can capacity plan this
>> with the knowledge of your internal requirements.
>>
>> On Tue, Feb 20, 2024 at 8:09 PM Puneet Singh <[email protected]>
>> wrote:
>>
>>> Hi Ben,
>>> Thank you for the response. So for scraping and storing/writing the data
>>> for this setup, prometheus would need ~30G memory .
>>>
>>> i have a query on 2 Data Query scenarios-
>>> scenario1: We run a query to get 3 years of load via grafana as -
>>> node_load15{instance="$node",job="$job"}[$__rate_interval]
>>
>>
>>> scenario2: we decide to query the system load worth 3 years for a
>>> single server from the grafana, which uses following query:
>>> avg_over_time(node_load15{instance="$node",job="$job"}[$__rate_interval])
>>> * 100 / on(instance) group_left sum by
>>> (instance)(irate(node_cpu_seconds_total{instance="$node",job="$job"}[$__rate_interval]))
>>>
>>> this involves 2 series, plus 3 functions (avg_over_time,sum by , irate)
>>> and a division operation
>>>
>>> Is there a way to get a rough estimate of the amount of CPU and RAM
>>> required for the queries mentioned in scenario1 and scenario2?
>>>
>>
>> Again, it depends on how exactly many series and samples you need to
>> load. Figure maybe 20MiB of memory per series with 15s scrape intervals.
>>
>> That query is a bit nonsensical. Load average is not really a useful
>> metric to look at, and I don't understand why you're dividing it by CPU
>> seconds.
>>
>> For things that you want to graph over long periods of time, you can use
>> recording rules to generate pre-computed data that is easier to query.
>>
>>
>> https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/
>>
>>
>>>
>>> Regards,
>>> Puneet
>>>
>>> On Tuesday 20 February 2024 at 23:17:19 UTC+5:30 Ben Kochie wrote:
>>>
>>>> Prometheus needs a minimum of about 4KiB per "active series". Retention
>>>> policy doesn't affect the memory usage very much.
>>>>
>>>> If you have 10,000 metrics per server and 300 servers, that's 3 million
>>>> series.
>>>>
>>>> 3 million * 4KiB = 11.4GiB of memory.
>>>>
>>>> Of course, you will also need some page cache and such. Usually 2x is
>>>> more than good enough, but it depends on your query load. So 20-30GiB of
>>>> memory should be enough.
>>>>
>>>> But of course, it highly depends on how many metrics per server you
>>>> have.
>>>>
>>>> On Tue, Feb 20, 2024 at 6:24 PM Puneet Singh <[email protected]>
>>>> wrote:
>>>>
>>>>> HI All,
>>>>> I am planning to store 3 years of data from 300 server in a single
>>>>> prometheus server. The data will primarily consist of default exporter
>>>>> metrics and the server has 500G memory and 80 cores.
>>>>>
>>>>> I'd like to ensure that my solution is optimal in terms of resource
>>>>> utilization, query performance and scalability. Is there a general
>>>>> recommendation about the amount of resources i should be having on this
>>>>> setup?
>>>>>
>>>>> Regards,
>>>>> Puneet
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Prometheus Users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/prometheus-users/2735babe-1905-45e0-90e7-2316c2f35f8bn%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/prometheus-users/2735babe-1905-45e0-90e7-2316c2f35f8bn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/31c10035-a36d-473b-95cb-016cd5060dd3n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/31c10035-a36d-473b-95cb-016cd5060dd3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/0f45cf21-2620-466c-92c1-ddd79d55922cn%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/0f45cf21-2620-466c-92c1-ddd79d55922cn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrYd%3DXR%2BeeUhqbduokHYy-W1S5eTrs8HB7p_bCrwtedQw%40mail.gmail.com.
