You may be interested in this talk I gave at PromCon recently, showing some detail around where the memory is used: https://youtu.be/vc5LgoiP_CA
Bryan On Friday, 25 November 2022 at 12:13:10 UTC [email protected] wrote: > This: > go_memstats_alloc_bytes / prometheus_tsdb_head_series > > gives you bytes per time series. This number is the base of my capacity > planning. It depends on how many labels end up on your metrics but usually > it's fairly stable. We usually see around 3-4KB per time series. > > If you have read_read or lots of expensive queries this number will > include that, which can make it less stable. > > process_resident_memory_bytes depends on a few factors. > First GOGC settings, by default it's 100 and so > process_resident_memory_bytes might get around 2x the size of Go heap as > the result. > Second data on disk and some recent but not yet written to disk time > series will be mmapped and that memory will end up in > process_resident_memory_bytes, but the OS will manage this so > process_resident_memory_bytes might change in ways that are a bit > unpredictable. > > In general there's a lot of different components that eat memory, usually > the biggest one is all the scraped time series, that's why I only focus on > go_memstats_alloc_bytes / prometheus_tsdb_head_series ratio for capacity > planning. > If you take your average "bytes per time series", then multiply that by > the number of time series you must store, then by 2 (to account for GC and > other things eating memory) then you usually get a rough idea how much > memory you need. > On Friday, 25 November 2022 at 12:01:08 UTC [email protected] wrote: > >> Hi everyone >> I'm trying understand and create an end of life of my prometheus instance. >> I think that my promethes will die as my number of timeseries increases >> and I need more memory ram. >> How can I create a correlation between my timeseries growth and my memory >> ram growth? >> >> I already try to use: >> >> 1. container_memory_working_set_bytes with prometheus_tsdb_head_series >> 2. go_memstats_alloc_bytes with prometheus_tsdb_head_series >> 3. go_memstats_heap_inuse_bytes with prometheus_tsdb_head_series >> 4. process_resident_memory_bytes with prometheus_tsdb_head_series >> >> >> The combination closest that I can was number 3 that I can a correlation >> of 0.56 >> >> Is there another way to create this correlation between Memory Ram and >> timeseries to mensure end of life or the growth of the prometheus instance? >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/799956b7-8a22-4b3c-87bb-fcc83e2466b7n%40googlegroups.com.
