This: go_memstats_alloc_bytes / prometheus_tsdb_head_series gives you bytes per time series. This number is the base of my capacity planning. It depends on how many labels end up on your metrics but usually it's fairly stable. We usually see around 3-4KB per time series.
If you have read_read or lots of expensive queries this number will include that, which can make it less stable. process_resident_memory_bytes depends on a few factors. First GOGC settings, by default it's 100 and so process_resident_memory_bytes might get around 2x the size of Go heap as the result. Second data on disk and some recent but not yet written to disk time series will be mmapped and that memory will end up in process_resident_memory_bytes, but the OS will manage this so process_resident_memory_bytes might change in ways that are a bit unpredictable. In general there's a lot of different components that eat memory, usually the biggest one is all the scraped time series, that's why I only focus on go_memstats_alloc_bytes / prometheus_tsdb_head_series ratio for capacity planning. If you take your average "bytes per time series", then multiply that by the number of time series you must store, then by 2 (to account for GC and other things eating memory) then you usually get a rough idea how much memory you need. On Friday, 25 November 2022 at 12:01:08 UTC [email protected] wrote: > Hi everyone > I'm trying understand and create an end of life of my prometheus instance. > I think that my promethes will die as my number of timeseries increases > and I need more memory ram. > How can I create a correlation between my timeseries growth and my memory > ram growth? > > I already try to use: > > 1. container_memory_working_set_bytes with prometheus_tsdb_head_series > 2. go_memstats_alloc_bytes with prometheus_tsdb_head_series > 3. go_memstats_heap_inuse_bytes with prometheus_tsdb_head_series > 4. process_resident_memory_bytes with prometheus_tsdb_head_series > > > The combination closest that I can was number 3 that I can a correlation > of 0.56 > > Is there another way to create this correlation between Memory Ram and > timeseries to mensure end of life or the growth of the prometheus instance? > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/26a75193-f1eb-4a93-ad5c-1aff2962a15dn%40googlegroups.com.
