Without knowing the rest of your configuration, there's not much anyone can do to help.
On Sat, Jul 31, 2021 at 1:17 PM 'Yaron Idan' via Prometheus Users < [email protected]> wrote: > Thanks for the information, my pod is already requesting 24gb and the host > it's running on has 32gb available, so I'm running out of space to maneuver > on these machines. > I can create a dedicated node group for prometheus alone but I'm trying to > avoid that if possible since it will make it more of a special creature > inside our cluster, and I'm usually try to avoid that whenever possible. > > On Saturday, July 31, 2021 at 12:24:44 PM UTC+3 [email protected] wrote: > >> There is no single correct answer. It depends on your ingestion rate and >> labels. >> >> Just give the pod more memory so that it does not crash. You do not want >> to be too tight on memory anyway, as Prometheus needs some non-process >> memory for page cache in the pod in order to perform queries. >> >> On Fri, Jul 30, 2021 at 9:21 PM 'Yaron Idan' via Prometheus Users < >> [email protected]> wrote: >> >>> I have a prometheus pod running on my cluster requesting 24gb of memory. >>> It runs great most of the time but when it needs to read a large WAL it >>> OOMs and the WAL has to be deleted manually. >>> I've saw multiple messages suggesting that the instance just needs more >>> memory in order to replay the WAL, but I wanted to get a better grasp of >>> how the size of WAL relates to the memory the pod needs - if I can find out >>> said ratio, I can probably alert well in advance if the WAL grows too big >>> and needs to be handled with before we get in a crashloop. >>> >>> Any idea how can I find out this information? >>> >>> Thanks. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/d1b4df82-6be0-4b50-8731-9fe9890e41f0n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/d1b4df82-6be0-4b50-8731-9fe9890e41f0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/cf7cc795-217d-4855-b369-6e802283d6e8n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/cf7cc795-217d-4855-b369-6e802283d6e8n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqJHaWO1OMbRL_K1x5T1-1B90MpEFO0DRfgnxK7VvMQaQ%40mail.gmail.com.
