Hi, the WAL will always need to contain all data from the last few hours (see https://www.robustperception.io/how-much-space-does-the-wal-take-up for more about WAL space usage), so indeed setting a shorter retention time will not affect it, but you can set a flag to enable WAL compression (saving disk space at the cost of more CPU).
However, that is about disk usage, not memory usage (which is what your OOM is about). There you can either give your Prometheus server more RAM to work with, or just give it less work to do (ingest / process / query less). You can get an idea of Prometheus's memory usage from this post: https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion. Note that that post doesn't consider memory usage by queries and other things going on. But ingestion load and especially the number of concurrently active time series are one large factor in how much memory Prometheus requires. Cheers, Julius On Mon, Apr 13, 2020 at 11:03 AM Nishant Ketu <[email protected]> wrote: > We have deployed Prometheus through helm and using after around 2 months > we get OOM error and the pods failed to restart. We have manually clean up > the /data to get the pod running again. I have used the retention flag but > it don't seem to work on wall folder of /data. Any help for this would be > nice. Thanks > > > The information in this message may be proprietary and/or confidential, > and protected from disclosure. If the reader of this message is not the > intended recipient, you are hereby notified that any dissemination, > distribution or copying of this communication is strictly prohibited. If > you have received this communication in error, please notify ATMECS and > delete it from your computer. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/19776f5b-3667-4dc0-bc49-0c4de42e0fdd%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/19776f5b-3667-4dc0-bc49-0c4de42e0fdd%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6YowUbwnFi-tWQb%2B48%2BZrCWzMxZ2GtUbFkFo3kAnzsQyGZA%40mail.gmail.com.
