You need to place the /prometheus/ directory on its own volume, not have it write to the same place as /tmp.
Look for "persistentVolume" in the helm chart, or in the documentation for it. I don't use helm myself. Bryan On Wednesday 17 July 2024 at 08:08:39 UTC+1 [email protected] wrote: > Hi Team, > Any updates? > > On Monday, July 1, 2024 at 1:10:52 PM UTC+5:30 Vaibhav Ingulkar wrote: > >> >> Hi @Shiram, >> >> Where can I increase the space? I searched in helm vhart but not found >> any configuration to increase /tmp directory space. >> Can you please suggest on it? >> On Monday, July 1, 2024 at 12:37:54 PM UTC+5:30 Shriram Datar wrote: >> >>> Increase the space. >>> On Wednesday, June 26, 2024 at 10:54:03 PM UTC+5:30 Vaibhav Ingulkar >>> wrote: >>> >>>> Hi Team, >>>> >>>> We are facing below error in prometheus pod and there after it stops >>>> pushing data to mimir - >>>> target=http://pushgateway:9091/metrics msg="appending scrape report >>>> failed" err="write to WAL: log samples: write /prometheus/wal/123456: no >>>> space left on device" >>>> >>>> Prometheus Version : 2.50.1 >>>> >>>> When verified by pod login it shows /tmp directory becomes full i.e. >>>> 11/11 Gi space is consumed. ALso tried to reduced retention period to 1h >>>> but still issue persist and /tmp directory gets full. >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fe9c268c-9313-42f0-b394-c6db0eb1c57an%40googlegroups.com.
