Hi ben, I have upload the heap file below is the link. https://pprof.me/248ce153c305ede009283fef7d9fd15e/?profileType=profile%253Aalloc_objects%253Acount%253Aspace%253Abytes&color_by=filename
*and tsdb-status* curl http://10.224.0.122:9090/api/v1/status/tsdb {"status":"success","data":{"headStats":{"numSeries":162587,"numLabelPairs":8046,"chunkCount":199721,"minTime":1730966400000,"maxTime":1730971 996605},"seriesCountByMetricName":[{"name":"apiserver_request_duration_seconds_bucket","value":6624},{"name":"etcd_request_duration_seconds_bu cket","value":5304},{"name":"apiserver_request_sli_duration_seconds_bucket","value":4400},{"name":"container_tasks_state","value":3280},{"name ":"kube_pod_status_reason","value":2930},{"name":"kube_pod_status_phase","value":2930},{"name":"container_memory_failures_total","value":2624} ,{"name":"kubelet_runtime_operations_duration_seconds_bucket","value":2235},{"name":"apiserver_request_body_size_bytes_bucket","value":2208},{ "name":"kube_replicaset_status_observed_generation","value":2060}],"labelValueCountByLabelName":[{"name":"__name__","value":1250},{"name":"nam e","value":692},{"name":"id","value":608},{"name":"replicaset","value":517},{"name":"mountpoint","value":450},{"name":"lease","value":310},{"n ame":"lease_holder","value":309},{"name":"uid","value":284},{"name":"container_id","value":272},{"name":"le","value":252}],"memoryInBytesByLab elName":[{"name":"id","value":100200},{"name":"mountpoint","value":52087},{"name":"__name__","value":44903},{"name":"name","value":33040},{"na me":"container_id","value":20944},{"name":"replicaset","value":15060},{"name":"lease_holder","value":10996},{"name":"lease","value":10506},{"n ame":"uid","value":10224},{"name":"image_id","value":9545}],"seriesCountByLabelValuePair":[{"name":"app_kubernetes_io_managed_by=Helm","value" :78324},{"name":"app_kubernetes_io_instance=loki-stack","value":78324},{"name":"job=kubernetes-service-endpoints","value":78324},{"name":"app_ kubernetes_io_component=metrics","value":78324},{"name":"app_kubernetes_io_part_of=kube-state-metrics","value":68200},{"name":"app_kubernetes_ io_version=2.8.0","value":68200},{"name":"helm_sh_chart=kube-state-metrics-4.30.0","value":68200},{"name":"app_kubernetes_io_name=kube-state-m etrics","value":68200},{"name":"service=loki-stack-kube-state-metrics","value":66492},{"name":"node=aks-grafana-14132910-vmss000025","value":6 6269}]}} Thanks, Bhanu On Thursday, November 7, 2024 at 2:27:48 PM UTC+5:30 Ben Kochie wrote: > I would recommend getting a heap snapshot and posting it to > https://pprof.me. > > curl -o heap.pprof http://localhost:9090/debug/pprof/heap > > Also including http://localhost:9090/tsdb-status would help. > > On Thu, Nov 7, 2024 at 9:53 AM Bhanu Prakash <[email protected]> > wrote: > >> Hello Prometheus Community, >> >> I’m encountering a memory usage issue with my Prometheus server, >> particularly during and after startup, and I’m hoping to get some insights >> on optimizing it. >> >> *Problem:* Upon startup, my Prometheus instance consumes a large amount >> of memory, primarily due to the WAL (Write-Ahead Log) replay. To address >> this, I enabled --enable-feature=memory-snapshot-on-shutdown, expecting it >> to reduce the startup memory spike by eliminating the need for full WAL >> replay. However, I’m still seeing the memory usage spike to around 5GB on >> startup. Once started, Prometheus continues to hold this high memory, >> without releasing it back to the system. >> >> Is there a recommended way to configure Prometheus to release memory >> post-startup? >> Are there additional configurations or optimizations for large WAL files >> or memory management that could help? >> >> Any guidance or suggestions would be greatly appreciated! >> >> Thank you, >> BhanuPrakash. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion visit >> https://groups.google.com/d/msgid/prometheus-users/99e6725f-f757-45f4-80a4-98c60e2b5063n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/99e6725f-f757-45f4-80a4-98c60e2b5063n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/1dbeaf12-c074-45da-8679-c2370e150861n%40googlegroups.com.
