I found several posts which ask if it is possible that prometheus agent can store the metrics locally if remote_write destination is not available.
This post describes that it should work at least for 2hrs.: https://prometheus.io/blog/2021/11/16/agent/ However here is a guthub issue which seems to describe the same issue I saw in my environment: https://github.com/prometheus/prometheus/issues/11919 Can someone confirm this is working with 2.49.1 to store metrics for some time on prometheus agent until remotw_write is available again? if yes how did you configure this? Alexander Wilke schrieb am Dienstag, 12. März 2024 um 23:16:10 UTC+1: > Hello, > > I am running a prometheus server as a central instance. > I configured prometheus in agent mode on another system and send the logs > via remote_write from agent to central prometheus. This works. > > I changed the password in web.config.yml on the central server so it does > not match for the prometheus agent anymore. I could see that metrics did > not arrive anymore. > > I waited form 2-3 minutes and then reconfigured the correct password on > the central prometheus, di a reload and I coud see metrics arrive > immediately after that change. Unfortunately I did not receive the metrics > from the outage time. > > my main intention was exactly this - that I can get the metrics if there > is an issue in the network or whatever between agent and central prometheus. > > Is this expected behaviour or misconfiguration/misunderstanding on my site? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f257c25-700a-4ed7-bdff-87e4fbcaabe9n%40googlegroups.com.
