Hi! At my team we're facing an issue using remote write with prometheus on a consul connect service mesh.
Right now we have prometheus configured to remote write to a consul connect service mesh endpoint (http://localhost:19090). Consul deals with handling sending connections to every discovered copy of our remote write service. Prometheus seems to be establishing long living TCP connections and at first when we have two copies of the remote service running they are balanced. The problem comes up when we one of the copies goes away for any reason, prometheus detects this, and establishes new TCP connections to the localhost endpoint but because on consul we only have one copy running all of them end up going to the one copy that is running and the one that comes up sits there doing nothing. In order to fix it I think we would need to expose some more configuration options on the remote write specifically the ones that are set here ( https://github.com/prometheus/prometheus/blob/d30f202c08a7bf4109f18c755c2cbc6a067666bb/vendor/github.com/prometheus/common/config/http_config.go#L158-L174) and ideally a max number of sends per http client. I wanted to write this to see what the opinion of the dev team is and if this change makes sense. I do think at least the keep alive option should be exposed to not reuse TCP connections. We could probably get it to work as we expect by lowering the idle timeouts so maybe exposing those two options would be enough. What do y'all think? Has anyone faced this kind of issue using another service mesh? -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/9802ad54-6c0e-43b7-b754-b1da763f7d3cn%40googlegroups.com.
