We use the VerticalPodAutoscaler to auto-scale Prometheus instances. This would probably be more reliable than trying to deal with horizontal sharding.
We also have a controller that shards by namespace, which helps isolate tenants on the cluster from causing problems with each other. On Thu, Jul 11, 2024 at 10:13 PM Julio Leal <[email protected]> wrote: > Hi everyone, > > I have Prometheus installed as a statefulset in a Kubernetes cluster > running in HA mode (with 2 instances). Currently, I use Prometheus as a > statefulset to enable retries in remote write, which I use to store my > metrics. However, this setup has its drawbacks: it prevents autoscaling > when I scrape a large volume of metrics. > > Is there a way to configure Prometheus solely as a collector that doesn’t > perform any relabeling or buffering, and only scrapes and sends metrics to > a remote write endpoint without losing metrics during the scrape? I > understand there might be some metric loss due to the removal of the buffer. > > I considered using a daemonset, but this approach is costly and there is a > risk of losing metrics during a pod rollout on the node. Alternatively, > using a deployment would require handling deduplication. > > Has anyone implemented a similar configuration? > > I hope this helps! Let me know if you need any further adjustments. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/39590679-6209-4fbb-ae21-75ceed36b7bcn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/39590679-6209-4fbb-ae21-75ceed36b7bcn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmo2nRR-_CXU-LQCS3QFE9_ADWpX03UVVHXYV4uwEXNh8g%40mail.gmail.com.
