On Wed, 22 Jul 2020 at 10:18, Julien Pivotto <[email protected]> wrote:
> On 22 Jul 02:14, Lili Cosic wrote: > > Only now seen in the docs that I am supposed to start any discussions > here > > first before opening an issue, sorry about that! :) > > > > Currently there is no way of a target to have higher scrape priority > over > > another, but if you have a setup and even if you set target limits and > > sample limits you can still overestimate your setup, you still want to > have > > a higher priority targets that are preferred over the entire Prometheus > to > > fail. It would need to be based on the inability to ingest into tsdb on > the > > current rate we are scrapping, if that is hit the priority class would > take > > affect and only the highest priority targets would be scrapped in favour > of > > lower priority. Another option which might be simpler would be to have a > > global limit on how much prometheus can handle based on perf testing. > > > > This would be treated as a last resort, and there would definitely be a > > need for a high severity alert to inform the admin that something went > > terribly wrong, but because we would still be able to ingest Prometheus > > metrics for example if they are higher priority class alerting would be > > possible. > > Hi, > > I think that limiting the number of targets you scrape is already a last > resort. I don't think we would need a second line of defense. > I agree with Julien here. If you've gotten to this point you're already seriously overloaded, and prioritising individual targets is just rearranging the deckchairs at that point. > > You can achieve this priority by setting 2 jobs, one which is limited > and one which is not, and use relabeling to decinde which target is > going in which job. > Or more generally, one Prometheus for the important targets and another for the less important and riskier targets. Brian > > > > > We could model this on something like PriorityClass > > < > https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass> > from > > Kubernetes, but I am open to other suggestions. > > That could be used in relabeling as I said. > > > > > I am open to other suggestions, or maybe there is something like this > but I > > missed it. The main purpose is to ensure there are protection mechanisms > in > > place, so any ideas and suggestions welcome! > > > > regards, > > > Thanks and kind regards, > > Lili > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com > . > > > -- > Julien Pivotto > @roidelapluie > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLpehh15H%3DmFuBNR1pH0Nk0xQ%2BpGs629t3f-uwoPhgC9wA%40mail.gmail.com.
