On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote: > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto <[email protected] > <javascript:>> wrote: > >> On 22 Jul 02:14, Lili Cosic wrote: >> > Only now seen in the docs that I am supposed to start any discussions >> here >> > first before opening an issue, sorry about that! :) >> > >> > Currently there is no way of a target to have higher scrape priority >> over >> > another, but if you have a setup and even if you set target limits and >> > sample limits you can still overestimate your setup, you still want to >> have >> > a higher priority targets that are preferred over the entire Prometheus >> to >> > fail. It would need to be based on the inability to ingest into tsdb on >> the >> > current rate we are scrapping, if that is hit the priority class would >> take >> > affect and only the highest priority targets would be scrapped in >> favour of >> > lower priority. Another option which might be simpler would be to have >> a >> > global limit on how much prometheus can handle based on perf testing. >> > >> > This would be treated as a last resort, and there would definitely be a >> > need for a high severity alert to inform the admin that something went >> > terribly wrong, but because we would still be able to ingest Prometheus >> > metrics for example if they are higher priority class alerting would be >> > possible. >> >> Hi, >> >> I think that limiting the number of targets you scrape is already a last >> resort. I don't think we would need a second line of defense. >> > > I agree with Julien here. If you've gotten to this point you're already > seriously overloaded, and prioritising individual targets is just > rearranging the deckchairs at that point. > > >> >> You can achieve this priority by setting 2 jobs, one which is limited >> and one which is not, and use relabeling to decinde which target is >> going in which job. >> > > Or more generally, one Prometheus for the important targets and another > for the less important and riskier targets. >
I get your point completely Brian, and agree to some degree but people are still going to be setting up a multi tenant prometheus which then causes the above problems I mentioned. Even within the riskier targets there will be some more important than others for users. I think we should still strive to making a single shared Prometheus as safe as possible, if this is not the priority class I suggested, open to other ideas! > > Brian > > >> >> > >> > We could model this on something like PriorityClass >> > < >> https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass> >> >> from >> > Kubernetes, but I am open to other suggestions. >> >> That could be used in relabeling as I said. >> >> > >> > I am open to other suggestions, or maybe there is something like this >> but I >> > missed it. The main purpose is to ensure there are protection >> mechanisms in >> > place, so any ideas and suggestions welcome! >> > >> >> regards, >> >> > Thanks and kind regards, >> > Lili >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected] >> <javascript:>. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com >> . >> >> >> -- >> Julien Pivotto >> @roidelapluie >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:> >> . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b0b9e5f7-239a-4cc7-9108-9e6e015a30d6o%40googlegroups.com.
