On 22 Jul 16:32, Frederic Branczyk wrote: > Can you explain what you mean by two jobs? Do you mean two scrape configs?
Yes. > > On Wed, 22 Jul 2020 at 11:40, Julien Pivotto <[email protected]> > wrote: > > > On 22 Jul 02:35, Lili Cosic wrote: > > > > > > > > > On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote: > > > > > > > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto <[email protected] > > > > <javascript:>> wrote: > > > > > > > >> On 22 Jul 02:14, Lili Cosic wrote: > > > >> > Only now seen in the docs that I am supposed to start any > > discussions > > > >> here > > > >> > first before opening an issue, sorry about that! :) > > > >> > > > > >> > Currently there is no way of a target to have higher scrape > > priority > > > >> over > > > >> > another, but if you have a setup and even if you set target limits > > and > > > >> > sample limits you can still overestimate your setup, you still want > > to > > > >> have > > > >> > a higher priority targets that are preferred over the entire > > Prometheus > > > >> to > > > >> > fail. It would need to be based on the inability to ingest into > > tsdb on > > > >> the > > > >> > current rate we are scrapping, if that is hit the priority class > > would > > > >> take > > > >> > affect and only the highest priority targets would be scrapped in > > > >> favour of > > > >> > lower priority. Another option which might be simpler would be to > > have > > > >> a > > > >> > global limit on how much prometheus can handle based on perf > > testing. > > > >> > > > > >> > This would be treated as a last resort, and there would definitely > > be a > > > >> > need for a high severity alert to inform the admin that something > > went > > > >> > terribly wrong, but because we would still be able to ingest > > Prometheus > > > >> > metrics for example if they are higher priority class alerting > > would be > > > >> > possible. > > > >> > > > >> Hi, > > > >> > > > >> I think that limiting the number of targets you scrape is already a > > last > > > >> resort. I don't think we would need a second line of defense. > > > >> > > > > > > > > I agree with Julien here. If you've gotten to this point you're > > already > > > > seriously overloaded, and prioritising individual targets is just > > > > rearranging the deckchairs at that point. > > > > > > > > > > > >> > > > >> You can achieve this priority by setting 2 jobs, one which is limited > > > >> and one which is not, and use relabeling to decinde which target is > > > >> going in which job. > > > >> > > > > > > > > Or more generally, one Prometheus for the important targets and > > another > > > > for the less important and riskier targets. > > > > > > > > > > I get your point completely Brian, and agree to some degree but people > > are > > > still going to be setting up a multi tenant prometheus which then causes > > > the above problems I mentioned. Even within the riskier targets there > > will > > > be some more important than others for users. I think we should still > > > strive to making a single shared Prometheus as safe as possible, if this > > is > > > not the priority class I suggested, open to other ideas! > > > > Then 2 jobs are the answer, one unlimited and one limited. > > > > The target_limit is already pretty advanced use case. > > > > > > > > > > > > > > > > Brian > > > > > > > > > > > >> > > > >> > > > > >> > We could model this on something like PriorityClass > > > >> > < > > > >> > > https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass> > > > > > >> from > > > >> > Kubernetes, but I am open to other suggestions. > > > >> > > > >> That could be used in relabeling as I said. > > > >> > > > >> > > > > >> > I am open to other suggestions, or maybe there is something like > > this > > > >> but I > > > >> > missed it. The main purpose is to ensure there are protection > > > >> mechanisms in > > > >> > place, so any ideas and suggestions welcome! > > > >> > > > > >> > > > >> regards, > > > >> > > > >> > Thanks and kind regards, > > > >> > Lili > > > >> > > > > >> > -- > > > >> > You received this message because you are subscribed to the Google > > > >> Groups "Prometheus Developers" group. > > > >> > To unsubscribe from this group and stop receiving emails from it, > > send > > > >> an email to [email protected] > > > >> <javascript:>. > > > >> > To view this discussion on the web visit > > > >> > > https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com > > > >> . > > > >> > > > >> > > > >> -- > > > >> Julien Pivotto > > > >> @roidelapluie > > > >> > > > >> -- > > > >> You received this message because you are subscribed to the Google > > Groups > > > >> "Prometheus Developers" group. > > > >> To unsubscribe from this group and stop receiving emails from it, > > send an > > > >> email to [email protected] > > <javascript:> > > > >> . > > > >> To view this discussion on the web visit > > > >> > > https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen > > > >> . > > > >> > > > > > > > > > > > > -- > > > > Brian Brazil > > > > www.robustperception.io > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > > Groups "Prometheus Developers" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected]. > > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/prometheus-developers/b0b9e5f7-239a-4cc7-9108-9e6e015a30d6o%40googlegroups.com > > . > > > > > > -- > > Julien Pivotto > > @roidelapluie > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/prometheus-developers/20200722094024.GA175281%40oxygen > > . > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CAOs1Umx-uFZFPoeOMA-ev4oN5QoRUyODiCWnSZML3hessHkmBQ%40mail.gmail.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200722143414.GA609330%40oxygen.
