Can you explain what you mean by two jobs? Do you mean two scrape configs? On Wed, 22 Jul 2020 at 11:40, Julien Pivotto <[email protected]> wrote:
> On 22 Jul 02:35, Lili Cosic wrote: > > > > > > On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote: > > > > > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto <[email protected] > > > <javascript:>> wrote: > > > > > >> On 22 Jul 02:14, Lili Cosic wrote: > > >> > Only now seen in the docs that I am supposed to start any > discussions > > >> here > > >> > first before opening an issue, sorry about that! :) > > >> > > > >> > Currently there is no way of a target to have higher scrape > priority > > >> over > > >> > another, but if you have a setup and even if you set target limits > and > > >> > sample limits you can still overestimate your setup, you still want > to > > >> have > > >> > a higher priority targets that are preferred over the entire > Prometheus > > >> to > > >> > fail. It would need to be based on the inability to ingest into > tsdb on > > >> the > > >> > current rate we are scrapping, if that is hit the priority class > would > > >> take > > >> > affect and only the highest priority targets would be scrapped in > > >> favour of > > >> > lower priority. Another option which might be simpler would be to > have > > >> a > > >> > global limit on how much prometheus can handle based on perf > testing. > > >> > > > >> > This would be treated as a last resort, and there would definitely > be a > > >> > need for a high severity alert to inform the admin that something > went > > >> > terribly wrong, but because we would still be able to ingest > Prometheus > > >> > metrics for example if they are higher priority class alerting > would be > > >> > possible. > > >> > > >> Hi, > > >> > > >> I think that limiting the number of targets you scrape is already a > last > > >> resort. I don't think we would need a second line of defense. > > >> > > > > > > I agree with Julien here. If you've gotten to this point you're > already > > > seriously overloaded, and prioritising individual targets is just > > > rearranging the deckchairs at that point. > > > > > > > > >> > > >> You can achieve this priority by setting 2 jobs, one which is limited > > >> and one which is not, and use relabeling to decinde which target is > > >> going in which job. > > >> > > > > > > Or more generally, one Prometheus for the important targets and > another > > > for the less important and riskier targets. > > > > > > > I get your point completely Brian, and agree to some degree but people > are > > still going to be setting up a multi tenant prometheus which then causes > > the above problems I mentioned. Even within the riskier targets there > will > > be some more important than others for users. I think we should still > > strive to making a single shared Prometheus as safe as possible, if this > is > > not the priority class I suggested, open to other ideas! > > Then 2 jobs are the answer, one unlimited and one limited. > > The target_limit is already pretty advanced use case. > > > > > > > > > > > Brian > > > > > > > > >> > > >> > > > >> > We could model this on something like PriorityClass > > >> > < > > >> > https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass> > > > >> from > > >> > Kubernetes, but I am open to other suggestions. > > >> > > >> That could be used in relabeling as I said. > > >> > > >> > > > >> > I am open to other suggestions, or maybe there is something like > this > > >> but I > > >> > missed it. The main purpose is to ensure there are protection > > >> mechanisms in > > >> > place, so any ideas and suggestions welcome! > > >> > > > >> > > >> regards, > > >> > > >> > Thanks and kind regards, > > >> > Lili > > >> > > > >> > -- > > >> > You received this message because you are subscribed to the Google > > >> Groups "Prometheus Developers" group. > > >> > To unsubscribe from this group and stop receiving emails from it, > send > > >> an email to [email protected] > > >> <javascript:>. > > >> > To view this discussion on the web visit > > >> > https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com > > >> . > > >> > > >> > > >> -- > > >> Julien Pivotto > > >> @roidelapluie > > >> > > >> -- > > >> You received this message because you are subscribed to the Google > Groups > > >> "Prometheus Developers" group. > > >> To unsubscribe from this group and stop receiving emails from it, > send an > > >> email to [email protected] > <javascript:> > > >> . > > >> To view this discussion on the web visit > > >> > https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen > > >> . > > >> > > > > > > > > > -- > > > Brian Brazil > > > www.robustperception.io > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/b0b9e5f7-239a-4cc7-9108-9e6e015a30d6o%40googlegroups.com > . > > > -- > Julien Pivotto > @roidelapluie > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/20200722094024.GA175281%40oxygen > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAOs1Umx-uFZFPoeOMA-ev4oN5QoRUyODiCWnSZML3hessHkmBQ%40mail.gmail.com.
