That's only effective in limiting the number of targets, the point here is that selectively scraping those with a higher priority based on backpressure of the system as a whole.
On Wed, 22 Jul 2020 at 17:00, Julien Pivotto <[email protected]> wrote: > On 22 Jul 16:47, Frederic Branczyk wrote: > > In practice even that can still be problematic. You only know that > > Prometheus has a problem when everything fails, the point is to keep > things > > alive well enough for more critical components. > > > > On Wed, 22 Jul 2020 at 16:38, Julien Pivotto <[email protected] > > > > wrote: > > > > > On 22 Jul 16:36, Frederic Branczyk wrote: > > > > It's unclear how that helps, can you help me understand? > > > > > > - job: highprio > > > relabel_configs: > > > - target_label: job > > > replacement: pods > > > - source_labels: [__meta_pod_priority] > > > regex: high > > > action: keep > > highprio job will always be scraped. > > > > - job: lowprio > > > relabel_configs: > > > - target_label: job > > > replacement: pods > > > - source_labels: [__meta_pod_priority] > > > regex: high > > > action: drop > > > target_limit: 1000 > > > > > > > > > > > On Wed, 22 Jul 2020 at 16:34, Julien Pivotto < > [email protected] > > > > > > > > wrote: > > > > > > > > > On 22 Jul 16:32, Frederic Branczyk wrote: > > > > > > Can you explain what you mean by two jobs? Do you mean two scrape > > > > > configs? > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > On Wed, 22 Jul 2020 at 11:40, Julien Pivotto < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > On 22 Jul 02:35, Lili Cosic wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil > wrote: > > > > > > > > > > > > > > > > > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto < > > > > > [email protected] > > > > > > > > > <javascript:>> wrote: > > > > > > > > > > > > > > > > > >> On 22 Jul 02:14, Lili Cosic wrote: > > > > > > > > >> > Only now seen in the docs that I am supposed to start > any > > > > > > > discussions > > > > > > > > >> here > > > > > > > > >> > first before opening an issue, sorry about that! :) > > > > > > > > >> > > > > > > > > > >> > Currently there is no way of a target to have higher > scrape > > > > > > > priority > > > > > > > > >> over > > > > > > > > >> > another, but if you have a setup and even if you set > target > > > > > limits > > > > > > > and > > > > > > > > >> > sample limits you can still overestimate your setup, you > > > still > > > > > want > > > > > > > to > > > > > > > > >> have > > > > > > > > >> > a higher priority targets that are preferred over the > entire > > > > > > > Prometheus > > > > > > > > >> to > > > > > > > > >> > fail. It would need to be based on the inability to > ingest > > > into > > > > > > > tsdb on > > > > > > > > >> the > > > > > > > > >> > current rate we are scrapping, if that is hit the > priority > > > class > > > > > > > would > > > > > > > > >> take > > > > > > > > >> > affect and only the highest priority targets would be > > > scrapped > > > > > in > > > > > > > > >> favour of > > > > > > > > >> > lower priority. Another option which might be simpler > would > > > be > > > > > to > > > > > > > have > > > > > > > > >> a > > > > > > > > >> > global limit on how much prometheus can handle based on > perf > > > > > > > testing. > > > > > > > > >> > > > > > > > > > >> > This would be treated as a last resort, and there would > > > > > definitely > > > > > > > be a > > > > > > > > >> > need for a high severity alert to inform the admin that > > > > > something > > > > > > > went > > > > > > > > >> > terribly wrong, but because we would still be able to > ingest > > > > > > > Prometheus > > > > > > > > >> > metrics for example if they are higher priority class > > > alerting > > > > > > > would be > > > > > > > > >> > possible. > > > > > > > > >> > > > > > > > > >> Hi, > > > > > > > > >> > > > > > > > > >> I think that limiting the number of targets you scrape is > > > already > > > > > a > > > > > > > last > > > > > > > > >> resort. I don't think we would need a second line of > defense. > > > > > > > > >> > > > > > > > > > > > > > > > > > > I agree with Julien here. If you've gotten to this point > you're > > > > > > > already > > > > > > > > > seriously overloaded, and prioritising individual targets > is > > > just > > > > > > > > > rearranging the deckchairs at that point. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > >> You can achieve this priority by setting 2 jobs, one > which is > > > > > limited > > > > > > > > >> and one which is not, and use relabeling to decinde which > > > target > > > > > is > > > > > > > > >> going in which job. > > > > > > > > >> > > > > > > > > > > > > > > > > > > Or more generally, one Prometheus for the important > targets and > > > > > > > another > > > > > > > > > for the less important and riskier targets. > > > > > > > > > > > > > > > > > > > > > > > > > I get your point completely Brian, and agree to some degree > but > > > > > people > > > > > > > are > > > > > > > > still going to be setting up a multi tenant prometheus which > then > > > > > causes > > > > > > > > the above problems I mentioned. Even within the riskier > targets > > > there > > > > > > > will > > > > > > > > be some more important than others for users. I think we > should > > > still > > > > > > > > strive to making a single shared Prometheus as safe as > possible, > > > if > > > > > this > > > > > > > is > > > > > > > > not the priority class I suggested, open to other ideas! > > > > > > > > > > > > > > Then 2 jobs are the answer, one unlimited and one limited. > > > > > > > > > > > > > > The target_limit is already pretty advanced use case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Brian > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > We could model this on something like PriorityClass > > > > > > > > >> > < > > > > > > > > >> > > > > > > > > > > > > > > > > https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass > > > > > > > > > > > > > > > > > > > > > >> from > > > > > > > > >> > Kubernetes, but I am open to other suggestions. > > > > > > > > >> > > > > > > > > >> That could be used in relabeling as I said. > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > I am open to other suggestions, or maybe there is > something > > > like > > > > > > > this > > > > > > > > >> but I > > > > > > > > >> > missed it. The main purpose is to ensure there are > > > protection > > > > > > > > >> mechanisms in > > > > > > > > >> > place, so any ideas and suggestions welcome! > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> regards, > > > > > > > > >> > > > > > > > > >> > Thanks and kind regards, > > > > > > > > >> > Lili > > > > > > > > >> > > > > > > > > > >> > -- > > > > > > > > >> > You received this message because you are subscribed to > the > > > > > Google > > > > > > > > >> Groups "Prometheus Developers" group. > > > > > > > > >> > To unsubscribe from this group and stop receiving emails > > > from > > > > > it, > > > > > > > send > > > > > > > > >> an email to > > > [email protected] > > > > > > > > >> <javascript:>. > > > > > > > > >> > To view this discussion on the web visit > > > > > > > > >> > > > > > > > > > > > > > > > > https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com > > > > > > > > >> . > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> -- > > > > > > > > >> Julien Pivotto > > > > > > > > >> @roidelapluie > > > > > > > > >> > > > > > > > > >> -- > > > > > > > > >> You received this message because you are subscribed to > the > > > Google > > > > > > > Groups > > > > > > > > >> "Prometheus Developers" group. > > > > > > > > >> To unsubscribe from this group and stop receiving emails > from > > > it, > > > > > > > send an > > > > > > > > >> email to > [email protected] > > > > > > > <javascript:> > > > > > > > > >> . > > > > > > > > >> To view this discussion on the web visit > > > > > > > > >> > > > > > > > > > > > > > > > > https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen > > > > > > > > >> . > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Brian Brazil > > > > > > > > > www.robustperception.io > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > > > Google > > > > > > > Groups "Prometheus Developers" group. > > > > > > > > To unsubscribe from this group and stop receiving emails > from it, > > > > > send > > > > > > > an email to [email protected] > . > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > > > > > https://groups.google.com/d/msgid/prometheus-developers/b0b9e5f7-239a-4cc7-9108-9e6e015a30d6o%40googlegroups.com > > > > > > > . > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Julien Pivotto > > > > > > > @roidelapluie > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the > Google > > > > > Groups > > > > > > > "Prometheus Developers" group. > > > > > > > To unsubscribe from this group and stop receiving emails from > it, > > > send > > > > > an > > > > > > > email to [email protected]. > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > > > > > https://groups.google.com/d/msgid/prometheus-developers/20200722094024.GA175281%40oxygen > > > > > > > . > > > > > > > > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the > Google > > > > > Groups "Prometheus Developers" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > send > > > > > an email to [email protected]. > > > > > > To view this discussion on the web visit > > > > > > > > > https://groups.google.com/d/msgid/prometheus-developers/CAOs1Umx-uFZFPoeOMA-ev4oN5QoRUyODiCWnSZML3hessHkmBQ%40mail.gmail.com > > > > > . > > > > > > > > > > -- > > > > > Julien Pivotto > > > > > @roidelapluie > > > > > > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > Groups "Prometheus Developers" group. > > > > To unsubscribe from this group and stop receiving emails from it, > send > > > an email to [email protected]. > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/prometheus-developers/CAOs1UmzgPKCrpmsDb4v3CrN9Oe%2Bmaka8bosCDuodmjmd-RAyLw%40mail.gmail.com > > > . > > > > > > -- > > > Julien Pivotto > > > @roidelapluie > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CAOs1UmyxR%3DQ%2B6_emwh12CVwkwemU%2B-tzenvgP1WQ%2BCHnw67UUQ%40mail.gmail.com > . > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAOs1UmwjYgxU9ABkATe04febF_010n3%3DKVoEm8J_5XGnf0je%2Bg%40mail.gmail.com.
