Thanks a lot Jonatan. Just a quick heads-up: We have a Prometheus dev summit on Thursday next week, and I put this on the agenda: https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit
Fabian On Fri, Nov 4, 2022 at 6:46 AM Jonatan Ivanov <[email protected]> wrote: > Hi, > > Sorry for the late reply let me go on-by-one, please bear with me. :) > > *>I recommend taking that question to an OpenMetrics list.* > Thanks, I will open a thread there too based on the discussion here. > > *>Prometheus could independently decide to go beyond what OpenMetrics says* > Since Micrometer uses the Prometheus Java Client this would solve the > issue for us but if it makes sense it would be great to have it > standardized later(?) (in OM). > > *>Maybe your point is that exemplars are tied to the _bucket metrics and > not the _count metric?* > Yes, that's exactly what I'm saying with the caveat that I think this > would be useful for Summaries too not only Histograms. > > *>How exactly would you change this while remaining backwards-compatible > for existing users?* > As far as I can see, adding Exemplars to _count should be a > backward-compatible change (it is an addition). > Would users be broken because of this? > > *>Perhaps the upcoming "native histograms" or "sparse histograms" feature > will suit what you need?* > I'm not sure but I'm not only talking about histograms, I'm also talking > about Summary. > > *>In an OM Histogram, the +Inf bucket fulfills exactly the same function* > > > *as the _count (spec says: "The +Inf bucket counts all requests.") Soif > you would like an examplar on the _count of a Histogram, you can aswell use > an exemplar on the +Inf bucket.* > > I think I disagree with the second sentence. Let's say you have an > application where processing the first request is significantly slower than > the rest (lazy init, populating caches, GC, establishing connections, > etc.). In this environment (I think this is true for lots of apps nowadays) > it can easily happen that the +Inf bucket will be populated with an > Exemplar for the first request and it will never get updated because the > app will never be as slow as it was for the first request. Also, nothing > guarantees that the +Inf bucker will have an exemplar, maybe the processing > was faster than that. As far as I understand, exemplars are not like > cumulative "le" counters so incrementing a bucket does not mean updating an > exemplar (quite the opposite, maybe all of the buckets will be incremented > but only one will get a new Exemplar). This is also true for apps that can > get significantly faster over time (i.e.: JIT). I think a solution here > would be give_me_the_last_updated_bucket(my_histogram) or adding Exemplar > to _count. > > *>Side note: In Java this would be particularly useful because the popular > Spring Boot framework exposes a Summary http_server_requests_seconds by > default that look like this (no quantiles, just _count and _sum)* > This was actually one of the drivers of this request, users are asking for > this from us. I also find it extremely useful: without this I need to > create an additional counter which does not seem right since I already have > one. > > Thanks, > Jonatan > > On Tuesday, October 18, 2022 at 6:29:27 AM UTC-7 [email protected] wrote: > >> Side note: In Java this would be particularly useful because the popular >> Spring Boot framework exposes a Summary http_server_requests_seconds by >> default that look like this (no quantiles, just _count and _sum): >> >> # HELP http_server_requests_seconds >> # TYPE http_server_requests_seconds summary >> http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/",} >> 1.0 >> http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/",} >> 1.014687278 >> >> I think this is pretty useful, you can get request rates and error rates >> out of it. If Prometheus / OpenMetrics had support for Exemplars on the >> _count, users could find example traces per HTTP status and URI. >> >> Fabian >> >> On Tue, Oct 18, 2022 at 3:05 PM Bjoern Rabenstein <[email protected]> >> wrote: >> >>> On 06.10.22 14:45, 'Fabian Stäber' via Prometheus Developers wrote: >>> > >>> > Great question from the CNCF Slack: What's the reason why we don't >>> allow >>> > Exemplars for _count in Summary metrics? >>> > >>> > What do you think? Any reason why Exemplars don't work in _count in >>> > Summaries? Would that be something we could consider supporting? >>> >>> The _count of a Summary _and_ the _count of a Histogram (both >>> conventional as well as the new native ones) is essentially a counter >>> within the larger "structured" metric of a Summary/Histogram. >>> >>> From that perspective, it should have the option of attaching an >>> examplar, as a regular Counter has, too. >>> >>> My speculation why it doesn't in OpenMetrics: >>> >>> In an OM Histogram, the +Inf bucket fulfills exactly the same function >>> as the _count (spec says: "The +Inf bucket counts all requests.") So >>> if you would like an examplar on the _count of a Histogram, you can as >>> well use an exemplar on the +Inf bucket. >>> >>> That obviously doesn't help in the case of a Summary, but I guess the >>> rationale is that Histograms are generally to be preferred over >>> Summaries, and therefore didn't get the thourough treatment when it >>> came to exemplars. >>> >>> >>> However, even if you really dislike the precalculated quantiles in >>> Summaries, there is still the case of a Summary without quantiles. I >>> think adding exemplars to such a Summary is as much needed as adding >>> exemplars to any regular Counter. >>> >>> -- >>> Björn Rabenstein >>> [PGP-ID] 0x851C3DA17D748D03 >>> [email] [email protected] >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/895413bd-416f-40b5-9268-26d3e394ca6fn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-developers/895413bd-416f-40b5-9268-26d3e394ca6fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAPX310gaMq6HvsPwkgBH0msDcp5eJDavKfUbWnEqoMxUbBehSQ%40mail.gmail.com.
