Would I be right in thinking this is the code which tripped you up? https://github.com/prometheus/prometheus/blob/c2b4de3611a6/model/textparse/openmetricsparse.go#L472-L479 This is insisting that exemplars only work on metrics ending in "_bucket" or "_total".
Personally I would be fine with relaxing that, although it does seem strictly aligned with the OpenMetrics spec. On Saturday, 5 November 2022 at 05:21:03 UTC+1 [email protected] wrote: > Hi, > > *@Fabian:* Thank you very much! Is the summit open, can I join? > > *@Bryan:* > I think I would make the decision if it would make sense for the > Prometheus Server to be able to process the Exemplars on _count first. If > so, then I would look into the different clients and their APIs. > > For Histograms, I don't think this should result in a client API change, I > think this should only affect the behavior of the implementation. Also, I > think source, binary, and behavioral compatibility can be kept. I think it > does not matter where the Exemplar is coming from (directly from the user > or from the sampler), the interesting part happens after the implementation > got the Exemplar. In the case of Histograms, this would mean not just > updating the reference of the Exemplar of the current bucket but also > updating an extra reference to the latest Exemplar (that actually belongs > to _count). So the histogram would hold references to N+1 exemplars where N > is the number of buckets and the +1 Exemplar is the latest recorded one. > > For Summaries, this needs a client API change (an addition) since right > now Summaries do not have Exemplars support. I think this should be very > similar to the Exemplars support of Counters. > > I would like to call out two things: > - I'm not an expert of any of the Prometheus clients (I only used the Java > and the JavaScript clients). > - I think I would not even be affected by the changes of the client APIs > since Micrometer is not using these, it directly creates a > Collector.MetricFamilySamples.Sample instead (that can accept Exemplars > as of today). So if the Prometheus server could process Exemplars on > _count, I think my use-case should be covered. But I would definitely add > the support to the client APIs too so that users who use the clients can > enjoy these features. > > Thanks, > Jonatan > > On Friday, November 4, 2022 at 1:09:22 AM UTC-7 [email protected] wrote: > >> >>> Also, what do you think about supporting _count in Histograms (since >> Histogram extends Summary with buckets)? >> >> How exactly would you change this while remaining backwards-compatible >> for existing users? >> > As far as I can see, adding Exemplars to _count should be a >> backward-compatible change (it is an addition). >> >> I would like to see what you can see. Please spell it out for me. >> >> The current API on Histogram is ObserveWithExemplar(), and it adds the >> exemplar to a _bucket metric. >> Would you change the behaviour of that API? Would you add a new API? >> >> Bryan >> >> On Friday, 4 November 2022 at 05:46:39 UTC [email protected] wrote: >> >>> Hi, >>> >>> Sorry for the late reply let me go on-by-one, please bear with me. :) >>> >>> *>I recommend taking that question to an OpenMetrics list.* >>> Thanks, I will open a thread there too based on the discussion here. >>> >>> *>Prometheus could independently decide to go beyond what OpenMetrics >>> says* >>> Since Micrometer uses the Prometheus Java Client this would solve the >>> issue for us but if it makes sense it would be great to have it >>> standardized later(?) (in OM). >>> >>> *>Maybe your point is that exemplars are tied to the _bucket metrics and >>> not the _count metric?* >>> Yes, that's exactly what I'm saying with the caveat that I think this >>> would be useful for Summaries too not only Histograms. >>> >>> *>How exactly would you change this while remaining backwards-compatible >>> for existing users?* >>> As far as I can see, adding Exemplars to _count should be a >>> backward-compatible change (it is an addition). >>> Would users be broken because of this? >>> >>> *>Perhaps the upcoming "native histograms" or "sparse histograms" >>> feature will suit what you need?* >>> I'm not sure but I'm not only talking about histograms, I'm also talking >>> about Summary. >>> >>> *>In an OM Histogram, the +Inf bucket fulfills exactly the same function* >>> >>> >>> *as the _count (spec says: "The +Inf bucket counts all requests.") Soif >>> you would like an examplar on the _count of a Histogram, you can aswell use >>> an exemplar on the +Inf bucket.* >>> >>> I think I disagree with the second sentence. Let's say you have an >>> application where processing the first request is significantly slower than >>> the rest (lazy init, populating caches, GC, establishing connections, >>> etc.). In this environment (I think this is true for lots of apps nowadays) >>> it can easily happen that the +Inf bucket will be populated with an >>> Exemplar for the first request and it will never get updated because the >>> app will never be as slow as it was for the first request. Also, nothing >>> guarantees that the +Inf bucker will have an exemplar, maybe the processing >>> was faster than that. As far as I understand, exemplars are not like >>> cumulative "le" counters so incrementing a bucket does not mean updating an >>> exemplar (quite the opposite, maybe all of the buckets will be incremented >>> but only one will get a new Exemplar). This is also true for apps that can >>> get significantly faster over time (i.e.: JIT). I think a solution here >>> would be give_me_the_last_updated_bucket(my_histogram) or adding >>> Exemplar to _count. >>> >>> *>Side note: In Java this would be particularly useful because the >>> popular Spring Boot framework exposes a Summary >>> http_server_requests_seconds by default that look like this (no quantiles, >>> just _count and _sum)* >>> This was actually one of the drivers of this request, users are asking >>> for this from us. I also find it extremely useful: without this I need to >>> create an additional counter which does not seem right since I already have >>> one. >>> >>> Thanks, >>> Jonatan >>> >>> On Tuesday, October 18, 2022 at 6:29:27 AM UTC-7 [email protected] wrote: >>> >>>> Side note: In Java this would be particularly useful because the >>>> popular Spring Boot framework exposes a Summary >>>> http_server_requests_seconds by default that look like this (no quantiles, >>>> just _count and _sum): >>>> >>>> # HELP http_server_requests_seconds >>>> # TYPE http_server_requests_seconds summary >>>> http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/",} >>>> 1.0 >>>> http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/",} >>>> 1.014687278 >>>> >>>> I think this is pretty useful, you can get request rates and error >>>> rates out of it. If Prometheus / OpenMetrics had support for Exemplars on >>>> the _count, users could find example traces per HTTP status and URI. >>>> >>>> Fabian >>>> >>>> On Tue, Oct 18, 2022 at 3:05 PM Bjoern Rabenstein <[email protected]> >>>> wrote: >>>> >>>>> On 06.10.22 14:45, 'Fabian Stäber' via Prometheus Developers wrote: >>>>> > >>>>> > Great question from the CNCF Slack: What's the reason why we don't >>>>> allow >>>>> > Exemplars for _count in Summary metrics? >>>>> > >>>>> > What do you think? Any reason why Exemplars don't work in _count in >>>>> > Summaries? Would that be something we could consider supporting? >>>>> >>>>> The _count of a Summary _and_ the _count of a Histogram (both >>>>> conventional as well as the new native ones) is essentially a counter >>>>> within the larger "structured" metric of a Summary/Histogram. >>>>> >>>>> From that perspective, it should have the option of attaching an >>>>> examplar, as a regular Counter has, too. >>>>> >>>>> My speculation why it doesn't in OpenMetrics: >>>>> >>>>> In an OM Histogram, the +Inf bucket fulfills exactly the same function >>>>> as the _count (spec says: "The +Inf bucket counts all requests.") So >>>>> if you would like an examplar on the _count of a Histogram, you can as >>>>> well use an exemplar on the +Inf bucket. >>>>> >>>>> That obviously doesn't help in the case of a Summary, but I guess the >>>>> rationale is that Histograms are generally to be preferred over >>>>> Summaries, and therefore didn't get the thourough treatment when it >>>>> came to exemplars. >>>>> >>>>> >>>>> However, even if you really dislike the precalculated quantiles in >>>>> Summaries, there is still the case of a Summary without quantiles. I >>>>> think adding exemplars to such a Summary is as much needed as adding >>>>> exemplars to any regular Counter. >>>>> >>>>> -- >>>>> Björn Rabenstein >>>>> [PGP-ID] 0x851C3DA17D748D03 >>>>> [email] [email protected] >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/456a8d9a-4db7-4185-96cb-ee6b835373d4n%40googlegroups.com.
