Hi Jack, I don't see any obvious mistakes, but if you provide some code to reproduce the error I'll have a look. There's a unit test for `incWithExemplar()` here, can you see the difference between your code and the test? https://github.com/prometheus/client_java/blob/master/simpleclient/src/test/java/io/prometheus/client/CounterTest.java#L110-L144 If you have a way to reproduce this, ideally open an issue on https://github.com/prometheus/client_java then I'll get back to it.
Fabian On Wed, Sep 22, 2021 at 3:20 PM [email protected] <[email protected]> wrote: > I'm still mystified by this problem. Using "curl -v" I can tell that my > Java app is responding in the OpenMetrics Version 1.0.0 format. Contacts > at Grafana have suggested removing with ".withExemplars()" on the metric > object builders, but I get the same results with or without it. I've also > attempted to remove the Micrometer Prometheus support in this Spring Boot > app in case that was conflicting with the native client_java. Yet, no > difference in output, exemplars are never present. > > I can definitely provide full code for this demo, there's nothing really > special here. But I was hoping that I've just made a simple mistake, but I > fear this may be more complex than that. Any advice with how to get this > demo app working and producing exemplars is appreciated. > > Jack Neely > > On Tuesday, September 21, 2021 at 9:44:07 AM UTC-4 [email protected] > wrote: > >> I am building a demo application to show off Prometheus Exemplars and >> OpenTelemetry Tracing abilities with Java / Spring applications. However, >> I cannot get exemplars to appear in the metrics HTTP endpoint. I'm using >> the following curl command and I get metrics in the OpenMetrics format, but >> no exemplars. >> >> $ curl -H 'Accept: application/openmetrics-text; version=1.0.0; >> charset=utf-8' http://localhost:8081/metrics >> >> I've got the Prometheus client_java 0.12.0 loaded and I'm using that to >> directly instrument. In build.gradle: >> >> dependencies { >> ... >> implementation 'io.prometheus:simpleclient:0.12.0' >> implementation 'io.prometheus:simpleclient_hotspot:0.12.0' >> implementation 'io.prometheus:simpleclient_httpserver:0.12.0' >> implementation 'io.prometheus:simpleclient_tracer_otel:0.12.0' >> implementation 'io.prometheus:simpleclient_tracer_otel_agent:0.12.0' >> implementation('io.opentelemetry:opentelemetry-api:1.0.0') >> >> implementation('io.opentelemetry:opentelemetry-extension-annotations:1.0.0') >> } >> >> I setup my Prometheus metrics in the class constructor: >> >> prometheusErrors = io.prometheus.client.Counter.build() >> .namespace("custommetricsdemo") >> .name("errors") >> .help("Test Prometheus Client Library errors counter") >> .withExemplars() >> .register(); >> prometheusTimer = io.prometheus.client.Summary.build() >> .namespace("custommetricsdemo") >> .name("latency_timer") >> .help("Test Prometheus Client Library latency summary") >> .quantile(0.5, 0.01) >> .quantile(0.95, 0.01) >> .register(); >> prometheusHistogram = io.prometheus.client.Histogram.build() >> .namespace("custommetricsdemo") >> .name("histogram") >> .help("Test Prometheus Client Library latency histogram") >> .withExemplars() >> .register(); >> >> try { >> io.prometheus.client.exporter.HTTPServer server >> = new io.prometheus.client.exporter.HTTPServer(8081); >> } catch (Exception e) { >> logger.error("Failed to setup Prometheus HTTP server", e); >> } >> >> This code runs a ScheduledTask at a random interval which sleeps for a >> random time. Thanks to running with the OpenTelemetry Agent, I can see >> trace_id and span_id in the logback logs. That part works well. I can >> also query the span IDs in code: >> >> logger.info("{}, {}", span.getSpanContext().getSpanId(), >> span.getSpanContext().getTraceId()); >> >> I could not get the auto-exemplars to work, so I began to observe them in >> code: >> >> prometheusErrors.incWithExemplar("span_id", >> span.getSpanContext().getSpanId(), "trace_id", >> span.getSpanContext().getTraceId()); >> >> I also tested with different exemplar tag names: >> >> prometheusHistogram.observeWithExemplar(sw.getTotalTimeSeconds(), >> "span_foo", span.getSpanContext().getSpanId(), "trace_bar", >> span.getSpanContext().getTraceId()); >> >> However, no exemplars ever appear in the output. >> >> $ curl -H 'Accept: application/openmetrics-text; version=1.0.0; >> charset=utf-8' http://localhost:8081/metrics >> # TYPE custommetricsdemo_histogram histogram >> # HELP custommetricsdemo_histogram Test Prometheus Client Library latency >> histogram >> custommetricsdemo_histogram_bucket{le="0.005"} 3.0 >> custommetricsdemo_histogram_bucket{le="0.01"} 3.0 >> custommetricsdemo_histogram_bucket{le="0.025"} 3.0 >> custommetricsdemo_histogram_bucket{le="0.05"} 4.0 >> custommetricsdemo_histogram_bucket{le="0.075"} 4.0 >> custommetricsdemo_histogram_bucket{le="0.1"} 4.0 >> custommetricsdemo_histogram_bucket{le="0.25"} 6.0 >> custommetricsdemo_histogram_bucket{le="0.5"} 9.0 >> custommetricsdemo_histogram_bucket{le="0.75"} 11.0 >> custommetricsdemo_histogram_bucket{le="1.0"} 11.0 >> custommetricsdemo_histogram_bucket{le="2.5"} 11.0 >> custommetricsdemo_histogram_bucket{le="5.0"} 11.0 >> custommetricsdemo_histogram_bucket{le="7.5"} 11.0 >> custommetricsdemo_histogram_bucket{le="10.0"} 11.0 >> custommetricsdemo_histogram_bucket{le="+Inf"} 11.0 >> custommetricsdemo_histogram_count 11.0 >> custommetricsdemo_histogram_sum 2.732210076 >> custommetricsdemo_histogram_created 1.632231018649E9 >> >> What am I missing here? Is something interfering with the Spring Boot >> libraries perhaps? >> >> Jack Neely >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/e3c3c6c6-ffda-4720-8bb6-a8b427b8c005n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-developers/e3c3c6c6-ffda-4720-8bb6-a8b427b8c005n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAPX310jh1Vx2wSUrZTchfwpFxOs8PrJ_zf%3Ddb7UH-DyZ6SaBHg%40mail.gmail.com.
