I am building a demo application to show off Prometheus Exemplars and
OpenTelemetry Tracing abilities with Java / Spring applications. However,
I cannot get exemplars to appear in the metrics HTTP endpoint. I'm using
the following curl command and I get metrics in the OpenMetrics format, but
no exemplars.
$ curl -H 'Accept: application/openmetrics-text; version=1.0.0;
charset=utf-8' http://localhost:8081/metrics
I've got the Prometheus client_java 0.12.0 loaded and I'm using that to
directly instrument. In build.gradle:
dependencies {
...
implementation 'io.prometheus:simpleclient:0.12.0'
implementation 'io.prometheus:simpleclient_hotspot:0.12.0'
implementation 'io.prometheus:simpleclient_httpserver:0.12.0'
implementation 'io.prometheus:simpleclient_tracer_otel:0.12.0'
implementation 'io.prometheus:simpleclient_tracer_otel_agent:0.12.0'
implementation('io.opentelemetry:opentelemetry-api:1.0.0')
implementation('io.opentelemetry:opentelemetry-extension-annotations:1.0.0')
}
I setup my Prometheus metrics in the class constructor:
prometheusErrors = io.prometheus.client.Counter.build()
.namespace("custommetricsdemo")
.name("errors")
.help("Test Prometheus Client Library errors counter")
.withExemplars()
.register();
prometheusTimer = io.prometheus.client.Summary.build()
.namespace("custommetricsdemo")
.name("latency_timer")
.help("Test Prometheus Client Library latency summary")
.quantile(0.5, 0.01)
.quantile(0.95, 0.01)
.register();
prometheusHistogram = io.prometheus.client.Histogram.build()
.namespace("custommetricsdemo")
.name("histogram")
.help("Test Prometheus Client Library latency histogram")
.withExemplars()
.register();
try {
io.prometheus.client.exporter.HTTPServer server
= new io.prometheus.client.exporter.HTTPServer(8081);
} catch (Exception e) {
logger.error("Failed to setup Prometheus HTTP server", e);
}
This code runs a ScheduledTask at a random interval which sleeps for a
random time. Thanks to running with the OpenTelemetry Agent, I can see
trace_id and span_id in the logback logs. That part works well. I can
also query the span IDs in code:
logger.info("{}, {}", span.getSpanContext().getSpanId(),
span.getSpanContext().getTraceId());
I could not get the auto-exemplars to work, so I began to observe them in
code:
prometheusErrors.incWithExemplar("span_id",
span.getSpanContext().getSpanId(), "trace_id",
span.getSpanContext().getTraceId());
I also tested with different exemplar tag names:
prometheusHistogram.observeWithExemplar(sw.getTotalTimeSeconds(),
"span_foo", span.getSpanContext().getSpanId(), "trace_bar",
span.getSpanContext().getTraceId());
However, no exemplars ever appear in the output.
$ curl -H 'Accept: application/openmetrics-text; version=1.0.0;
charset=utf-8' http://localhost:8081/metrics
# TYPE custommetricsdemo_histogram histogram
# HELP custommetricsdemo_histogram Test Prometheus Client Library latency
histogram
custommetricsdemo_histogram_bucket{le="0.005"} 3.0
custommetricsdemo_histogram_bucket{le="0.01"} 3.0
custommetricsdemo_histogram_bucket{le="0.025"} 3.0
custommetricsdemo_histogram_bucket{le="0.05"} 4.0
custommetricsdemo_histogram_bucket{le="0.075"} 4.0
custommetricsdemo_histogram_bucket{le="0.1"} 4.0
custommetricsdemo_histogram_bucket{le="0.25"} 6.0
custommetricsdemo_histogram_bucket{le="0.5"} 9.0
custommetricsdemo_histogram_bucket{le="0.75"} 11.0
custommetricsdemo_histogram_bucket{le="1.0"} 11.0
custommetricsdemo_histogram_bucket{le="2.5"} 11.0
custommetricsdemo_histogram_bucket{le="5.0"} 11.0
custommetricsdemo_histogram_bucket{le="7.5"} 11.0
custommetricsdemo_histogram_bucket{le="10.0"} 11.0
custommetricsdemo_histogram_bucket{le="+Inf"} 11.0
custommetricsdemo_histogram_count 11.0
custommetricsdemo_histogram_sum 2.732210076
custommetricsdemo_histogram_created 1.632231018649E9
What am I missing here? Is something interfering with the Spring Boot
libraries perhaps?
Jack Neely
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/7a09c0bc-a002-4643-a8fa-8cb6a24db87dn%40googlegroups.com.
