I don't see this with v2.45.5, and I'm also concerned about why "app":
"another-testapp" occurs in one of your discoveredLabels.
I suggest you try that, and/or the latests v2.52.1 (you can of course set
up a completely separate instance but point it to the same service
discovery source) and see if you can replicate the issue. Also check the
changelogs and git history to see if there's anything relevant there.
On Friday 31 May 2024 at 20:22:00 UTC+1 Vu Nguyen wrote:
> Hi all,
>
> We have a test code that reads target metadata info and job label name
> from `discoveredLabels` list. That list is included in the response we get
> from '/api/v1/targets' endpoint.
>
> During the test, we noticed that the response from target endpoint is
> inconsistent: the job label sometimes is missing from `discoveredLabels`
> for a few discovered targets.
>
> Below output is what I extracted from our deployment: the first target
> have the job label in its discovered labels but missing in the second
> target, on the same config job.
>
> {
> "status" : "success",
> "data" : {
> "activeTargets" : [ {
> "discoveredLabels" : {
> "__meta_kubernetes_pod_phase" : "Running",
> "__meta_kubernetes_pod_ready" : "true",
> "__meta_kubernetes_pod_uid" :
> "a7b4cce2-1be7-4df9-a032-c7a51bb655db",
> "__metrics_path__" : "/metrics",
> "__scheme__" : "http",
> "__scrape_interval__" : "15s",
> "__scrape_timeout__" : "10s",
> "job" : "kubernetes-pods"
> },
> "labels" : {
> "app" : "testapp",
> "job" : "kubernetes-pods",
> "kubernetes_namespace" : "spider1"
> },
> "health" : "down",
> "scrapeInterval" : "15s",
> "scrapeTimeout" : "10s"
> }, {
> "discoveredLabels" : {
> "__meta_kubernetes_pod_phase" : "Running",
> "__meta_kubernetes_pod_ready" : "true",
> "__meta_kubernetes_pod_uid" :
> "85dfeac6-985d-479e-8459-fc20ae8dcec3",
> "__metrics_path__" : "/metrics",
> "__scheme__" : "http",
> "__scrape_interval__" : "15s",
> "__scrape_timeout__" : "10s",
> "app" : "another-testapp"
> },
> "labels" : {
> "app" : "another-testapp",
> "job" : "kubernetes-pods",
> "kubernetes_namespace" : "spider1"
> },
> "scrapePool" : "kubernetes-pods",
> "health" : "down",
> "scrapeInterval" : "15s",
> "scrapeTimeout" : "10s"
> } ]
> }
> }
>
> Could you please help us understand why we have this inconsistency? Is
> that correct way to get job level value from `discoveredLabels` set?
>
> Thanks,
> Vu
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bfe354cf-a4f6-40d8-b88b-571bf9d1289fn%40googlegroups.com.
