Hi,
With endpoints scraping role, the job should scrape POD endpoint that is up
and running. That is what we are expected.
I think by concept, K8S does not create an endpoint if Pod is in other
phases like Pending, Failed, etc.
In our environments, Prometheus 2.46.0 on K8S v1.28.2, we currently have
issues:
1) POD is up and running from `kubectl get pod`, but from Prometheus
discovery page, it shows:
__meta_kubernetes_pod_phase="Pending"
__meta_kubernetes_pod_ready="false"
2) The the endpoints job discover POD targets with pod phase=`Pending`.
Those issues disappear after we restart Prometheus pod.
I am not sure if 1) that is K8S that does not trigger event after POD phase
changes so Prometheus is not able to refresh its endpoints discovery or 2)
it is a known problem of Prometheus?
And do you think it is worth to add the following relabeling rule to
endpoints job role?
- source_labels: [ __meta_kubernetes_pod_phase ]
regex: Pending|Succeeded|Failed|Completed
action: drop
Thanks, Vu
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/c0f97ed7-1421-4c7c-a57d-2d301bb12418n%40googlegroups.com.
