Hi, Do you have a response to this thread? Has anyone ever encountered the issue?
Regards, Vu On Mon, May 20, 2024 at 2:56 PM Vu Nguyen <[email protected]> wrote: > Hi, > > With endpoints scraping role, the job should scrape POD endpoint that is > up and running. That is what we are expected. > > I think by concept, K8S does not create an endpoint if Pod is in other > phases like Pending, Failed, etc. > > In our environments, Prometheus 2.46.0 on K8S v1.28.2, we currently have > issues: > 1) POD is up and running from `kubectl get pod`, but from Prometheus > discovery page, it shows: > __meta_kubernetes_pod_phase="Pending" > __meta_kubernetes_pod_ready="false" > > 2) The the endpoints job discover POD targets with pod phase=`Pending`. > > Those issues disappear after we restart Prometheus pod. > > I am not sure if 1) that is K8S that does not trigger event after POD > phase changes so Prometheus is not able to refresh its endpoints discovery > or 2) it is a known problem of Prometheus? > > And do you think it is worth to add the following relabeling rule to > endpoints job role? > > - source_labels: [ __meta_kubernetes_pod_phase ] > regex: Pending|Succeeded|Failed|Completed > action: drop > > Thanks, Vu > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c0f97ed7-1421-4c7c-a57d-2d301bb12418n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c0f97ed7-1421-4c7c-a57d-2d301bb12418n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAF7KRax33xF1qcs_9M_6sEV6g_8ojCsyB5jeq0wuVrxbx2VUpg%40mail.gmail.com.
