FrankChen021 commented on issue #13330: URL: https://github.com/apache/druid/issues/13330#issuecomment-6093759384
Disclosure: I’m OpenAI Codex, an AI coding assistant. I investigated this issue at a user’s request and am posting this summary on their behalf. I checked current `master` at `c59ba760d63c395abdc7d863e53d25af9595b89d` (October 10, 2026). The shared-pod discovery metadata problem described above is still present. ### Findings With MiddleManager-based execution, a peon is a child JVM inside the MiddleManager pod. It inherits the pod identity environment variables, so both processes call `patchPod` with the same pod name and namespace. [`K8sDruidNodeAnnouncer.announce()`](https://github.com/apache/druid/blob/c59ba760d63c395abdc7d863e53d25af9595b89d/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java#L84) writes role-specific labels and annotations, but these two label keys are shared across all processes in the pod: - `druidDiscoveryAnnouncement-id-hash` - `druidDiscoveryAnnouncement-cluster-identifier` The resulting sequence is: 1. MiddleManager announces its role, node-ID hash, and cluster identifier. 2. A streaming peon announces on the same pod, overwriting the shared node-ID hash with its own `host:port` hash. 3. When the peon finishes, [`unannounce()`](https://github.com/apache/druid/blob/c59ba760d63c395abdc7d863e53d25af9595b89d/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java#L125) unconditionally removes the shared ID and cluster labels, along with the peon-specific metadata. 4. The MiddleManager role label remains, but the pod no longer matches the [role discovery selector](https://github.com/apache/druid/blob/c59ba760d63c395abdc7d863e53d25af9595b89d/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java#L205), which also requires the cluster identifier. The MiddleManager can remain alive while disappearing from discovery. The streaming-task announcement and cleanup calls are still present in [`SeekableStreamIndexTaskRunner`](https://github.com/apache/druid/blob/c59ba760d63c395abdc7d863e53d25af9595b89d/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java#L977). There is a second collision when multiple peons share a pod: they all use the same `druidDiscoveryAnnouncement-peon` label and `druidNodeInfo-peon` annotation. Making only the ID label role-specific would therefore be insufficient. ### Validation and scope I compiled the announcer source from the master commit above and exercised MiddleManager announce → peon announce → peon unannounce against an in-memory simulated Kubernetes API. The final discovery labels contained only `druidDiscoveryAnnouncement-middleManager=true`, reproducing the reported metadata loss. This was an announcer-level reproduction, not a live Kubernetes cluster test. This finding concerns Kubernetes discovery. The reporter separately confirmed that the ZooKeeper/Curator failure linked in the original description was fixed by #13295; see https://github.com/apache/druid/issues/13330#issuecomment-1321923215. ### Suggested direction Keep pod metadata as the discovery registry, but distinguish pod-level metadata from process-owned discovery records: - Treat the cluster identifier as pod-level metadata; a peon must not remove it on shutdown. - Give each process an instance-specific discovery annotation, keyed by role and instance identity, so multiple peons can coexist without overwriting each other. - Announce/unannounce should add/remove only that instance’s record. - Update discovery readers to enumerate these records, filter by role, and match individual node identities rather than relying on the shared `id-hash` label. This is a design suggestion, not an implemented patch. It needs coordinated announcer/reader changes, a compatibility plan for rolling upgrades, and regression coverage for multiple peons starting and stopping in different orders. Preventing cluster-label deletion alone would address the immediate disappearance symptom but leave the ID and multi-peon collisions unresolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
