rlankfo opened a new pull request, #19004: URL: https://github.com/apache/pinot/pull/19004
## Problem The docker-compose demo under `docker/images/pinot` breaks after a `docker compose down && up`. Two independent root causes: 1. **Instance IDs are derived from the container IP.** No host is passed to the start commands, so each service picks up its container IP for its Helix instance ID. On restart the containers get new IPs, so every service registers under a *new* instance ID and existing segment assignments are orphaned on the now-dead instances. Symptom: segments report as unavailable and the ExternalView is empty even though the ideal state looks fine. 2. **The data bind mounts point at container paths the processes don't use.** The controller actually writes to `/tmp/data/PinotController`, the server to `/tmp/data/pinotServerData` + `/tmp/data/pinotSegments`, and Kafka to `/tmp/kraft-combined-logs` — none of which are the mounted volumes. So all data lives on the ephemeral container layer and is lost on `down`. Only the ZooKeeper mount was correct. The combination means even the cluster metadata that *does* survive (in ZooKeeper) points at instances and segment data that no longer exist after a restart. ## Fix - Pass `-controllerHost` / `-brokerHost` / `-serverHost` / `-minionHost` so instance IDs are tied to the stable compose `hostname:` values instead of the container IP. - Align each process's data directory to its mounted volume via `-dataDir` / `-segmentDir` (controller, server) and `KAFKA_LOG_DIRS` (Kafka), so segment data, the deep store, and Kafka logs persist across restarts. ## Verification With the patched compose: 1. Created a realtime `transcript` table, produced records, and force-committed a segment. 2. Confirmed the committed segment and Kafka logs land in the mounted host directories (previously empty). 3. Ran `docker compose down && docker compose up -d` **without** wiping ZooKeeper. Result: all instances came back with identical stable IDs (`Server_pinot-server_8098`, etc.), the committed segment returned to `ONLINE` on the same server, and queries returned the expected rows (`SELECT count(*)` → 12, no exceptions). Before the fix, the same restart left the segment orphaned/`ERROR` and unqueryable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
