This is an automated email from the ASF dual-hosted git repository.
noob-se7en pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 54fdacede32 Fix docker-compose demo so cluster state and data survive
restarts (#19004)
54fdacede32 is described below
commit 54fdacede325180a0746c8cd07d59499b752757f
Author: Robert Lankford <[email protected]>
AuthorDate: Sat Jul 18 01:28:13 2026 -0700
Fix docker-compose demo so cluster state and data survive restarts (#19004)
The docker-compose demo under docker/images/pinot had two issues that
made it break after a `docker compose down && up`:
1. Pinot instance IDs were derived from the container IP because no host
was passed to the start commands. On restart, containers get new IPs,
so each service registered under a new instance ID and existing segment
assignments were orphaned on now-dead instances (segments reported as
unavailable, empty ExternalView). Pass -controllerHost/-brokerHost/
-serverHost/-minionHost so instance IDs are tied to the stable compose
hostnames instead.
2. The controller, server, and Kafka data bind mounts pointed at container
paths the processes did not actually use (the controller wrote to
/tmp/data/PinotController, the server to /tmp/data/pinotServerData +
/tmp/data/pinotSegments, and Kafka to /tmp/kraft-combined-logs), so all
data lived on the ephemeral container layer and was lost on restart.
Align each process's data directory to its mounted volume via -dataDir/
-segmentDir and KAFKA_LOG_DIRS.
Verified: after loading a realtime table and committing a segment, a
`docker compose down && up` (without wiping ZooKeeper) brings the segment
back ONLINE on the same stable server and queries return the expected rows.
---
docker/images/pinot/docker-compose.yml | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/docker/images/pinot/docker-compose.yml
b/docker/images/pinot/docker-compose.yml
index 9b4a20ed46a..fdad648c4bc 100644
--- a/docker/images/pinot/docker-compose.yml
+++ b/docker/images/pinot/docker-compose.yml
@@ -50,6 +50,8 @@ services:
# Cluster settings
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
CLUSTER_ID: pinot-kafka-cluster-001
+ # Persist Kafka logs to the mounted volume (default is
/tmp/kraft-combined-logs, which is ephemeral)
+ KAFKA_LOG_DIRS: /var/lib/kafka/data
volumes:
- ./pinot-docker-demo/kafka/data:/var/lib/kafka/data
pinot-controller:
@@ -59,7 +61,7 @@ services:
- ./pinot-docker-demo/pinot/controller:/tmp/data/controller
ports:
- "9000:9000"
- command: StartController -zkAddress zookeeper:2181
+ command: StartController -zkAddress zookeeper:2181 -controllerHost
pinot-controller -dataDir /tmp/data/controller
depends_on:
- zookeeper
- kafka
@@ -68,7 +70,7 @@ services:
hostname: pinot-broker
ports:
- "8099:8099"
- command: StartBroker -zkAddress zookeeper:2181
+ command: StartBroker -zkAddress zookeeper:2181 -brokerHost pinot-broker
depends_on:
- zookeeper
- kafka
@@ -80,7 +82,7 @@ services:
- ./pinot-docker-demo/pinot/server:/tmp/data/server
ports:
- "8098:8098"
- command: StartServer -zkAddress zookeeper:2181
+ command: StartServer -zkAddress zookeeper:2181 -serverHost pinot-server
-dataDir /tmp/data/server/data -segmentDir /tmp/data/server/segments
depends_on:
- zookeeper
- kafka
@@ -92,7 +94,7 @@ services:
- ./pinot-docker-demo/pinot/minion:/tmp/data/minion
ports:
- "9514:9514"
- command: StartMinion -zkAddress zookeeper:2181
+ command: StartMinion -zkAddress zookeeper:2181 -minionHost pinot-minion
depends_on:
- zookeeper
- kafka
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
