praveenc7 opened a new issue, #15890: URL: https://github.com/apache/pinot/issues/15890
## Issue A recent change (PR [#15312](https://github.com/apache/pinot/pull/15312)) added the [`SendStatsPredicate`](https://github.com/apache/pinot/blame/f0c9638a0eafae7f5119e2551204368600409688/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SendStatsPredicate.java#L54) class to Pinot-server on startup and [registers ZooKeeper watches](https://github.com/apache/pinot/blame/f0c9638a0eafae7f5119e2551204368600409688/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java#L724) on **every** participant-config znode under /pinot-cluster/pinot/CONFIGS/PARTICIPANT and all of its children --- ### What Happens Assume **3 K servers** and **4 K participants** (servers + brokers + controllers + minions): 1. **Each** of the 3 K servers sets watches on **all** 4 K participant-config znodes. 2. When **any** single participant config is updated: - **All 3 K servers** receive a ZooKeeper event. - **Each** server issues ~4 K read requests (one per znode) to rebuild its `onInstanceConfigChange` list. 3. Total watches = 3 K servers × 4 K znodes ≈ **12 million**. 4. This watch/read flood **dramatically increases** ZooKeeper latency. --- ### Impact During our deployment: - **Ingestion lag** spiked - **Broker error rates** increased - **Segment-state updates** were delayed moving from *ideal-state* to *external-view* --- ### Possible next Steps 1. **Revert** PR [#15312](https://github.com/apache/pinot/pull/15312) until this is addressed 2. Or introduce a **configuration** to exclude this feature for SSE Currently, even with different mode options, **all** watches are still being registered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
