yupeng9 opened a new pull request #5631: URL: https://github.com/apache/incubator-pinot/pull/5631
## Description Add a constraint to the message queue for all instances in Helix, with a large default value of 100000. This throttles the total number of messages to avoid the sudden surge of zk nodes, which could be larger than the zookeeper buffer size. Today, when the buffer overflows, the controller and server run into exception and cannot proceed with any state change, until the zk nodes are trimmed manually.  The diagram shows how this constraint works. Without constraint, the first spike of ~3k messages within 1 minute is from the state transition messages generated under this instance after restarting a server. With the constraint, the zk node increase is capped at 1k for that instance. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) * [ ] No Does this PR fix a zero-downtime upgrade introduced earlier? * [ ] No Does this PR otherwise need attention when creating release notes? Things to consider: * [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes) ## Release Notes - New configuration options A new configuration `pinot.helix.instance.messages.max` is added for the constraint, with default value of 100k ## Documentation If you have introduced a new feature or configuration, please add it to the documentation as well. See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
