satwik-pachigolla opened a new pull request, #16080: URL: https://github.com/apache/pinot/pull/16080
### Summary Making existing functionality to control multistage thread limits at the broker and server that is configurable with cluster configs now configurable with broker and server configs. This adds support for thread limiting heterogeneous infra rather than a single value to apply across the entire cluster. The previously existing cluster configs are: - pinot.beta.multistage.engine.max.server.query.threads (broker limit) - pinot.beta.multistage.engine.max.server.query.threads.hardlimit.factor (server limit) and we are now adding: - pinot.broker.mse.max.server.query.threads (broker limit) - pinot.server.query.executor.mse.max.execution.threads (server limit) ### Comptability This is backwards compatible since this is an opt-in feature. Limiting is only in effect when one of the configs is enabled. If both configs are enabled, the lower one is chosen. This is not forwards compatible once opted-in to using either of the new configs. ### Testing 1. Set brokers and server side limits to 100 using the new configs 2. Load tested 3. Observed expected limits being hit  Broker metric limited to under 222 (100 config * 20 servers / 9 brokers). Limits at ~170 since any further queries we were sending would cross the 222 limit.  Server JVM thread count increases by ~130. With the extra 30% accounted for by thread overhead apart from just the threads performing query execution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
