gortiz commented on code in PR #14946:
URL: https://github.com/apache/pinot/pull/14946#discussion_r1936947130
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -112,6 +118,7 @@ public MultiStageBrokerRequestHandler(PinotConfiguration
config, String brokerId
CommonConstants.MultiStageQueryRunner.KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN,
CommonConstants.MultiStageQueryRunner.DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN);
_queryThrottler = queryThrottler;
+ _queryCompileExecutor = Executors.newCachedThreadPool(new
NamedThreadFactory("query-compile-executor"));
Review Comment:
My reason for using a fixed number of threads is that optimizing a query is
a purely CPU-bound task. What is the advantage of having more threads than CPUs
in this case? In fact, I would use fewer threads than CPUs.
Imagine a situation where users are running tons of queries that are
expensive to parse. In that case, having a large number of threads running
optimizations would be expensive and make the system unresponsive. In the same
case, using a number of fixed threads smaller than the number of CPUs would
queue up queries, but other parts of the system shouldn't be affected.
Remeber that Jersey can easily spawn hundred of threads because it assumes
most work will be blocking. And IIRC the new MSE query throttling mechanism is
applied after the query is been optimized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
