gortiz commented on code in PR #15445:
URL: https://github.com/apache/pinot/pull/15445#discussion_r2100313412
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/server/QueryServer.java:
##########
@@ -163,37 +169,48 @@ public void submit(Worker.QueryRequest request,
StreamObserver<Worker.QueryRespo
return;
}
- try (QueryThreadContext.CloseableContext queryTlClosable =
QueryThreadContext.openFromRequestMetadata(reqMetadata);
+ try (QueryThreadContext.CloseableContext qTlClosable =
QueryThreadContext.openFromRequestMetadata(reqMetadata);
QueryThreadContext.CloseableContext mseTlCloseable =
MseWorkerThreadContext.open()) {
+
long requestId = QueryThreadContext.getRequestId();
QueryThreadContext.setQueryEngine("mse");
- Tracing.ThreadAccountantOps.setupRunner(Long.toString(requestId),
ThreadExecutionContext.TaskType.MSE);
- ThreadExecutionContext parentContext =
Tracing.getThreadAccountant().getThreadExecutionContext();
- try {
- forEachStage(request,
- (stagePlan, workerMetadata) -> {
+ // Submit the stage for each worker
+ CompletableFuture<List<Object>>[] futures =
forEachStageAndWorker(request,
+ (stagePlan, workerMetadata) -> {
+ Tracing.ThreadAccountantOps.setupRunner(Long.toString(requestId),
ThreadExecutionContext.TaskType.MSE);
+ ThreadExecutionContext parentContext =
Tracing.getThreadAccountant().getThreadExecutionContext();
+
+ try {
_queryRunner.processQuery(workerMetadata, stagePlan,
reqMetadata, parentContext);
Review Comment:
The logic in the code is still complicated to follow. I've applied another
refactor to make it easier to understand. The change has lost some parallelism,
but that is probably for the better.
Specifically, the plans are deserialized sequentially, but the new code is
probably even better than before, given that it has less contention.
Before, when a query was submitted, several threads parsed the stages while
others tried to start the chain executor.
Now, for each submitted query, we only have a single thread deserializing in
a loop while other stages belonging to the same request begin to execute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
