gortiz opened a new pull request, #15445: URL: https://github.com/apache/pinot/pull/15445
We have detected production cases where the number of GRPC and query submission threads is too high. There are two reasons for that: 1. We are blocking submissions threads when we shouldn't. 2. We are blocking GPRC threads waiting for submission threads. Submission threads spent time in mainly two places: - Parsing TableConfigs, which we should cache (given they don't change that often). This is CPU time spent - Waiting for pipeline breaker results. Here we are blocking for other threads (calling QueryRunner.processQuery), which are mostly IO based. In the javadoc of the referenced method, it is explicitly said that the caller should not wait for the result or exceptions, but we were doing that. This PR doesn't fix the inefficiency of parsing TableConfigs but instead focuses on fixing the blocks. The main change makes the code uglier by chaining CompletableFutures instead of blocking for results in the caller thread. The original code was easier to read but forced us to use cached thread pools. Ideally, in the future (see https://github.com/apache/pinot/issues/15343), we should be able to use virtual threads, which would let us write the code in the former way without the performance impact it has right now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
