gortiz opened a new pull request, #16075: URL: https://github.com/apache/pinot/pull/16075
Given that all workers were using the same deadtime, any worker could fail. Due to the transmission relay, workers running stages closer to the root have a higher chance of producing the message that ultimately reaches the broker and, therefore, is used to report the error. This PR solves this problem by adding two different deadlines: - Active deadline, the one that users specified so far, which is used whenever we have to wait for some _cpu_ work. For example, a join building the final block. - Passive deadline, a larger timeout than the active deadline. This is the timeout used when we need to wait for another stage, like when a mailbox receive operator waits for more data. The active deadline can be customized in the same way we are used to. The passive timeout can be configured: - With the new query option `extraPassiveTimeoutMs`, which is the extra number of ms we want to wait on passive waiters. This option is not required. - When the query option `extraPassiveTimeoutMs` is not provided, brokers use a default timeout. This default can be changed with the new broker configuration `pinot.broker.passiveTimeoutMs`, which defaults to 100ms To be clear, in normal situations, this PR doesn't increase the normal timeout. It divides the timeout errors in two different types: active and passive and makes active timeouts more likely to reach the broker before the passive ones, which means we end up reporting the actual error message. This also reduces the warning logs printed when we try to send a message and the grpc connection has already timed out. Passive timeouts should only be reached when the network is extremely slow or when the extra passive timeout is set to 0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
