walterddr commented on PR #10711: URL: https://github.com/apache/pinot/pull/10711#issuecomment-1538771214
after discussion internally we realized several issues. 1. while receiving mailbox is yielding the opChain by returning a no-op block; sending mailbox is not yielding if it is blocked on a sender: say InMemoryMailbox if the receiving side is not polling it out fast enough then sending opChain will block on only 1 particular mailbox 2. putting the push-based and pull-based opChains into the same queue exaggerated the above problem b/c of the sending mailbox block 3. this causes other troubles if other operators can potentially cause the OpChain thread to park somewhere that's not returning a no-op block going back to the drawing board and see what's the best solution. the current solution is to use a cached threadpool for intermediate stage PRO: it immediately unblock the distributed deadlock issue. CON: under high QPS threads start to thrash. (observed 20-30% overhead in QPS degradation) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
