walterddr commented on PR #10711: URL: https://github.com/apache/pinot/pull/10711#issuecomment-1533459715
putting it back to draft. there's a fairness issue with this approach. 1. the leaf-stage execution is non-yielding thus it will always be in the ready queue upon register; 2. the intermediate-stage execution is yielding thus they will be yielding at register unless data is already available for this stage; once yield it will be put to the back of the queue 3. each time intermediate-stage need to fight for round-robin scheduler to clear all the previous leaf-stage executions coming in before intermediate stage can execute. This creates a fairness issue b/c not all threads are executing [collaborative multithreaded](https://en.wikipedia.org/wiki/Cooperative_multitasking) -- thus the "bad guys" always gets the priority. possible redesign - implement the PiplineBreakerOperator in this PR so that PiplineBreakerOperator will actually yield leaf-stage operator until the leaf-stage execution finishes - this will also make that `serverExecutor.execute()` to be run on QueryRunner threadpool. which is different from the QueryWorker (for leaf) and QueryIntermediate (for intermediate stage) - use 2 threadpool for scheduler service - use priority queue CC @xiangfu0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
