yashmayya opened a new pull request, #15129: URL: https://github.com/apache/pinot/pull/15129
- After https://github.com/apache/pinot/pull/15071 was merged, we noticed that the quickstarts in PR / commit builds became extremely flaky. - Multi-stage queries like `SELECT a.playerName, a.teamID, b.teamName FROM baseballStats_OFFLINE AS a JOIN dimBaseballTeams_OFFLINE AS b ON a.teamID = b.teamID LIMIT 10` were frequently failing with errors like: ``` Processed requestId=-1161140471995497472,table=dimBaseballTeams_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=1/1/1/-1/0/0/0/0/0,schedulerWaitMs=-1,reqDeserMs=-1,totalExecMs=0,resSerMs=-1,totalTimeMs=-1,minConsumingFreshnessMs=-1,broker=unknown,numDocsScanned=51,scanInFilter=0,scanPostFilter=102,sched=MultistageEngine,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0 Processed requestId=-1161140471995497728,table=baseballStats_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=1/0/0/-1/0/0/0/0/0,schedulerWaitMs=-1,reqDeserMs=-1,totalExecMs=4,resSerMs=-1,totalTimeMs=-1,minConsumingFreshnessMs=-1,broker=unknown,numDocsScanned=0,scanInFilter=0,scanPostFilter=0,sched=MultistageEngine,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0 (OpChain{645404404000000097_0_3}): Completed erroneously null {503=QueryCancellationError: org.apache.pinot.spi.exception.QueryCancelledException: Cancelled while streaming results at org.apache.pinot.core.operator.streaming.StreamingInstanceResponseOperator.getNextBlock(StreamingInstanceResponseOperator.java:93) at org.apache.pinot.core.operator.streaming.StreamingInstanceResponseOperator.getNextBlock(StreamingInstanceResponseOperator.java:45) at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:48) at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:57) ... Caused by: org.apache.pinot.spi.exception.EarlyTerminationException: Interrupted while streaming results blocks at org.apache.pinot.core.operator.streaming.BaseStreamingCombineOperator.getNextBlock(BaseStreamingCombineOperator.java:103) at org.apache.pinot.core.operator.streaming.BaseStreamingCombineOperator.getNextBlock(BaseStreamingCombineOperator.java:46) at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:48) at org.apache.pinot.core.operator.streaming.StreamingInstanceResponseOperator.getCombinedResults(StreamingInstanceResponseOperator.java:106) ... Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1770) at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435) at org.apache.pinot.core.operator.streaming.BaseStreamingCombineOperator.getNextBlock(BaseStreamingCombineOperator.java:82) ... 18 more} ==[RECEIVE]== Error block found from : 645404404000000097_0_2 in mailbox 645404404000000097|3|0|2|0 (OpChain{645404404000000097_0_2}): Completed erroneously null {503=QueryCancellationError: org.apache.pinot.spi.exception.QueryCancelledException: Cancelled while streaming results at org.apache.pinot.core.operator.streaming.StreamingInstanceResponseOperator.getNextBlock(StreamingInstanceResponseOperator.java:93) at org.apache.pinot.core.operator.streaming.StreamingInstanceResponseOperator.getNextBlock(StreamingInstanceResponseOperator.java:45) at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:48) at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:57) ... Caused by: org.apache.pinot.spi.exception.EarlyTerminationException: Interrupted while streaming results blocks at org.apache.pinot.core.operator.streaming.BaseStreamingCombineOperator.getNextBlock(BaseStreamingCombineOperator.java:103) at org.apache.pinot.core.operator.streaming.BaseStreamingCombineOperator.getNextBlock(BaseStreamingCombineOperator.java:46) at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:48) at org.apache.pinot.core.operator.streaming.StreamingInstanceResponseOperator.getCombinedResults(StreamingInstanceResponseOperator.java:106) ... Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1770) at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435) at org.apache.pinot.core.operator.streaming.BaseStreamingCombineOperator.getNextBlock(BaseStreamingCombineOperator.java:82) ... 18 more} ==[RECEIVE]== Error block found from : 645404404000000097_0_1 in mailbox 645404404000000097|2|0|1|0 ``` - The reason for this is that in the above query, the `SortOperator` will send an early termination signal to its children after it receives more than 10 rows and this will propagate down to the leaf stage operators through the join node. With the new changes, the leaf stage operator will actually try to cancel the execution of the SSE combine operator which is indeed the right thing to do. - However, one small issue was that the leaf stage operator was checking for early termination towards the end of the `getNextBlock` method so it's possible for the interruption of the leaf stage combine operator to propagate upwards as an early termination exception (see [here](https://github.com/apache/pinot/blob/0169021a55a60430676dcae0d34dc1dd413aa031/pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/BaseStreamingCombineOperator.java#L102-L103)). This is a race condition that occurs quite frequently (and can be reproduced locally via the quickstart) depending on whether the early termination signal / interruption reaches the streaming combine operator first or the operator finishes streaming all the results first. - The fix is simple - if the leaf stage operator has been early terminated, we always return a metadata block. This is also what the parent operators would expect ([example](https://github.com/apache/pinot/blob/0169021a55a60430676dcae0d34dc1dd413aa031/pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java#L176-L177)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
