Nemesis123925 commented on PR #11941: URL: https://github.com/apache/pinot/pull/11941#issuecomment-1811542285
I found that a part of the non-determinism of `numSegmentsForThisPlan` happens insde of [reduceOnDataTable](https://github.com/Nemesis123925/pinot/blob/master/pinot-core/src/test/java/org/apache/pinot/queries/ExplainPlanQueriesTest.java#L416), which in my understanding it compare and select the best query generated from the two servers, and add up the `numSegments` of them if the two queries are identical, which happens in [extractUniqueExplainPlansAcrossServers](https://github.com/Nemesis123925/pinot/blob/be74dc1429e4a72c2d51946167c281f039b60165/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/ExplainPlanDataTableReducer.java#L131). Due to the non-determinism inside of the `PROJECT` clause created by `NonDex`, the test which suppose to have the two generates queries identical will differ from the `PROJECT` field, thus be understood as different queries by [extractUniqueExplainPlansAcrossServers](https://github.com/Nemesis123925/pinot/blob/be74dc1429e4a72c2d51946167c281 f039b60165/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/ExplainPlanDataTableReducer.java#L131). Thus the function will select the best one instead of adding their `numSegments` together, and create the nondeternism inside of the `numSegmentsForThisPlan` part. For this part, I think the solution would be sorting the `PROJECT` clause before the two generated queries get into the [extractUniqueExplainPlansAcrossServers](https://github.com/Nemesis123925/pinot/blob/be74dc1429e4a72c2d51946167c281f039b60165/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/ExplainPlanDataTableReducer.java#L131) function, so the function will add the `numSegment` up instead of choising the better one. However this is not the only cause of the non-deternism of the `numSegmentsForThisPlan` part. I found that applying `NonDex` will change the `numSegments` part even before the two generqte queries get into the [extractUniqueExplainPlansAcrossServers](https://github.com/Nemesis123925/pinot/blob/be74dc1429e4a72c2d51946167c281f039b60165/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/ExplainPlanDataTableReducer.java#L131) function. So I will be keep exploring that part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
