yashmayya commented on PR #15263: URL: https://github.com/apache/pinot/pull/15263#issuecomment-2723467849
@ankitsultana the large `IN` clause compilation issue comes from two sources. One is in certain optimization rules like `CoreRules.FILTER_REDUCE_EXPRESSIONS` and `PinotSortExchangeNodeInsertRule` when run after `PinotFilterExpandSearchRule`. This was fixed in https://github.com/apache/pinot/pull/14615 (after the original fix in https://github.com/apache/pinot/pull/13614 was unintentionally undone in https://github.com/apache/pinot/pull/14448). The other source of the issue was in the `SqlToRelConverter` phase when converting a tree of `SqlNode`s to a tree of `RelNode`s which is even before any of our optimization rules can be applied. You can attach a profiler to this test on `master` to see this - https://github.com/apache/pinot/blob/ebfa23669fea2d08a9531f12037f78a0f0a9a2de/pinot-query-planner/src/test/java/org/apache/pinot/query/QueryCompilationTest.java#L490   This issue has been fixed in Calcite `1.39.0`. In the long run though, we need to explore using the default value for `expand` [here](https://github.com/apache/pinot/blob/ebfa23669fea2d08a9531f12037f78a0f0a9a2de/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotRuleUtils.java#L51) since the value `true` we're using has been deprecated. Setting `expand` to `false` keeps `IN` clauses with literals as `IN` clauses when we also set `InSubQueryThreshold` to `0` (i.e., no conversion to a `JOIN` or to an `OR` list of `=` expressions) although it also introduces a bunch of things that Pinot can't currently handle, for instance planning a correlated subquery using a `SCALAR_QUERY` filter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
