viirya opened a new pull request, #7055: URL: https://github.com/apache/datafusion-comet/pull/7055
## Which issue does this PR close? Backport of #6763 to `branch-1.1`, for #6762. ## Rationale for this change `branch-1.1` has the bug that #6763 fixes on `main`. `IN` builds its candidate list once when every candidate returns a scalar for an empty batch. On this branch CASE WHEN and COALESCE evaluate through DataFusion's `CaseExpr`, and IF and `nullif` through `IfExpr`, which delegates to `CaseExpr`. `CaseExpr` returns a scalar for an empty batch when the branch it takes there is a literal, so `id IN (IF(id = 1, NULL, id))` returns NULL for every row. The SQL file test below fails on `branch-1.1` without this change. ## What changes are included in this PR? `git cherry-pick -x` of 21e57333a, adapted because `branch-1.1` does not have `CaseWhenExpr`, which #6763 changed: - A new `SparkCaseExpr` wraps the `CaseExpr` that `create_case_expr` builds, and returns an empty array of the result type for an empty batch. It forwards everything else to `CaseExpr`, including how it displays. - `IfExpr::evaluate` returns an empty array for an empty batch in the same way. - The `spark_in_list` change is the same as on `main`: a candidate is a constant only when every leaf is a `Literal` and no node is volatile. The conflicts were in imports and in tests that only exist on `main`. - `branch-1.1` has no native `Range`, so the SQL file test sets `spark.comet.sparkToColumnar.enabled=true` instead of `spark.comet.exec.range.enabled=true`. - The `adding_a_new_expression.md` change is dropped, since the contributor guide is not published from release branches. `branch-1.1` is the newest release branch. Reading the code, `branch-1.0` (DataFusion 54.1.0) has the same `CaseExpr` behavior. This PR does not include it. ## How are these changes tested? - `SparkCaseExpr` and `IfExpr` unit tests check that an empty batch gives an empty array, and that `with_new_children` keeps the wrapper. `candidates_reading_a_column_remain_dynamic` is cherry-picked from `main`. Disabling each of the three checks fails its test. - `expressions/conditional/in_case_when_candidate.sql`, from #6763, fails on `branch-1.1` without this change and passes with it. The other SQL file tests under `expressions/conditional/` pass. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
