shinzoxD opened a new pull request, #24406:
URL: https://github.com/apache/datafusion/pull/24406
## Which issue does this PR close?
Closes #24372
## Rationale for this change
Physical `LambdaVariable` bindings are planned in a combined `input_schema
++ params` index space. `LambdaExpr` then built its outer-batch *capture*
projection from every `Column` **and** `LambdaVariable` index in the body.
That is safe only while the input width stays the same as at plan time.
After a later rewrite moves the expression across a schema boundary (collapsing
consecutive `ProjectionExec`s, or pushing a join filter below
`NestedLoopJoinExec`), a stale param index such as `x@1` can become a real
input column. Evaluation then fails:
```
Field of physical LambdaVariable with index 0 doesn't match batch field
during evaluation Field { x: nullable Int64 } != Field { column2: Int64 }
```
The two issue reproducers were:
```sql
SELECT array_transform(arr, x -> x)
FROM (
SELECT arr
FROM (VALUES ([1, 2], 7)) AS t(arr, padding)
) AS q;
```
and the `array_filter` join-filter pushdown case.
## What changes are included in this PR?
- Own lambda parameters are no longer added to the outer-batch capture
projection.
- They are rebound by name into the `captures ++ used_params` layout that
`LambdaArgument::new` already builds.
- Nested-lambda parameters stay shadowed and are remapped by the inner
`LambdaExpr`.
## Are these changes tested?
- Unit tests in `datafusion/physical-expr/src/expressions/lambda.rs`,
including a regression that a planner-assigned `x@1` is not treated as an
outer-batch capture.
- SQLLogic tests for both #24372 reproducers in `array_transform.slt` and
`array_filter.slt`.
- Existing nested-lambda slt files (`array_transform`, `array_filter`,
`array_any_match`, `array_all`, `array_any_value`, `array_first`) still pass.
## Are there any user-facing changes?
No public API change. Queries that previously failed after a
projection/join-filter rewrite now return the correct result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
