adriangb opened a new pull request, #26421:
URL: https://github.com/apache/datafusion/pull/26421
## Which issue does this PR close?
None filed. The reproduction is below.
## Rationale for this change
`PushDownLeafProjections` can make a `UNION ALL` return values from the
wrong column, or fail to plan. It happens when the union's inputs take
different pushdown paths.
```sql
CREATE TABLE a (s STRUCT<y VARCHAR>, k VARCHAR) AS VALUES (named_struct('y',
'struct-a'), 'column-a');
CREATE TABLE b (s STRUCT<y VARCHAR>, k VARCHAR) AS VALUES (named_struct('y',
'struct-b'), 'column-b');
SELECT get_field(s, 'y') AS y FROM (
SELECT s, k2 AS k FROM (SELECT s, k AS k2 FROM a LIMIT 10)
UNION ALL
(SELECT s, k FROM b LIMIT 10)
) ORDER BY y;
```
Expected: `struct-a`, `struct-b`. Actual on `main` and on 54.0.0:
`column-b`, `struct-a`.
If the query reads two fields, it fails to plan:
```text
Optimizer rule 'optimize_projections' failed
caused by
Error during planning: UNION field 0 have different type in inputs: left has
Float64 whereas right has Utf8View
```
`Union` matches its inputs by position. The first input is a projection, so
`build_extraction_projection_impl` merges the extraction into it and appends
the alias after the existing expressions. The second input is a `Limit`, so the
rule builds a new projection that puts the alias first. The two inputs then
disagree on where the alias is. The union takes the alias of the second input
from an unrelated column, and `optimize_projections` prunes by that position.
## What changes are included in this PR?
After the rule pushes the extraction into each input of a `Union`, it
projects that input to one layout: the extracted aliases in pair order, then
the needed columns in column order. All inputs then have the same column
positions.
## Are these changes tested?
Yes. There are two new `projection_pushdown.slt` cases: one returned a wrong
value and one failed to plan before the fix. The existing optimizer unit tests
and the full sqllogictest suite pass.
## Are there any user-facing changes?
No. Queries that returned wrong values or failed to plan now return correct
results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
