gmhelmold opened a new pull request, #23485:
URL: https://github.com/apache/datafusion/pull/23485
## Which issue does this PR close?
Closes #23434.
## Rationale for this change
An aggregate over a pure cross join of two tables that carry the *same*
schema-metadata key with different values fails at planning time:
```
Internal error: Physical input schema should be the same as the one
converted from
logical input schema. Differences:
- schema metadata differs: (physical) {"pandas": "RIGHT-SCHEMA"}
vs (logical) {"pandas": "LEFT-SCHEMA"}
```
The logical plan builds a cross join as `Join { join_type: Inner, .. }`
(`LogicalPlanBuilder::cross_join`), so its schema comes from
`build_join_schema`, which merges input metadata **left-wins**.
`CrossJoinExec::new`, however, merged metadata inline with an `extend`, which
is **right-wins**. The two schemas therefore disagree on any conflicting
metadata key, and the physical-vs-logical schema check rejects the plan.
`CrossJoinExec` is the only join operator that still hand-rolled its output
schema. `HashJoinExec`, `NestedLoopJoinExec`, `SortMergeJoinExec`,
`SymmetricHashJoinExec` and the piecewise merge join all already build their
schema via `build_join_schema`. This aligns cross join with that precedent, in
the same family as the union-metadata consistency fix #21127.
## What changes are included in this PR?
- `CrossJoinExec::new` now derives its output schema from
`build_join_schema(&left.schema(), &right.schema(), &JoinType::Inner)` instead
of the inline field-chain + right-wins metadata `extend`. The field list
(left-then-right, no induced nullability — Inner sets `force_nullable = false`)
is unchanged; only the metadata-merge order changes to left-wins, matching the
logical schema.
- A unit test (`test_cross_join_metadata`) mirroring
`joins::utils::test_join_metadata`.
- An end-to-end regression test
(`cross_join_with_conflicting_schema_metadata`) reproducing the exact failure
from #23434.
## Are these changes tested?
Yes. Both tests fail on `main` with the errors quoted above and pass with
the fix. The e2e's inputs (2-row left, 1-row right) also drive the
`JoinSelection` swap path, exercising `swap_inputs`.
## Are there any user-facing changes?
Yes — a bug fix. The output schema **metadata** of a cross join now follows
left-wins on conflicting keys (consistent with the logical plan and with every
other join operator). Field names, types, nullability and column order are
unchanged. Aggregations over cross joins of metadata-carrying tables that
previously errored now plan and execute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
