zhuqi-lucas opened a new issue, #21826:
URL: https://github.com/apache/datafusion/issues/21826
## Describe the bug
`InterleaveExec::with_new_children` uses
`assert_or_internal_err!(can_interleave(children.iter()))` which panics when
children's partitioning diverges after optimizer rewrites. This can happen when
multiple physical optimizer passes modify the plan between when
`InterleaveExec` is created and when `with_new_children` is called.
## To Reproduce
This occurs in practice when:
1. `EnforceDistribution` creates `InterleaveExec` for a `UnionExec` whose
children all have matching Hash partitioning
2. Subsequent optimizer passes (e.g. `JoinSelection`, additional
`EnforceDistribution`/`EnforceSorting` passes) modify children's plan
structure, changing their output partitioning
3. `with_new_children` is called on the `InterleaveExec` with the modified
children → assertion panic
We hit this with a complex UNION ALL query (5 subqueries with JOINs, each
reading from different table types — partitioned, single-file, MVs). The
materialization task panics every time:
```
join_selection
caused by
Internal error: Assertion failed: can_interleave(children.iter()): Can not
create InterleaveExec: new children can not be interleaved.
```
## Expected behavior
`InterleaveExec::with_new_children` should gracefully fall back to
`UnionExec` when children are no longer interleavable, instead of panicking.
The existing comment in the code already acknowledges this possibility:
```rust
// New children are no longer interleavable, which might be a bug of
optimization rewrite.
```
Correctness is preserved when falling back to `UnionExec` —
`EnforceDistribution` will add `RepartitionExec` as needed. Only the interleave
optimization is lost.
## Proposed fix
```rust
fn with_new_children(
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
if can_interleave(children.iter()) {
Ok(Arc::new(InterleaveExec::try_new(children)?))
} else {
UnionExec::try_new(children)
}
}
```
Will submit a PR with the fix and unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
