Dandandan opened a new pull request, #24084:
URL: https://github.com/apache/datafusion/pull/24084
## Which issue does this PR close?
Finishes the work started in #23082 (`HashJoinExec`, `NestedLoopJoinExec`)
and #21885 (`FilterExec`) — `MemoryScanExec` was the last physical plan node
left with an ambiguous projection.
## Rationale for this change
`MemorySourceConfig`'s projection is `Option<Vec<usize>>` but is encoded as
a bare `repeated uint32`:
```rust
// encode
let proto_projection =
source_conf.projection().as_ref().map_or_else(Vec::new, |v| ...);
// decode
let projection = if !scan.projection.is_empty() { Some(...) } else { None };
```
So `Some(vec![])` — project away every column — serializes exactly like
`None` and always decodes back as `None`, and the decoded scan reports every
column again.
That silently changes a plan's output schema mid-flight. In a distributed
engine it also breaks the parent: a `UnionExec` over a zero-column scan and a
zero-column sibling fails on the worker as soon as the plan is decoded, because
`UnionExec::try_new` validates field counts:
```
UnionExec/InterleaveExec requires all inputs to have the same number of
fields.
Input 0 has 0 fields, but input 1 has 2 fields
```
We hit exactly this on the join side in production (a `count`-style query
over a union), which is what #23082 fixed; the memory-scan path has the same
defect.
## What changes are included in this PR?
Encode an empty projection with the single-element `[u32::MAX]` sentinel —
the same convention `HashJoinExec` and `NestedLoopJoinExec` already use — and
decode it back to `Some(vec![])`. Every other state is unchanged on the wire,
so `None` and non-empty projections round-trip exactly as before.
## Are these changes tested?
Yes — `roundtrip_memory_source_projections` covers `None`, `Some(vec![])`
and a partial projection. Without the fix the `Some(vec![])` case fails with
`projection: Some([])` vs `projection: None` and a schema that grows from `[]`
back to every column.
(The 6 pre-existing `roundtrip_parquet_*` / `roundtrip_empty_projection` /
`roundtrip_physical_plan_node` failures in my environment are unrelated — they
fail identically on unmodified `main`. Passing count goes 205 → 206.)
## Are there any user-facing changes?
No API change. The wire format changes only for the
previously-unrepresentable `Some(vec![])` case, matching what the joins already
do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
