alamb opened a new issue, #24721:
URL: https://github.com/apache/datafusion/issues/24721
### Describe the bug
Field metadata on a projection's output schema (for example Arrow extension
type metadata, or custom key/value metadata added via a select with an alias)
can silently disappear when the physical optimizer removes or rewrites
`ProjectionExec` nodes.
The user-visible effects are:
- The schema of query results loses field-level metadata that the plan
declared (e.g. extension types like `arrow.uuid` are stripped)
- Expressions that read field metadata return wrong results, because they
see the underlying source field instead of the projected field that carried the
metadata
### To Reproduce
Run a query whose plan contains a projection that only adds field metadata,
e.g.:
```text
ProjectionExec: i@0 AS i
output field metadata = {"event_field": "true"}
DataSourceExec: i
field metadata = {}
```
The optimizer removes the projection (it looks like an identity projection),
and the output schema no longer has the metadata.
Similarly, with a function that reads field metadata (such as a UDF like
`arrow_metadata`):
```text
ProjectionExec: arrow_metadata(i, 'event_field') AS metadata
ProjectionExec: i@0 AS i
output metadata = {"event_field": "true"}
DataSourceExec: i
metadata = {}
```
the projections are collapsed and the function evaluates against the scan
field (no metadata) instead of the inner projection field.
### Expected behavior
Physical plan optimization should not change the output schema of the plan:
field metadata present on projection output fields before optimization should
still be present afterwards, and metadata-reading expressions should return the
same results with and without optimization (`true` rather than `NULL` in the
example above).
### Additional context
Found while upgrading to DataFusion 55:
https://github.com/apache/datafusion/issues/24462#issuecomment-5437241594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
