kosiew commented on code in PR #24699:
URL: https://github.com/apache/datafusion/pull/24699#discussion_r3870189952
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -215,6 +275,17 @@ pub fn cast_column(
target_value_type,
cast_options,
),
+ (
+ DataType::Union(source_fields, source_mode),
+ DataType::Union(target_fields, target_mode),
+ ) => cast_union_column(
+ source_col,
+ source_fields,
+ source_mode,
+ target_fields,
+ target_mode,
+ cast_options,
+ ),
_ => Ok(cast_with_options(source_col, target_type, cast_options)?),
Review Comment:
Could we add a focused `MemoryStream` regression for a contained `Map<...,
Struct>` where the runtime nested field is non-nullable and the declared nested
field is nullable? My understanding is that this should already work because
shapes accepted by `Schema::contains` retain the same Map entry/key/value
structure, and Arrow's generic Map cast should recursively cast the value
Struct and rebuild it with the target fields. A regression here would confirm
that assumption for this branch. The Map-specific adapter in #23914 covers
broader schema evolution, such as nested Struct add/drop/reorder and sliced or
null-backed Map compaction, which I don't think this backport needs to support.
##########
datafusion/physical-plan/src/memory.rs:
##########
@@ -673,4 +684,128 @@ mod lazy_memory_tests {
Review Comment:
The new Union adaptation has good coverage at the helper level, but I think
it would also be useful to exercise it through `MemoryStream`, where we enforce
the producer-side schema invariant. Could we add a small Union case with a
nullable declared child and a non-nullable runtime child, then assert that
`emitted_batch.schema() == stream.schema()`? That would give us coverage for
the Union reconstruction path at the actual integration boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
