yuhao-su commented on code in PR #2188:
URL: https://github.com/apache/iceberg-rust/pull/2188#discussion_r3532122861
##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -81,7 +81,111 @@ impl ArrowReader {
Self::include_leaf_field_id(&map_type.key_field, field_ids);
Self::include_leaf_field_id(&map_type.value_field, field_ids);
}
+ // Variant is a leaf type for Parquet projection purposes (like a
primitive).
+ Type::Variant(_) => {
+ field_ids.push(field.id);
+ }
+ }
+ }
+
+ /// Recursive DFS over an Arrow `Fields` tree whose leaf numbering matches
+ /// `arrow_schema::Fields::filter_leaves`. For every leaf sitting inside a
+ /// variant column, stores `leaf_idx → variant_field_id` in `out`. A
+ /// "variant column" is any Arrow field whose embedded Parquet field id
+ /// resolves to `Type::Variant` in the Iceberg schema — including variants
+ /// nested inside a struct, list, or map.
Review Comment:
collect_variant_leaves_in_field claims its leaf numbering matches
Fields::filter_leaves, but the traversal is missing three cases that
filter_leaves (arrow-schema 57/58, identical impl) handles:
Dictionary(_, v) is unwrapped to v before matching
RunEndEncoded(_, v) is unwrapped to v's data type
Union children are recursed into, each counted separately
If any of these wraps a nested type (or a Union appears at all), the leaf
indices shift and every variant leaf after it maps to the wrong column —
silently, since the map is only consulted for variant columns.
Today this isn't reachable from Parquet-derived schemas (Parquet can't
represent Union, and arrow-rs can't write dictionary/REE-wrapped nested types),
but the invariant depends on writer capabilities rather than structure. Suggest
either mirroring the three missing branches from try_filter_leaves, or
explicitly erroring on Dictionary/RunEndEncoded/Union before the _ => leaf arm
so a future mismatch fails loudly instead of corrupting the projection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
