Baunsgaard commented on code in PR #3658:
URL: https://github.com/apache/iceberg-rust/pull/3658#discussion_r4550425533
##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -329,6 +331,83 @@ pub(super) fn find_leaf_by_field_id(
})
}
+/// Parquet leaf index ranges of one map's key and value.
+struct MapLeafRanges {
+ key: Range<usize>,
+ value: Range<usize>,
+ value_is_struct: bool,
+}
+
+/// Collects map leaf ranges in post-order, recognising maps the way
parquet-rs does.
+fn collect_map_leaf_ranges(ty: &ParquetType, next_leaf: &mut usize, maps: &mut
Vec<MapLeafRanges>) {
+ if ty.is_primitive() {
+ *next_leaf += 1;
+ return;
+ }
+ let is_map = matches!(
+ ty.get_basic_info().converted_type(),
+ ConvertedType::MAP | ConvertedType::MAP_KEY_VALUE
+ );
+ if let (true, [entries]) = (is_map, ty.get_fields())
+ && !entries.is_primitive()
+ && let [key, value] = entries.get_fields()
+ {
+ let key_start = *next_leaf;
+ collect_map_leaf_ranges(key, next_leaf, maps);
+ let value_start = *next_leaf;
+ collect_map_leaf_ranges(value, next_leaf, maps);
+ maps.push(MapLeafRanges {
+ key: key_start..value_start,
+ value: value_start..*next_leaf,
+ value_is_struct: !value.is_primitive()
+ && value.get_basic_info().converted_type() ==
ConvertedType::NONE,
+ });
+ } else {
+ for field in ty.get_fields() {
+ collect_map_leaf_ranges(field, next_leaf, maps);
+ }
+ }
+}
+
+/// Parquet reads a map only when both its key and value are projected. When
the key is selected
+/// but no value leaf matched the table (#3557), also read the value leaves:
they match no table
+/// field and come back as a struct of NULLs (#3437). Only struct values are
rescued, because a
+/// primitive value is matched by position and would be cast.
+fn add_map_value_leaves(
+ parquet_schema: &SchemaDescriptor,
+ indices: &mut Vec<usize>,
+ type_rejected_leaves: &HashSet<usize>,
+) -> Result<()> {
+ let mut maps = Vec::new();
+ collect_map_leaf_ranges(parquet_schema.root_schema(), &mut 0, &mut maps);
+ if maps.is_empty() {
+ return Ok(());
+ }
+ let mut selected: HashSet<usize> = indices.iter().copied().collect();
+ // Reversed post-order visits an outer map before the maps nested in its
value.
+ for map in maps.iter().rev() {
+ let key_read = map.key.clone().any(|idx| selected.contains(&idx));
+ let value_read = map.value.clone().any(|idx| selected.contains(&idx));
+ if !key_read || value_read || !map.value_is_struct {
+ continue;
+ }
+ let value_leaves: Vec<usize> = map
+ .value
+ .clone()
+ .filter(|idx| !type_rejected_leaves.contains(idx))
+ .collect();
Review Comment:
Confirmed the cast happens, but it doesn't come from the map rescue in this
PR. On upstream/main, a list or map nested inside a struct map value whose
element/key/value field id doesn't match the table **is still cast by
position.**
The presence-leaf logic from #3438 reads that nested leaf, and the
transformer then casts it. I've kept this PR scoped to #3557 and removed the
nested-cast tests, since they would have locked in the current behaviour.
Perhaps we should track a different issue for this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]