ewoodbury commented on code in PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438#discussion_r4544304133


##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -395,6 +417,118 @@ fn parquet_field_id(field: &Field) -> Option<i32> {
         .and_then(|field_id| i32::from_str(field_id).ok())
 }
 
+struct PresenceCandidate {
+    repetition: u32,
+    /// Leaf indexes to read: one leaf, or a whole map.
+    leaves: Range<usize>,
+}
+
+/// One file leaf under each projected struct that no selected leaf covers, so 
the
+/// struct keeps its definition level. Like Java's `PresenceColumnSelector`, 
inner
+/// structs go first, and the leaf with the lowest repetition level wins, then 
the first.
+fn presence_leaves(
+    fields: &Fields,
+    projected_structs: &HashSet<i32>,
+    leaf_field_ids: &HashSet<i32>,
+) -> Vec<PresenceCandidate> {
+    if projected_structs.is_empty() {
+        return Vec::new();
+    }
+    let mut walk = PresenceWalk {
+        projected_structs,
+        leaf_field_ids,
+        index: 0,
+        chosen: Vec::new(),
+    };
+    for field in fields {
+        walk.visit(field, 0, false);
+    }
+    walk.chosen
+}
+
+struct PresenceWalk<'a> {
+    projected_structs: &'a HashSet<i32>,
+    leaf_field_ids: &'a HashSet<i32>,
+    index: usize,
+    chosen: Vec<PresenceCandidate>,
+}
+
+impl PresenceWalk<'_> {
+    /// Returns whether a selected leaf is under `field`, and if not, the best 
presence
+    /// candidate. A candidate also goes through the projection's type check, 
so what it
+    /// reads must convert and every field on its path needs a field id. 
Candidates are
+    /// only built under a projected struct, since nothing else can use them.
+    fn visit(
+        &mut self,
+        field: &Field,
+        repetition: u32,
+        in_projected: bool,
+    ) -> (bool, Option<PresenceCandidate>) {
+        // Unwrap once, like `Fields::filter_leaves`, to count leaves the same 
way.
+        let data_type = match field.data_type() {

Review Comment:
   Added, and this turned out to actually catch a new bug: since Arrow 60 bump 
([merged yesterday](https://github.com/apache/iceberg-rust/pull/3257)), 
filter_leaves counts the columns inside a ListView, but the walk counted the 
whole ListView as one column so any column after it got a wrong index. 
   
   Updated so this walk now counts the same way Arrow does, and never picks a 
column inside a ListView, since Iceberg can't read those. The test covers that, 
including a struct nested inside a ListView.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to