sdf-jkl opened a new issue, #3706:
URL: https://github.com/apache/iceberg-rust/issues/3706

   ### Apache Iceberg Rust version
   
   Upstream `main` at `4be78085219af9f1c2093bfcea66ff3d847c7613`; locked 
Arrow/Parquet 60.1.0.
   
   ### Describe the bug
   
   After replacing a struct's only Variant child with a new optional field, 
reading an older Parquet file alongside a sibling column returns NULL for the 
entire struct, even where that struct was present.
   
   This is a remaining case of #3378 after #3438. 
[`PresenceWalk::visit`](https://github.com/apache/iceberg-rust/blob/4be78085219af9f1c2093bfcea66ff3d847c7613/crates/iceberg/src/arrow/reader/projection.rs#L674-L755)
 discards candidates without field IDs. A valid Variant has an ID on its group, 
but its `metadata` and `value` leaves have no IDs, so neither can preserve the 
enclosing struct's presence. With another column selected, the projection is 
nonempty and the struct is omitted; batch conversion then fills it with NULL.
   
   Related: #3685. #3609 / #3624 address leaf numbering and absent-field 
projection; this case also needs presence selection to handle a Variant's 
ID-less storage leaves.
   
   ### To Reproduce
   
   1. Write three rows with file schema `1: id int`, `2: s struct<3: old 
variant>`. Every `s` is present.
   2. Read through the evolved schema `1: id int`, `2: s struct<4: new int?>`.
   3. Project both `id` and `s`.
   
   Actual: `s = [NULL, NULL, NULL]`.
   Expected: `s = [{new: NULL}, {new: NULL}, {new: NULL}]`.
   
   The following test uses the existing Variant fixtures in 
`crates/iceberg/src/arrow/reader/projection.rs`; add it to that file's `tests` 
module:
   
   ```rust
   #[tokio::test]
   async fn review_variant_dropped_child_preserves_parent_presence() {
       use arrow_array::{Int32Array, StructArray};
       let dir = TempDir::new().unwrap();
       let children = Fields::from(vec![variant_arrow_field("old", 3)]);
       let source = Arc::new(ArrowSchema::new(vec![
           Field::new("id", DataType::Int32, 
false).with_metadata(field_id_meta(1)),
           Field::new("s", DataType::Struct(children.clone()), 
true).with_metadata(field_id_meta(2)),
       ]));
       let path = write_variant_parquet(source, vec![
           Arc::new(Int32Array::from(vec![1,2,3])) as ArrayRef,
           Arc::new(StructArray::new(children, vec![variant_struct_array()], 
None)) as ArrayRef,
       ], dir.path().to_str().unwrap());
       let target = Arc::new(Schema::builder().with_fields(vec![
           NestedField::required(1, "id", 
Type::Primitive(PrimitiveType::Int)).into(),
           NestedField::optional(2, "s", Type::Struct(StructType::new(vec![
               NestedField::optional(4, "new", 
Type::Primitive(PrimitiveType::Int)).into(),
           ]))).into(),
       ]).build().unwrap());
       let batches = read_variant(target, &path, vec![1,2]).await;
       assert_eq!(batches[0].column(1).null_count(), 0,
           "present structs must survive replacement of their Variant child");
       assert_eq!(batches[0].column(1).as_struct().column(0).null_count(), 3);
   }
   ```
   
   Run `cargo test -p iceberg 
review_variant_dropped_child_preserves_parent_presence --lib`.
   
   The parent-null-count assertion fails with actual `3`, expected `0`.
   
   ### Expected behavior
   
   Preserve the file's parent struct validity while supplying NULL for the 
newly added optional child. This must also work when the available physical 
leaves under the parent are a Variant's ID-less storage fields.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to