Baunsgaard commented on code in PR #3629:
URL: https://github.com/apache/iceberg-rust/pull/3629#discussion_r4495235421
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -464,6 +483,30 @@ impl PromotePlan {
}
}
+/// parquet-rs pads each null FixedSizeList row with `size` null children;
drop them so a null
+/// list owns no child slots, as in a 3-level LIST and Java. Required elements
reject the pads.
+fn list_from_fixed_size_list(
+ fixed_size_list: &FixedSizeListArray,
+ element_field: &FieldRef,
+) -> Result<ArrayRef> {
+ let Some(nulls) = fixed_size_list.nulls().filter(|n| n.null_count() > 0)
else {
+ return Ok(cast(
+ fixed_size_list,
+ &DataType::List(Arc::clone(element_field)),
+ )?);
+ };
+ let size = fixed_size_list.value_length() as usize;
Review Comment:
Done.
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -464,6 +483,30 @@ impl PromotePlan {
}
}
+/// parquet-rs pads each null FixedSizeList row with `size` null children;
drop them so a null
+/// list owns no child slots, as in a 3-level LIST and Java. Required elements
reject the pads.
+fn list_from_fixed_size_list(
+ fixed_size_list: &FixedSizeListArray,
+ element_field: &FieldRef,
+) -> Result<ArrayRef> {
+ let Some(nulls) = fixed_size_list.nulls().filter(|n| n.null_count() > 0)
else {
+ return Ok(cast(
+ fixed_size_list,
+ &DataType::List(Arc::clone(element_field)),
+ )?);
+ };
+ let size = fixed_size_list.value_length() as usize;
+ let offsets =
+ OffsetBuffer::<i32>::from_lengths(nulls.iter().map(|valid| if valid {
size } else { 0 }));
Review Comment:
Done. Checking `(len - null_count) * value_length` against `i32::MAX` before
the nulls split, so both the `cast` path and `from_lengths` return `Err`
instead of panicking.
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -1972,6 +2026,85 @@ mod test {
assert_eq!(out.as_list::<i32>(), &expected);
}
+ #[test]
+ fn promote_fixed_size_list_file_column_to_list() {
Review Comment:
Done. Added `test_read_fixed_size_list_column_as_list` in `projection.rs`:
writes a FixedSizeList with a null row, reads through `ArrowReader`, and covers
both optional and required elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
