zhangfengcdt commented on PR #3437: URL: https://github.com/apache/iceberg-rust/pull/3437#issuecomment-6196358126
> Checked the two points you flagged against iceberg-java (core/parquet/data built from main at 059e7ea94, which includes [apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320)), reading Parquet files with no field ids and no name mapping through `GenericParquetReaders`: > > file / read schema iceberg-java `main` this PR > `s: struct<a: required int>` `Missing required field: a` stored values `required nested field a is absent ...` > `m: map<string, struct<x: int>>` `{k1={x=null}}` stored values `{k1: {x: }}` > `s: struct<a: optional int>`, row 3 `s` null `s` null on every row stored values `{a: }`, `{a: }`, null > So the required-field error and matching map key/value by position both match Java. One difference that isn't in the description: for an optional nested field, Java returns a null struct on every row, because its presence selection only uses columns that have ids and none under `s` do. This PR keeps `s` present on rows 1 and 2 (it reads the whole top-level column), which I think is the more accurate result, but it differs from Java. Thanks @JeonDaehong for the review. For the struct validity: Java's readers disagree here. Spark's reader keeps the struct present with null fields (the Comet repro in #3436 shows [0,[null,null]]), while GenericParquetReaders nulls the struct. This PR follows the Spark result, which is also how iceberg-rust treats a struct whose fields are all absent from a file that has ids. I added a null struct row to promote_struct_without_source_field_ids_reads_null to pin it, and noted the difference in the description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
