zhangfengcdt opened a new pull request, #3437: URL: https://github.com/apache/iceberg-rust/pull/3437
## Which issue does this PR close? - Closes #3436. ## What changes are included in this PR? For a Parquet file with no field ids and no name mapping, only top-level columns get fallback ids. `RecordBatchTransformer` then matched the file's id-less struct fields to the table's by position, so `ArrowReader` returned their stored values. iceberg-java matches nested struct fields by id, finds none, and reads them as absent. Id-less struct fields now go through the existing absent-field path, as in Java's `ReadBuilder.struct`: identity partition constant, then null, or an error for a required field with no initial-default. This removes the positional fallback kept in #3255 until recursive name mapping, which #3384 added. Map key and value now match by position, as in iceberg-java's map readers. Without that, an id-less map would fail on its required key. For files with ids the result is the same, since map entries are always key then value. Lists already matched their element by position. Structs with only some field ids still error. Behavior a reviewer may not expect: - A required nested field in such a file now errors with "required nested field ... is absent", matching Java's "Missing required field". `test_read_parquet_without_field_ids_with_struct` used required fields, so it now uses optional ones. - A list of structs comes back with its structs and null fields. Spark returns an empty list there when it reads no nested leaf; this follows the spec and Java's readers instead. - `get_arrow_projection_mask_fallback` still reads whole top-level columns, so nested values are read and then discarded. Pruning them is left out of this fix. ## Are these changes tested? Yes. Unit tests in `record_batch_transformer.rs` for an id-less struct, an id-less nested struct and an id-less map with a struct value, and the `ArrowReader` test for a file without field ids, which now expects null struct fields. All four fail on main. ## AI Disclosure I used Claude Code to investigate the issue and write the fix and tests. Worth a close look: the claim that Java errors on a required nested field absent from the file, and the decision to match map key and value by position for files with ids too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
