comphead opened a new pull request, #3384: URL: https://github.com/apache/iceberg-rust/pull/3384
## Which issue does this PR close? - Closes #3374. This is also the nested-type handling that #1845 asks for. ## What changes are included in this PR? `apply_name_mapping_to_arrow_schema` gave name-mapped field IDs to top-level columns only. Fields inside a struct, list, or map kept no ID, so the projection never selected their leaves. With a top-level column in the projection, the nested column came back NULL. With only nested columns, no leaf matched and the mask fell back to reading every column, which hid the bug. The mapping is now applied at every depth, the way Java's `ApplyNameMapping` does it: - A struct's children are looked up among that struct's own mapped `fields`, so a nested name never matches a top-level entry. - A list element is looked up as `element`, and a map key and value as `key` and `value`, whatever the file calls them. arrow-rs writes `item`, `keys`, and `values`. - `List`, `LargeList`, and `FixedSizeList` are handled alike. - A field the mapping does not cover keeps no ID, and so does everything under it, as before at the top level. - The name lookup compares borrowed strings instead of allocating a `String` per comparison. I used plain recursion over `DataType` instead of the `ArrowSchemaVisitor` that #1845 suggests. The visitor's callbacks do not receive the field they rebuild, so it would need a field stack, as `Int96CoercionVisitor` keeps, plus a stack of mapping scopes. `visit_type` also returns an error for Arrow types it cannot visit. The recursion keeps every type as it is and only adds IDs. Two behavior changes: - A predicate on a nested field of a name-mapped file was evaluated as if the column were absent. `s.x > 15` matched no rows, and `s.x < 15` matched every row with `s` NULL. The column now resolves, so the row filter returns the same ``Leaf column `x` in predicates isn't a root column`` error that files with embedded field IDs already get. - A migrated file whose struct gained a field after it was written (#2617) now fails with `Incorrect number of arrays for StructArray fields` for every projection. `main` already returns that error when only nested columns are projected, and reads the struct as NULL when a top-level column is projected too. #3255 fixes the error. With #3255 merged into this branch, both projections return the file's values with the new field NULL. The two PRs merge without conflicts. The lookup within each level is still a linear scan, as it was at the top level. #3269 changes the same lookup. ## Are these changes tested? Yes. - `test_read_parquet_with_name_mapping_maps_nested_fields` writes a file without field IDs that holds a struct, a list, and a map, then reads it through `ArrowReader` with a name mapping. It checks the values with and without a top-level column in the projection. On `main` it fails with `s` NULL in all 3 rows. - `test_apply_name_mapping_to_large_and_fixed_size_lists` covers `LargeList`, `FixedSizeList`, and a struct inside a list element. A file reads back as those list types only when its embedded Arrow schema asks for them, and arrow-rs's `ArrowWriter` embeds that schema by default. As a spot check, I broke each new branch one at a time: struct children, the list element name, the map key and value names, swapped key and value, a flat name index, a missing parent ID, and the `LargeList` and `FixedSizeList` arms. Each mutation failed one of the two new tests and none of the existing ones. ## AI Disclosure Drafted with Claude Code. The reproducer comes from #3374. The behavior changes above were checked by running reader tests on `main`, on this branch, and on this branch merged with #3255. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
