zhangfengcdt commented on PR #3437: URL: https://github.com/apache/iceberg-rust/pull/3437#issuecomment-6215992096
> Verified against iceberg-java: `ParquetSchemaUtil.addFallbackIds` assigns ids to top-level fields only, so reading id-less nested fields as absent matches Java. CI is green and the branch merges cleanly with main. I also ran four single-line production mutations against the new and changed tests (details on the test comment below). > > * `get_arrow_projection_mask_fallback` still decodes whole top-level columns, and for a struct column the nested values are now always discarded. Pruning is not free because the struct keeps the file's validity, so at least one leaf per struct must still be read. A tracked follow-up issue would help. > * This turns values that #3255 returned into nulls with no signal, which matches Java. A line in the release notes or the name-mapping docs saying migrated tables need a name mapping for nested fields would help users who hit it. > * Nit: `PromotePlan::Map::entries` is now always `ChildPlan::FromSource`, so the `Add` arm in `apply_struct` is only reachable from structs. Fine to leave, since splitting it would add more code than it removes. @comphead Thanks for the review. Both comments should be addressed now: - Pruning: opened #3480 to track reading only the needed leaves in the fallback projection, keeping one leaf per struct for validity. - Release notes: added a CHANGELOG.md line saying tables migrated without field ids need a name mapping covering nested fields. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
