c-thiel commented on PR #2880: URL: https://github.com/apache/iceberg-rust/pull/2880#issuecomment-6585138724
Thanks for the reviews, and sorry for the long gap! I rebased onto main and went through every comment; the replies are inline. The main changes since your last pass: - Returned batches now carry `arrow.parquet.variant`. It's set in `RecordBatchTransformer`, which also matches a variant's `metadata` and `value` by name. They have no field ids, and on current main struct children are matched by id, so most variant reads failed after the rebase. - Projected variants are now validated (`DataInvalid` or `FeatureUnsupported`), and only projected variants are tagged. - Files with neither field ids nor a name mapping still reject variant reads, as before this PR. - Shredded variants stay out of this PR. They're tracked in #3606. While working on this I ran into a few problems in the reader that are unrelated to variants. I opened issues for them and compared each one with iceberg-java: - #3607: equality deletes on uuid columns fail when the delete file has no Parquet UUID annotation. - #3608: field-id detection only looks at the first column, so some files are read by position even though they have ids. - #3609: projection depends on arrow's leaf numbering. #3393 is one way this breaks, and #3467 already fixes that case. @laskoviymishka @blackmwk @dannycjones could you take another look when you have time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
