mbutrovich commented on PR #3467: URL: https://github.com/apache/iceberg-rust/pull/3467#issuecomment-6958200413
> What do you think about picking up the fix from upstream arrow-rs/parquet-rs instead, and keeping #3393 open until then? I agreed with this earlier, and I'd like to reopen it. arrow 60.1.0 shipped apache/arrow-rs#11657, and main has locked `arrow-schema` 60.1.0 since #3257. The reproduction from #3393 still fails on main at `7cf825fcf`. I ran it with a `ListView` and a `LargeListView` column, each read three ways. The results were the same for both types: | File | Project `id` | Project `lv`, `id` | Project `lv` | | --- | --- | --- | --- | | Field IDs at every level | correct | error | error | | No field IDs, name mapping | correct | `lv` all NULL | error | | No field IDs, fallback IDs | correct | error | error | The upstream fix only repaired the leaf shift, so projecting `id` alone now works. With field IDs, the mask builder now keeps the list view and fails converting it in [`arrow_schema_to_schema`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/arrow/schema.rs#L379) (`Cannot visit Arrow data type: ListView(...)`). The fallback and name-mapping errors come from [`RecordBatchTransformer`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/arrow/record_batch_transformer.rs#L372) (`cannot promote ListView(...) to List(...)`). The name-mapped `lv, id` case returns wrong results without an error. Since #3629 we rewrite `FixedSizeList` hints to `List` in `coerce_fixed_size_lists`, which follows this PR's design. That answers the `FixedSizeList` maintenance concern: both types can go through one rewrite. When I added `ListView` and `LargeListView` arms to that function, all 18 cases read correctly and the `arrow::` tests passed. The rewrite also keeps the reader correct for a downstream crate that resolves `arrow-schema` 60.0.0, which our `arrow-schema = "60.0"` requirement allows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
