ewoodbury commented on code in PR #3467:
URL: https://github.com/apache/iceberg-rust/pull/3467#discussion_r4306040662
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -655,25 +655,24 @@ impl FileScanTaskReader {
// otherwise, use applyNameMapping() followed by pruneColumns() when
configured, or
// addFallbackIds() followed by pruneColumnsFallback() for
position-based fallback.
// The fast path returns early without materializing an owned schema.
+ let without_list_views = coerce_list_views(arrow_metadata.schema());
+ let file_schema = without_list_views
+ .as_ref()
+ .unwrap_or(arrow_metadata.schema());
let arrow_schema = if missing_field_ids {
let schema = if let Some(name_mapping) = task.name_mapping() {
- apply_name_mapping_to_arrow_schema(
- Arc::clone(arrow_metadata.schema()),
- name_mapping,
- )?
+ apply_name_mapping_to_arrow_schema(Arc::clone(file_schema),
name_mapping)?
Review Comment:
Imported files are probably the most common source of list views.
iceberg-rust's writers emit `List`, and a pyarrow `pa.list_view(...)` file has
no field IDs, so it is read through this name-mapping branch.
This branch doesn't seem to be covered by the new tests yet. If I pass the
uncoerced `arrow_metadata.schema()` here, both tests still pass. Meanwhile, a
pyarrow-written `lv list_view<struct<a, b>>, id` file read with a name mapping
goes back to returning `id` as all NULLs, and to erroring when `lv` is also
projected.
I think it would be worth having `test_read_list_view_columns` also read the
file through a name mapping, for example by writing the same batch without the
field-ID metadata and setting `with_name_mapping` on the task. Not blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
