gedaliakliger opened a new pull request, #4400:
URL: https://github.com/apache/iceberg-python/pull/4400

   Closes #4399
   
   # Rationale for this change
   
   A scan whose `row_filter` tests a list or map column for null (`tags is 
null`, `props is not null`) fails in `to_arrow()` with `ValueError: Cannot 
explicitly project List or Map types`.
   
   `ArrowScan._projected_field_ids` leaves list and map IDs out of the 
projected schema's IDs, because `prune_columns` rebuilds those columns from 
their nested fields, but it adds the filter's field IDs as they are. A null 
test on a list or map column adds the column's own ID, and `prune_columns` 
rejects it. This change projects the nested fields of such a column instead, 
the same way a selected list or map column is read.
   
   ## Are these changes tested?
   
   Yes. `test_projection_null_filter_on_list_and_map` in 
`tests/io/test_pyarrow.py` runs `IsNull` and `NotNull` on a list, a map and a 
list of structs, with the column selected and with only `id` selected. It fails 
on `main` with the `ValueError` above and passes with this change.
   
   Ran locally: `pytest tests/io/test_pyarrow.py -m unit`, and the `ruff`, 
`ruff-format` and `mypy` hooks on the two changed files. The rest of the suite 
and the integration tests are left to CI.
   
   ## Are there any user-facing changes?
   
   `IS NULL` and `IS NOT NULL` filters on list and map columns return the 
matching rows instead of raising.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to