FurkhanShaikh opened a new issue, #4399:
URL: https://github.com/apache/iceberg-python/issues/4399
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
A scan whose `row_filter` tests a list or map column for null (`IS NULL` /
`IS NOT NULL`) fails in `to_arrow()` with a `ValueError`, although the same
table reads fine without the filter or with a filter on a primitive column.
```python
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("c", uri=f"sqlite:///{warehouse}/catalog.db",
warehouse=f"file://{warehouse}")
catalog.create_namespace("db")
data = pa.table(
{
"id": pa.array([1, 2, 3], pa.int32()),
"tags": pa.array([["a"], [], None], pa.list_(pa.string())),
"props": pa.array([[("k", 1)], [], None], pa.map_(pa.string(),
pa.int32())),
}
)
table = catalog.create_table("db.t", schema=data.schema)
table.append(data)
print(table.scan(row_filter="id = 3").to_arrow().to_pylist()) # works
print(table.scan(row_filter="tags is null").to_arrow()) #
ValueError
print(table.scan(row_filter="props is not null").to_arrow()) #
ValueError
```
```
[{'id': 3, 'tags': None, 'props': None}]
ValueError: Cannot explicitly project List or Map types, 2:tags of type
list<string> was selected
ValueError: Cannot explicitly project List or Map types, 3:props of type
map<string, int> was selected
```
It fails the same way when only `id` is selected
(`selected_fields=("id",)`), so the column named in the message is not one the
caller asked to project.
Expected: `tags is null` returns the row with `id = 3`, and `props is not
null` returns the rows with `id` 1 and 2. Spark returns the matching rows for
the same predicates on Iceberg tables with list and map columns holding the
same three kinds of values.
**Where it seems to come from**
The field ids referenced by the filter are added to the projected ids
(`ArrowScan._projected_field_ids`,
`...union(extract_field_ids(self._bound_row_filter))`). For a null test on a
list or map column that is the id of the list/map field itself.
`_task_to_record_batches` then calls `prune_columns(file_schema,
projected_field_ids, select_full_types=False)`, and
`_PruneColumnsVisitor.field` raises for any selected field that is neither
primitive nor a struct. The same code is on `main` (checked by reading
`pyiceberg/io/pyarrow.py` and `pyiceberg/schema.py`; I only ran 0.12.0).
Tested with pyiceberg 0.12.0, pyarrow 26.0.1, Python 3.12 on Linux.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]