FurkhanShaikh opened a new issue, #4399:
URL: https://github.com/apache/iceberg-python/issues/4399

   ### Apache Iceberg version
   
   0.12.0 (latest release)
   
   ### Please describe the bug 🐞
   
   A scan whose `row_filter` tests a list or map column for null (`IS NULL` / 
`IS NOT NULL`) fails in `to_arrow()` with a `ValueError`, although the same 
table reads fine without the filter or with a filter on a primitive column.
   
   ```python
   import tempfile
   
   import pyarrow as pa
   from pyiceberg.catalog.sql import SqlCatalog
   
   warehouse = tempfile.mkdtemp()
   catalog = SqlCatalog("c", uri=f"sqlite:///{warehouse}/catalog.db", 
warehouse=f"file://{warehouse}")
   catalog.create_namespace("db")
   data = pa.table(
       {
           "id": pa.array([1, 2, 3], pa.int32()),
           "tags": pa.array([["a"], [], None], pa.list_(pa.string())),
           "props": pa.array([[("k", 1)], [], None], pa.map_(pa.string(), 
pa.int32())),
       }
   )
   table = catalog.create_table("db.t", schema=data.schema)
   table.append(data)
   
   print(table.scan(row_filter="id = 3").to_arrow().to_pylist())     # works
   print(table.scan(row_filter="tags is null").to_arrow())           # 
ValueError
   print(table.scan(row_filter="props is not null").to_arrow())      # 
ValueError
   ```
   
   ```
   [{'id': 3, 'tags': None, 'props': None}]
   ValueError: Cannot explicitly project List or Map types, 2:tags of type 
list<string> was selected
   ValueError: Cannot explicitly project List or Map types, 3:props of type 
map<string, int> was selected
   ```
   
   It fails the same way when only `id` is selected 
(`selected_fields=("id",)`), so the column named in the message is not one the 
caller asked to project.
   
   Expected: `tags is null` returns the row with `id = 3`, and `props is not 
null` returns the rows with `id` 1 and 2. Spark returns the matching rows for 
the same predicates on Iceberg tables with list and map columns holding the 
same three kinds of values.
   
   **Where it seems to come from**
   
   The field ids referenced by the filter are added to the projected ids 
(`ArrowScan._projected_field_ids`, 
`...union(extract_field_ids(self._bound_row_filter))`). For a null test on a 
list or map column that is the id of the list/map field itself. 
`_task_to_record_batches` then calls `prune_columns(file_schema, 
projected_field_ids, select_full_types=False)`, and 
`_PruneColumnsVisitor.field` raises for any selected field that is neither 
primitive nor a struct. The same code is on `main` (checked by reading 
`pyiceberg/io/pyarrow.py` and `pyiceberg/schema.py`; I only ran 0.12.0).
   
   Tested with pyiceberg 0.12.0, pyarrow 26.0.1, Python 3.12 on Linux.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to