Erigara opened a new issue, #2028:
URL: https://github.com/apache/iceberg-python/issues/2028

   ### Apache Iceberg version
   
   0.9.1 (latest release)
   
   ### Please describe the bug 🐞
   
   I've noticed than when including filters on projected fields (not included 
in parquet data files) scan return empty result.
   
   Spark implementation produce correct result here.
   
   I did a bit of digging and it looks like the problem is 
`ds.Scanner.from_fragment` function 
[here](https://github.com/apache/iceberg-python/blob/c4679570b4b9b909c9bd602fbf919385998e1760/pyiceberg/io/pyarrow.py#L1421)
 becuse it produces empty batch iterator.
   
   Second issue might be that batch is first 
[filtred](https://github.com/apache/iceberg-python/blob/c4679570b4b9b909c9bd602fbf919385998e1760/pyiceberg/io/pyarrow.py#L1447)
 and only than projected column is 
[added](https://github.com/apache/iceberg-python/blob/c4679570b4b9b909c9bd602fbf919385998e1760/pyiceberg/io/pyarrow.py#L1465).
   
   I would like to contribute but some directions would be nice.
   
   Here is self-contained script to reproduce the issue:
   
   ```python
   
   #!/usr/bin/env python
   import polars as pl
   from pyiceberg.catalog import load_catalog
   from pyiceberg.partitioning import PartitionSpec, PartitionField
   from pyiceberg import expressions as expr
   from pyiceberg.transforms import IdentityTransform
   from pyiceberg.table import _parquet_files_to_data_files
   from pyiceberg.table.name_mapping import NameMapping, MappedField
   from pyiceberg.io.pyarrow import pyarrow_to_schema
   
   catalog = load_catalog(
       "default",
       **{
           "type": "in-memory",
           "warehouse": "warehouse/",
       }
   )
   
   catalog.create_namespace_if_not_exists("default")
   
   # write filtered data 
   df = 
pl.read_parquet("https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet";)
   df.filter(pl.col("VendorID") == 
1).drop("VendorID").write_parquet("data/VendorID=1_yellow_tripdata_2025-01.parquet")
   
   # create iceberg table
   df = df.to_arrow()
   schema = df.schema
   mapping = NameMapping([MappedField(field_id=i,names=[name]) for i, name in 
enumerate(schema.names, 1)])
   schema = pyarrow_to_schema(schema, mapping)
   table = catalog.create_table_if_not_exists(
       "default.taxi",
       schema=schema,
       partition_spec=PartitionSpec(
           PartitionField(source_id=schema.find_field("VendorID").field_id, 
field_id=schema.find_field("VendorID").field_id, transform=IdentityTransform(), 
name="VendorID"),
       ),
       properties={"schema.name-mapping.default": mapping.model_dump_json()},
   )
   
   # add_files
   files = ["data/VendorID=1_yellow_tripdata_2025-01.parquet"]
   with table.transaction() as tx:
       with tx.update_snapshot().fast_append() as fast_append:
           data_files = _parquet_files_to_data_files(
               table_metadata=tx.table_metadata, file_paths=files, 
io=tx._table.io
           )
           for data_file in data_files:
               # set partition for VendorID
               # current file has only one partition anyway
               # VendorID = 1
               data_file.partition[0] = 1
               fast_append.append_data_file(data_file)
   
   # query with projected field in predicate
   scan = table.scan(row_filter=expr.EqualTo("VendorID", 1))
   print(f"WITH PROJECTED COLUMN LEN: {len(scan.to_arrow())}")
   
   # query without projected field in predicate
   scan = table.scan()
   print(f"WITHOUT PROJECTED COLUM LEN: {len(scan.to_arrow())}")
   ```
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to