kevinjqliu commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2251045032
Here's something interesting. I think the `pyarrow_filter` [here](https://github.com/apache/iceberg-python/blob/9b6503def4f07a70149da07378f4795c3a4b113b/pyiceberg/io/pyarrow.py#L1185-L1188) is wrong. ``` (Pdb) bound_row_filter BoundEqualTo(term=BoundReference(field=NestedField(field_id=13, name='status', field_type=StringType(), required=False), accessor=Accessor(position=5,inner=Accessor(position=0,inner=None))), literal=literal('Employed')) (Pdb) translated_row_filter EqualTo(term=Reference(name='employment.status'), literal=literal('Employed')) (Pdb) bound_file_filter BoundEqualTo(term=BoundReference(field=NestedField(field_id=13, name='status', field_type=StringType(), required=False), accessor=Accessor(position=5,inner=Accessor(position=0,inner=None))), literal=literal('Employed')) (Pdb) pyarrow_filter <pyarrow.compute.Expression (status == "Employed")> ``` The `pyarrow_filter` expression dropped its parent reference. Setting `pyarrow_filter = None` allows pyarrow to scan successfully. ``` (Pdb) pyarrow_filter = None (Pdb) c age contact employment 0 28 {'email': 'al...@example.com'} {'status': 'Employed', 'position': 'Software E... 1 35 {'email': 'b...@example.com'} {'status': 'Self-employed', 'position': 'Consu... ``` ### Code to repro ``` from pyiceberg.catalog.sql import SqlCatalog import pyarrow as pa data = [{'id': 1, 'name': 'Alice', 'age': 28, 'address': {'street': '123 Maple St', 'city': 'Springfield', 'postal_code': '12345'}, 'contact': {'email': 'al...@example.com', 'phone': '555-1234'}, 'employment': {'status': 'Employed', 'position': 'Software Engineer', 'company': {'name': 'Tech Corp', 'location': 'Silicon Valley'}}, 'preferences': {'newsletter': True, 'notifications': {'email': True, 'sms': False}}}, {'id': 2, 'name': 'Bob', 'age': 35, 'address': {'street': '456 Oak St', 'city': 'Metropolis', 'postal_code': '67890'}, 'contact': {'email': 'b...@example.com', 'phone': '555-5678'}, 'employment': {'status': 'Self-employed', 'position': 'Consultant', 'company': {'name': 'Freelance', 'location': 'Remote'}}, 'preferences': {'newsletter': False, 'notifications': {'email': True, 'sms': True}}}] schema = pa.schema([ ('id', pa.int32()), ('name', pa.string()), ('age', pa.int32()), ('address', pa.struct([ ('street', pa.string()), ('city', pa.string()), ('postal_code', pa.string()) ])), ('contact', pa.struct([ ('email', pa.string()), ('phone', pa.string()) ])), ('employment', pa.struct([ pa.field('status', pa.string(), nullable=True), pa.field('position', pa.string(), nullable=True), pa.field('company', pa.struct([ ('name', pa.string()), ('location', pa.string()) ]), nullable=True) ])), ('preferences', pa.struct([ ('newsletter', pa.bool_()), ('notifications', pa.struct([ ('email', pa.bool_()), ('sms', pa.bool_()) ])) ])) ]) pyarrow_table = pa.Table.from_pylist(data, schema=schema) catalog = SqlCatalog("default", **{"uri": "sqlite:///:memory:", "warehouse": "."}) catalog.create_namespace("foo") table = catalog.create_table("foo.bar", schema=schema) table.overwrite(pyarrow_table) table.scan(row_filter="employment.status = 'Employed'", selected_fields=["age", "employment", 'contact.email', 'employment.status']).to_pandas() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
