Fokko commented on code in PR #6775:
URL: https://github.com/apache/iceberg/pull/6775#discussion_r1118590084
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -546,11 +568,45 @@ def project_table(
id for id in projected_schema.field_ids if not
isinstance(projected_schema.find_type(id), (MapType, ListType))
}.union(extract_field_ids(bound_row_filter))
+ tasks_data_files: List[FileScanTask] = []
+ tasks_positional_deletes: List[FileScanTask] = []
+ for task in tasks:
Review Comment:
> Scan planning should use an index of delete files to find delete files for
each file task. Then the file task has a data file to read and deletes to
apply. That process should handle sequence number filtering so we don't apply
deletes to files with newer sequence numbers.
I agree, but this isn't relevant for positional deletes.
Looking at the partition pruning, done here:
https://github.com/apache/iceberg/blob/2e45760c8d2b62cc46584cbff28bcb2df422d7ff/core/src/main/java/org/apache/iceberg/DeleteFileIndex.java#L529-L556
I checked and we already prune on a partition level for Python, and I'm not
sure why there is a different path in Java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
