ooreally opened a new pull request, #4082:
URL: https://github.com/apache/iceberg-python/pull/4082
# Rationale for this change
When an identity partition column is absent from a Parquet file and excluded
from `selected_fields`, filtering on that column can return an empty result
even when its manifest partition value matches the predicate.
For example, a file containing `other_field = ["foo", "bar", "baz"]` with
manifest partition `partition_id = 1` should return all three rows for:
```python
table.scan(row_filter="partition_id = 1",
selected_fields=("other_field",)).to_arrow()
```
Resolve missing identity partition values using the existing union of output
and filter field IDs before translating the Arrow filter.
## Are these changes tested?
The existing identity partition projection tests now cover name-mapped
Parquet files and files written by PyIceberg with embedded field IDs, one and
multiple partitions, mixed partition/data predicates, partial projections, and
zero, empty-string, and null partition values. Six regression cases fail on the
original code; all seven focused cases pass with the fix.
Validation on Python 3.13.11 and PyArrow 25.0.1:
- `make test PYTEST_ARGS="-q --tb=short"`: 4,253 passed, 7 skipped, 126
deselected.
- `make lint`: all repository-wide hooks passed.
- Arrow and expression visitor unit suites: 284 passed, 3 skipped.
- `git diff --check`: passed.
## Are there any user-facing changes?
Yes. Scans that filter on identity partition columns omitted from both the
physical file and the output projection now return the matching rows while
preserving the requested output columns.
AI assistance: OpenAI Codex assisted with investigation, implementation, and
local validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
