hedger9487 opened a new pull request, #3863: URL: https://github.com/apache/iceberg-python/pull/3863
Closes #3148 # Rationale for this change `dynamic_partition_overwrite` previously constructed the delete predicate using only the current partition spec. When a table underwent partition spec evolution (e.g., adding a new partition field), files written under older specs lacked partition values for the newly added field (where the values are `NULL`). As a result, the `_StrictMetricsEvaluator` would fail to match older spec files against the new spec's predicate (e.g., `category = 'A' AND region = 'us'`), silently skipping them and leaving stale data files behind. This PR: 1. Identifies evolved partition fields added across historical partitioned specs. 2. Extends `_build_partition_predicate` to include `IS NULL` for evolved fields (e.g. `category = 'A' AND (region = 'us' OR region IS NULL)`). 3. Adds unit tests covering predicate construction and an end-to-end regression test for dynamic partition overwrite across evolved specs. ## Are these changes tested? Yes: - Added `test_build_partition_predicate_with_evolved_fields` - Added `test_dynamic_partition_overwrite_with_partition_spec_evolution` (regression test verified to fail without the fix when stale spec-0 rows remain, and pass with this fix). - All 12 pre-commit linters and 1,324 tests across `table/`, `expressions/`, and `transforms/` pass cleanly. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
