Anurag-S1ngh opened a new pull request, #4338: URL: https://github.com/apache/iceberg-python/pull/4338
# Rationale for this change An append plans its data files against the table's partition spec, and each `DataFile` carries the `spec_id` it was planned with. When the commit is retried, the producer rebuilt its snapshot against the refreshed metadata and wrote every added file into a manifest using the *refreshed* default spec: ```python with self.new_manifest_writer(spec=self._transaction.table_metadata.spec()): ``` If another writer changed the partition spec in between, the manifest labelled the planned partitions with fields from the new spec. A file written with `p=7` under spec 1 was then listed under spec 2 as `q=7`, so a filter on `q` pruned it, and when the specs had different numbers of fields the write raised `IndexError`. `_summary()` used the same refreshed default spec, so the partition summaries were mislabelled too. Added files are now grouped by their own `spec_id`, one manifest per spec, and the snapshot summary uses the spec each file was planned with. This also covers files from `add_files` that belong to an older spec than the current default. Deleted files already used their own spec id for both the manifest and the summary. # Are these changes tested? Yes, in `tests/table/test_commit_retry.py`: - `test_retried_append_records_the_spec_it_was_planned_with` changes the spec concurrently, asserts the retried append is found by a filter on the new spec's column, and asserts the new manifest keeps spec 1. - `test_retried_append_with_a_spec_that_has_more_fields` covers the `IndexError` case, where the refreshed spec has more fields than the planned one. Both fail on main and pass with the change. The synthetic `DataFile`s in `test_snapshot_producer_bounded_metadata_access` now pass `spec_id=0`, like real data files do. Full unit suite: `4568 passed, 5 skipped`. `prek run -a` passes. # Are there any user-facing changes? Yes. A retried append after a concurrent partition spec change now records its file under the spec it was written with, so filters on the new spec no longer prune it and the append no longer raises `IndexError` when the field counts differ. Closes #4188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
