qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3096424562
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -165,6 +165,50 @@ def _calculate_added_rows(self, manifests:
list[ManifestFile]) -> int:
added_rows += manifest.added_rows_count
return added_rows
+ def _get_existing_manifests(self) -> list[ManifestFile]:
+ """Filter existing manifests and rewrite those containing deleted data
files."""
+ existing_files: list[ManifestFile] = []
+ # Use manifest pruning if a predicate is set (primarily for Overwrite)
+ manifest_evaluators: dict[int, Callable[[ManifestFile], bool]] =
KeyDefaultDict(self._build_manifest_evaluator)
+
+ if snapshot :=
self._transaction.table_metadata.snapshot_by_name(name=self._target_branch):
+ for manifest_file in snapshot.manifests(io=self._io):
+ # Skip pruning for rewrite operations unless we want to
optimize later
+ if self._operation == Operation.OVERWRITE and not
manifest_evaluators[manifest_file.partition_spec_id](
Review Comment:
Hi @geruh, great catch, I agree that the base class shouldn't be coupled to
specific child operations. I've refactored `_get_existing_manifests` to accept
a `should_use_manifest_pruning` flag so the child classes can explicitly
opt-in. The updates are in 8f1f9b949ebcc83e99ccc80e0b3d9fda050a096d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
