HonahX commented on code in PR #363:
URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1479215719
##########
pyiceberg/table/__init__.py:
##########
@@ -2411,11 +2428,29 @@ def _fetch_existing_manifests() -> List[ManifestFile]:
executor = ExecutorFactory.get_or_create()
- added_manifests = executor.submit(_write_added_manifest)
- delete_manifests = executor.submit(_write_delete_manifest)
- existing_manifests = executor.submit(_fetch_existing_manifests)
+ added_manifests_future = executor.submit(_write_added_manifest)
+ delete_manifests_future = executor.submit(_write_delete_manifest)
+ existing_manifests_future = executor.submit(_fetch_existing_manifests)
+
+ added_manifests = added_manifests_future.result()
+ delete_manifests = delete_manifests_future.result()
+ existing_manifests = existing_manifests_future.result()
+ unmerged_data_manifests = (
+ added_manifests
+ + delete_manifests
+ + [manifest for manifest in existing_manifests if manifest.content
== ManifestContent.DATA]
+ )
+ # TODO: need to re-consider the name here: manifest containing
positional deletes and manifest containing deleted entries
+ unmerged_deletes_manifests = [manifest for manifest in
existing_manifests if manifest.content == ManifestContent.DELETES]
+
+ data_manifest_merge_manager = ManifestMergeManager(
Review Comment:
Thanks for the explanation! Totally agree! I was thinking it might be a good
time to bring `FastAppend` and `MergeAppend` to pyiceberg, making them inherit
from a `_SnapshotProducer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
