syun64 commented on code in PR #810:
URL: https://github.com/apache/iceberg-python/pull/810#discussion_r1637326766
##########
pyiceberg/table/__init__.py:
##########
@@ -474,6 +474,26 @@ def add_files(self, file_paths: List[str],
snapshot_properties: Dict[str, str] =
for data_file in data_files:
update_snapshot.append_data_file(data_file)
+ def add_files_overwrite(self, file_paths: List[str], snapshot_properties:
Dict[str, str] = EMPTY_DICT) -> None:
+ """
+ Shorthand API for adding files as data files and overwriting the table.
+
+ Args:
+ file_paths: The list of full file paths to be added as data files
to the table
+ snapshot_properties: Custom properties to be added to the snapshot
summary
+
+ Raises:
+ FileNotFoundError: If the file does not exist.
+ """
+ if self._table.name_mapping() is None:
+ self.set_properties(**{TableProperties.DEFAULT_NAME_MAPPING:
self._table.schema().name_mapping.model_dump_json()})
+ with
self.update_snapshot(snapshot_properties=snapshot_properties).overwrite() as
update_snapshot:
Review Comment:
I think you will still be able to achieve that by setting the filter as
[ALWAYS_TRUE](https://github.com/Fokko/iceberg-python/blob/4cd67ac741b41adf607d2d388a14947cf16d4cdf/pyiceberg/table/__init__.py#L160)
My opinion on this is that when we introduce a new API, we should try to
make it serve a more generic use case (overwriting based on a filter) versus a
very specific use case (overwriting all data), especially given that there's
already a great similar API pattern in `overwrite`. I think mimicking the
inputs there will make the API feel more consistent for other users as well.
Another option we can take to support your specific use case, if the
community decides that we don't want to add a new API for
`add_files_overwrite`, is to just build your own transaction like below:
```
with tbl.transaction() as txn:
txn.delete(delete_filter=ALWAYS_TRUE)
txn.add_files(file_paths=file_paths)
```
This will achieve your desired outcome in a single atomic transaction to the
iceberg table without needing to add a new API to the repository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
