BsoBird commented on code in PR #9546:
URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650772614
##########
core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java:
##########
@@ -159,18 +160,30 @@ public void commit(TableMetadata base, TableMetadata
metadata) {
int nextVersion = (current.first() != null ? current.first() : 0) + 1;
Path finalMetadataFile = metadataFilePath(nextVersion, codec);
FileSystem fs = getFileSystem(tempMetadataFile, conf);
-
- // this rename operation is the atomic commit operation
- renameToFinal(fs, tempMetadataFile, finalMetadataFile, nextVersion);
-
- LOG.info("Committed a new metadata file {}", finalMetadataFile);
-
- // update the best-effort version pointer
- writeVersionHint(nextVersion);
-
- deleteRemovedMetadataFiles(base, metadata);
-
- this.shouldRefresh = true;
+ boolean versionCommitSuccess = false;
+ try {
+ fs.delete(versionHintFile(), false /* recursive delete*/);
Review Comment:
> That is correct, but it is also highly coupled to the Iceberg project
since Iceberg optimizes for object stores. Object stores are terrible at
listing files since they are often paged responses. These are both slow and
costly, therefore the pointer in the `versionHintFile` makes this much faster.
@Fokko
For object storage, since the current implementation of fileSystemCatalog is
based on the assumption that the file system supports atomic operations, the
current implementation can not be used for object storage, either we provide
another implementation, or we use some CNCF's middleware to proxy, which
usually can provide atomic operation features. Example: cubefs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
