BsoBird commented on code in PR #10623:
URL: https://github.com/apache/iceberg/pull/10623#discussion_r1669863261
##########
core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java:
##########
@@ -159,18 +169,124 @@ public void commit(TableMetadata base, TableMetadata
metadata) {
int nextVersion = (current.first() != null ? current.first() : 0) + 1;
Path finalMetadataFile = metadataFilePath(nextVersion, codec);
FileSystem fs = getFileSystem(tempMetadataFile, conf);
+ boolean versionCommitSuccess = false;
+ boolean useObjectStore =
+ metadata.propertyAsBoolean(
+ TableProperties.OBJECT_STORE_ENABLED,
TableProperties.OBJECT_STORE_ENABLED_DEFAULT);
+ int previousVersionsMax =
+ metadata.propertyAsInt(
+ TableProperties.METADATA_PREVIOUS_VERSIONS_MAX,
+ TableProperties.METADATA_PREVIOUS_VERSIONS_MAX_DEFAULT);
+ // todo:Currently, if the user is using an object store, we assume that he
must be using the
+ // global locking service. But we should support add some other
conditions in future.
+ boolean supportGlobalLocking = useObjectStore;
+ try {
+ tryLock(tempMetadataFile, metadataRoot());
Review Comment:
>Current version of the code is locked only for commits of a certain
version, e.g., multiple clients are committing to version 3 at the same time,
and the lock is in effect.
However, when multiple clients submit different versions at the same time,
they are not locked to each other.
For object storage filesystems, such locking may cause the information in
the versionHintFile to be incorrect.
For normal filesystems, such locking is meaningless, since the OS provides
the same functionality when calling fs.rename. And it also causes the
versionHintFile content to be messed up.
So, we need to lock the metadataRoot.
Of course, if we want to deprecate the lockManager in the whole iceberg, we
can also remove all the code related to the lockManager.It is up to the user to
make sure that this catalog is only used for block filesystems.Examples include
HDFS, LocalFileSystem, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
