rdblue commented on issue #8390:
URL: https://github.com/apache/iceberg/issues/8390#issuecomment-1703950561
Just looked into this a bit more. There are test cases where replace table
is failing when a concurrent operation adds a snapshot. The problem is that the
replace table produces a snapshot with the next sequence number after the
metadata it was based on, but the concurrent commit also uses the same sequence
number. Then the commit fails with this error message:
> Cannot add snapshot with sequence number 2 older than last sequence number
2
That's a little inaccurate because it is not strictly older, but the point
is that the table's current sequence number is 2 and the new sequence number is
not newer (greater than 2).
I looked into whether we can fix this by sending a `CommitFailedException`
back to the client because it was out of sync with the current table metadata,
but a replace table transaction doesn't currently recommit operations. For a
normal transaction, a retry results in each operation getting re-applied
(recommitted) on top of the last operation to rebuild metadata. That's
unnecessary for replace transactions because any data is completely independent
of prior versions. All that needs to happen is that the new snapshot is added
and made the current snapshot.
Given that the replace transaction won't rebuild metadata to rewrite the
sequence number, retry isn't helpful. We could update retry to rewrite
metadata, but that isn't really necessary because the new snapshot is
independent. A simpler fix that appears to fix the tests is to update the check
in `TableMetadata.Builder` to allow sequence number reuse if the added
snapshot's parent is null (as in the case of a replace table operation). That's
safe because there are no existing files.
```diff
diff --git a/core/src/main/java/org/apache/iceberg/TableMetadata.java
b/core/src/main/java/org/apache/iceberg/TableMetadata.java
index be510dea20..82953c8385 100644
--- a/core/src/main/java/org/apache/iceberg/TableMetadata.java
+++ b/core/src/main/java/org/apache/iceberg/TableMetadata.java
@@ -1141,7 +1141,7 @@ public class TableMetadata implements Serializable {
snapshot.snapshotId());
ValidationException.check(
- formatVersion == 1 || snapshot.sequenceNumber() >
lastSequenceNumber,
+ formatVersion == 1 || snapshot.sequenceNumber() >
lastSequenceNumber || snapshot.parentId() == null,
"Cannot add snapshot with sequence number %s older than last
sequence number %s",
snapshot.sequenceNumber(),
lastSequenceNumber);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
