Fokko commented on code in PR #4392:
URL: https://github.com/apache/iceberg-python/pull/4392#discussion_r4602394200


##########
pyiceberg/table/update/snapshot.py:
##########
@@ -690,8 +691,39 @@ def _refresh_for_retry(self) -> None:
 
 
 class _FastAppendFiles(_SnapshotProducer["_FastAppendFiles"]):
+    _upsert_insert_filter: BooleanExpression | None = None
+
+    def set_upsert_insert_filter(self, insert_filter: BooleanExpression, 
case_sensitive: bool) -> None:
+        self._upsert_insert_filter = insert_filter
+        self._case_sensitive = case_sensitive
+
     def _validate_concurrency(self) -> None:
-        """Skip validation; appends do not conflict with other operations."""
+        """Skip validation for plain appends.
+
+        An upsert registers the filter for the rows it inserts, so a retry 
fails instead of
+        inserting a key that another writer added after the upsert was planned.
+        """
+        if self._upsert_insert_filter is None or self._commit_window is None 
or self._commit_window.is_empty():
+            return
+
+        from pyiceberg.table.update.validate import _validate_added_data_files
+
+        catalog_head = self._commit_window.head
+        if catalog_head is None:
+            return
+
+        try:
+            _validate_added_data_files(
+                self._transaction._table,
+                catalog_head,
+                self._upsert_insert_filter,
+                self._commit_window.base,
+                self._case_sensitive,
+            )
+        except ValidationException as e:
+            raise CommitFailedException(

Review Comment:
   A conflict on the update half of the upsert raises `ValidationException`, 
but here we convert it into a `CommitFailedException`. This means that the same 
upsert raises a different exception depending on which half conflicted. I can 
see the argument for `CommitFailedException` (it's what 0.11.1 raised, and the 
caller can simply retry the whole upsert), but I think we should be consistent 
across both halves. What do you think about letting the `ValidationException` 
propagate?



##########
pyiceberg/table/__init__.py:
##########
@@ -512,6 +512,8 @@ def append(
         df: pa.Table | pa.RecordBatchReader,
         snapshot_properties: dict[str, str] = EMPTY_DICT,
         branch: str | None = MAIN_BRANCH,
+        _upsert_insert_filter: BooleanExpression | None = None,

Review Comment:
   I'm not a big fan of adding private `_upsert_insert_filter` / 
`_case_sensitive` arguments to the public `Transaction.append` signature, since 
they show up in the API docs. Could `upsert` configure the producer directly 
instead? Also, the concept is not specific to upsert; it's a conflict detection 
filter. Something like `_conflict_detection_filter` / 
`validate_no_conflicting_data(filter)` would line up with Java's 
`OverwriteFiles`/`RowDelta`. Last, `set_upsert_insert_filter` also overwrites 
the inherited `_case_sensitive`, which is harmless today but easy to trip over 
later.



##########
pyiceberg/table/update/snapshot.py:
##########
@@ -690,8 +691,39 @@ def _refresh_for_retry(self) -> None:
 
 
 class _FastAppendFiles(_SnapshotProducer["_FastAppendFiles"]):
+    _upsert_insert_filter: BooleanExpression | None = None
+
+    def set_upsert_insert_filter(self, insert_filter: BooleanExpression, 
case_sensitive: bool) -> None:
+        self._upsert_insert_filter = insert_filter
+        self._case_sensitive = case_sensitive
+
     def _validate_concurrency(self) -> None:

Review Comment:
   This check runs regardless of the table's isolation level. The update half 
of the upsert only runs `_validate_added_data_files` under `serializable` (see 
the base `_validate_concurrency`), so with 
`write.update.isolation-level=snapshot` the two halves of a single upsert 
behave differently: the update half tolerates concurrently added files, while 
the insert half rejects them. This matches Java, where MERGE only validates 
conflicting data under serializable isolation. Should we check 
`isolation_level(Operation.OVERWRITE)` here so both halves follow the same 
setting?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to