Fokko commented on code in PR #4392:
URL: https://github.com/apache/iceberg-python/pull/4392#discussion_r4602394200
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -690,8 +691,39 @@ def _refresh_for_retry(self) -> None:
class _FastAppendFiles(_SnapshotProducer["_FastAppendFiles"]):
+ _upsert_insert_filter: BooleanExpression | None = None
+
+ def set_upsert_insert_filter(self, insert_filter: BooleanExpression,
case_sensitive: bool) -> None:
+ self._upsert_insert_filter = insert_filter
+ self._case_sensitive = case_sensitive
+
def _validate_concurrency(self) -> None:
- """Skip validation; appends do not conflict with other operations."""
+ """Skip validation for plain appends.
+
+ An upsert registers the filter for the rows it inserts, so a retry
fails instead of
+ inserting a key that another writer added after the upsert was planned.
+ """
+ if self._upsert_insert_filter is None or self._commit_window is None
or self._commit_window.is_empty():
+ return
+
+ from pyiceberg.table.update.validate import _validate_added_data_files
+
+ catalog_head = self._commit_window.head
+ if catalog_head is None:
+ return
+
+ try:
+ _validate_added_data_files(
+ self._transaction._table,
+ catalog_head,
+ self._upsert_insert_filter,
+ self._commit_window.base,
+ self._case_sensitive,
+ )
+ except ValidationException as e:
+ raise CommitFailedException(
Review Comment:
A conflict on the update half of the upsert raises `ValidationException`,
but here we convert it into a `CommitFailedException`. This means that the same
upsert raises a different exception depending on which half conflicted. I can
see the argument for `CommitFailedException` (it's what 0.11.1 raised, and the
caller can simply retry the whole upsert), but I think we should be consistent
across both halves. What do you think about letting the `ValidationException`
propagate?
##########
pyiceberg/table/__init__.py:
##########
@@ -512,6 +512,8 @@ def append(
df: pa.Table | pa.RecordBatchReader,
snapshot_properties: dict[str, str] = EMPTY_DICT,
branch: str | None = MAIN_BRANCH,
+ _upsert_insert_filter: BooleanExpression | None = None,
Review Comment:
I'm not a big fan of adding private `_upsert_insert_filter` /
`_case_sensitive` arguments to the public `Transaction.append` signature, since
they show up in the API docs. Could `upsert` configure the producer directly
instead? Also, the concept is not specific to upsert; it's a conflict detection
filter. Something like `_conflict_detection_filter` /
`validate_no_conflicting_data(filter)` would line up with Java's
`OverwriteFiles`/`RowDelta`. Last, `set_upsert_insert_filter` also overwrites
the inherited `_case_sensitive`, which is harmless today but easy to trip over
later.
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -690,8 +691,39 @@ def _refresh_for_retry(self) -> None:
class _FastAppendFiles(_SnapshotProducer["_FastAppendFiles"]):
+ _upsert_insert_filter: BooleanExpression | None = None
+
+ def set_upsert_insert_filter(self, insert_filter: BooleanExpression,
case_sensitive: bool) -> None:
+ self._upsert_insert_filter = insert_filter
+ self._case_sensitive = case_sensitive
+
def _validate_concurrency(self) -> None:
Review Comment:
This check runs regardless of the table's isolation level. The update half
of the upsert only runs `_validate_added_data_files` under `serializable` (see
the base `_validate_concurrency`), so with
`write.update.isolation-level=snapshot` the two halves of a single upsert
behave differently: the update half tolerates concurrently added files, while
the insert half rejects them. This matches Java, where MERGE only validates
conflicting data under serializable isolation. Should we check
`isolation_level(Operation.OVERWRITE)` here so both halves follow the same
setting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]