sameer-malve commented on issue #4194: URL: https://github.com/apache/iceberg-python/issues/4194#issuecomment-6593969765
Hi, I'd be interested in working on this issue. I reproduced the problem on current main (`e3564a5`) using Python 3.13 and a SQLite-backed `SqlCatalog`. When two concurrent upserts insert the same new key, both succeed after the retry, leaving duplicate rows. I also verified that retries involving different new keys work correctly, while concurrent updates to an existing key already raise `ValidationException`. From the code investigation, the insert half of `Transaction.upsert` uses `self.append()`, which goes through `_FastAppendFiles`. Its concurrency validation currently doesn't check for conflicting inserts during a retry. My proposed approach is to pass a conflict-detection filter built from the inserted rows' join keys and reuse the existing added-data-file validation. Ordinary appends would remain unchanged. With Claude Code's assistance, an in-memory prototype was developed and tested. It prevented the duplicate under serializable isolation, preserved safe retries for non-conflicting keys, and passed the existing 162 relevant tests. Claude Code ran these additional prototype tests; I personally verified the original bug reproduction. The fix has not yet been applied to the repository. Before implementing the fix, I'd appreciate guidance on two points: 1. Should the check use the existing `write.update.isolation-level` setting so both halves of an upsert follow the same isolation level, or would a separate merge isolation setting be preferable? 2. Would a private `_conflict_detection_filter` parameter on `Transaction.append` be an acceptable way to pass the filter? The approach may conservatively reject some non-conflicting writes when file-level statistics overlap or are missing. I'd be happy to prepare a focused PR with regression tests once we agree on the design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
