Vanlightly commented on issue #10720: URL: https://github.com/apache/iceberg/issues/10720#issuecomment-2241142730
Correct, I missed that check. I've added it to the model and that consistency violation is no longer occurring. Thanks. However, I'm now hitting a different consistency violation. This time with merge-on-read, Snapshot Isolation and a conflicting update op and a delete op. 1. A single row (['jack', 'red', 'A']) is added to 'data-1' in an append op. 2. An update op and a delete op start, using the same starting snapshot. 3. The update op sets the color to 'blue'. It creates a 'data-2' file with the row ['jack', 'blue', 'A']. It also creates delete file 'delete-1' for the single row of 'data-1'. It does everything but commit. 4. The delete op wants to delete any rows where id = 'jack'. It adds delete file 'delete-2' for the single row of 'data-1'. This means 'data-1' has two delete files referencing it, though none is committed yet at this point. 5. The update op commits. 6. The delete op refreshes metadata, does validation, writes its metadata files, then commits. The problem is that in my model, the delete op validation does not include checks for detecting conflicting deletes. The reason is this line of code: https://github.com/apache/iceberg/blob/e02b5c90ef305b4d1ca5c19f0b9b2e99f9392e44/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java#L219 Only update and merge operations cause Spark to enable the necessary validation checks, a delete op will not perform them. I could be wrong again. Not sure of the best way to test this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
