szehon-ho commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2154449447
Yea , I think @RussellSpitzer has a good point. I think this code https://github.com/matthijseikelenboom/iceberg-acid-verification/blob/master/src/main/java/org/example/writer/TransactionWriter.java#L167 swallows the exception. Iceberg isolation was designed work mostly by throwing exceptions like ValidationException when it detects concurrent changes to the table that would break the given isolation level. The user then needs to retry the query. Running this test, I saw some instances of IllegalStateException ``` Runtime file filtering is not possible: the table has been concurrently modified. " + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. " + "If an external process modifies the table, enable table caching in the catalog. " + "If multiple threads modify the table, use independent Spark sessions in each thread.", ``` This is related to ValidationException but actually thrown because a certain optimization for MERGE INTO (runtime file-filtering) seems to actually requires the table to be completely up to date. This can be turned off if you wish: spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled=false But for either case, the test should handle the exception and retry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
