RussellSpitzer commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2154964023
In between flights I found another suspect thing which was when a row was deleted the transaction log still attempts to find it rather than check for its absence. Gonna sleep now look at this again later.Sent from my iPhoneOn Jun 7, 2024, at 4:24 AM, Szehon Ho ***@***.***> wrote: Yea , I think @RussellSpitzer has a good point. I think this code https://github.com/matthijseikelenboom/iceberg-acid-verification/blob/master/src/main/java/org/example/writer/TransactionWriter.java#L167 swallows the exception. Iceberg isolation was designed work mostly by throwing exceptions like ValidationException when it detects concurrent changes to the table that would break the given isolation level. The user then needs to retry the query. Running this test, I saw some instances of IllegalStateException Runtime file filtering is not possible: the table has been concurrently modified. " + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. " + "If an external process modifies the table, enable table caching in the catalog. " + "If multiple threads modify the table, use independent Spark sessions in each thread.", This is related to ValidationException but actually thrown because a certain optimization for MERGE INTO (runtime file-filtering) seems to actually requires the table to be completely up to date. This can be turned off if you wish: spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled=false But for either case, the test should handle the exception and retry. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
