c-thiel opened a new pull request, #3697: URL: https://github.com/apache/iceberg-rust/pull/3697
## Which issue does this PR close? No issue yet. A scan can hang forever when two data files share an equality delete file: a task can wait on a notifier that is never woken. @JeonDaehong found this race and proposed this fix on #3275, planned as a follow-up once the open PR limit allows. This PR sends that fix now, with @JeonDaehong as co-author of the commit. The reviews of #2873 and #3268 also raised it. ## What changes are included in this PR? `CachingDeleteFileLoader` marks an equality delete file as loading with `try_start_eq_del_load`, which stores a notifier. It then calls `insert_equality_delete`, which replaced that state with a second notifier and woke only the second one when the predicate arrived. A task that read the state between the two calls waited on the first notifier forever, so the record batch stream from `ArrowReader::read` never finished. This needs a multi-thread runtime and a pause between the two calls, so it is rare, but the hang is permanent. `insert_equality_delete` now takes the notifier from `try_start_eq_del_load` and keeps the loading state that it set. A failed load is a separate case (#3274, #3275). ## Are these changes tested? Yes, in `arrow::delete_filter`: - `test_eq_del_waiter_registered_before_insert_is_woken` registers a waiter between the two calls and checks that it gets the predicate. Without the fix it times out. - `test_build_equality_delete_predicate_case_sensitive` now starts the load first, as the loader does. ## AI Disclosure Supported by Claude Code. Reviewed and checked by me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
