c-thiel opened a new pull request, #3697:
URL: https://github.com/apache/iceberg-rust/pull/3697

   ## Which issue does this PR close?
   
   No issue yet. A scan can hang forever when two data files share an equality 
delete file: a task can wait on a notifier that is never woken.
   
   @JeonDaehong found this race and proposed this fix on #3275, planned as a 
follow-up once the open PR limit allows. This PR sends that fix now, with 
@JeonDaehong as co-author of the commit. The reviews of #2873 and #3268 also 
raised it.
   
   ## What changes are included in this PR?
   
   `CachingDeleteFileLoader` marks an equality delete file as loading with 
`try_start_eq_del_load`, which stores a notifier. It then calls 
`insert_equality_delete`, which replaced that state with a second notifier and 
woke only the second one when the predicate arrived. A task that read the state 
between the two calls waited on the first notifier forever, so the record batch 
stream from `ArrowReader::read` never finished. This needs a multi-thread 
runtime and a pause between the two calls, so it is rare, but the hang is 
permanent.
   
   `insert_equality_delete` now takes the notifier from `try_start_eq_del_load` 
and keeps the loading state that it set. A failed load is a separate case 
(#3274, #3275).
   
   ## Are these changes tested?
   
   Yes, in `arrow::delete_filter`:
   
   - `test_eq_del_waiter_registered_before_insert_is_woken` registers a waiter 
between the two calls and checks that it gets the predicate. Without the fix it 
times out.
   - `test_build_equality_delete_predicate_case_sensitive` now starts the load 
first, as the loader does.
   
   ## AI Disclosure
   
   Supported by Claude Code. Reviewed and checked by me.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to