sam-1112 commented on PR #6601: URL: https://github.com/apache/datafusion-comet/pull/6601#issuecomment-6235161286
@andygrove Thanks for digging into this. Your reproductions helped a lot. I’ve addressed the issues you pointed out: - Moved committed() outside the commit try/catch, and added tests showing that a throwing probe won’t call abort or delete committed files. - Moved the pause into IcebergSchedulerProbeRDD and removed the native gate and test-only proto field. Files are now tracked using the attempt-specific filename prefix. - Added write.parquet.page-row-limit=1000. The JVM gate now pauses after supplying 4,000 rows to account for native prefetch, and the docs explain that. - Added the local executor-loss probe test with release-all created up front. It checks that the progress marker records files with bytes on disk. I also added coverage for blocking, both release paths, timeout/retry, and commit failure. - Updated the Testing table and handled the command argument from #6693. I reran both executor-loss scenarios and both speculation cases, and all four passed. These ran on two Docker worker containers on one host, with separate IPs and PID namespaces. The storage audits found no missing referenced files, and any remaining orphans belonged to known rejected attempts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
