andygrove commented on PR #2677: URL: https://github.com/apache/datafusion-ballista/pull/2677#issuecomment-6481183858
Thanks for tracking this down, a silent duplicate is a nasty one! I checked it locally. Both new tests fail on the unfixed code with exactly the doubled partitions you describe, and pass with the fix. Clearing the outputs in `rollback_running_stage` covers both the fetch-failure and lost-executor paths, and late updates from the old attempt are already dropped, so nothing can sneak the old locations back in. The chaos scenario is a nice addition too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
