phillipleblanc opened a new issue, #2036: URL: https://github.com/apache/datafusion-ballista/issues/2036
**Describe the bug** `TaskManager::succeed_job` and `TaskManager::abort_job` remove a job from the active cache before saving its terminal state. During that window, `get_job_status` misses the active cache and reads shared state, which may still report `Running`. A completed, failed, or cancelled job can therefore appear to move back to `Running`. The abort path has an additional failure mode. It collects the running tasks that need cancellation, then awaits `save_job`. If persistence fails, the method returns before those tasks reach `QueryStageScheduler`, so executor work is not cancelled. **To Reproduce** Use a `JobState` implementation whose `save_job` method can block or fail: 1. Add a running job to the active cache and shared state. 2. Mark the execution graph as successful or failed. 3. Call `succeed_job` or `abort_job`. 4. Block `save_job` before it updates shared state. 5. Call `get_job_status` while persistence is blocked. 6. Observe that the status falls back to stale shared state and reports `Running`. For the abort failure: 1. Assign at least one task to an executor. 2. Call `abort_job`. 3. Make `save_job` return an error. 4. Observe that the captured running tasks are not dispatched for cancellation. **Expected behavior** Terminal job status should remain continuously visible while it is persisted. Status requests should not observe `Running` after the local graph becomes successful, failed, or cancelled. Running executor tasks should be dispatched for cancellation even if terminal-state persistence fails. The scheduler should also avoid holding execution-graph locks across asynchronous persistence. **Additional context** The race is difficult to observe with the in-memory state backend because persistence is fast. It becomes more likely with a slower external state backend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
