andygrove opened a new issue, #2029: URL: https://github.com/apache/datafusion-ballista/issues/2029
## Describe the bug If every executor dies while a query is running, the job **never terminates**. The scheduler waits indefinitely rather than failing the query, so the client hangs forever. ## To Reproduce Reproduced end to end on a real multi-process cluster by the chaos harness in #PR_NUM (scenario `killing_every_executor_terminates_the_job`, which fails under **both** AQE on and AQE off): 1. Start a scheduler and 2 executors. 2. Submit a query (the harness slows a stage with a delay UDF so the kill lands while tasks are genuinely running). 3. Wait until stage 1 has running tasks. 4. `SIGKILL` **both** executors. 5. Wait. The job does not terminate within 120 seconds. The harness asserts only that the job *ends* — it makes no claim about success — and even that does not hold. ## Expected behavior The job should fail with a clear error once the scheduler determines its tasks cannot be scheduled — there are no executors left to run them, and no prospect of recovery without a new executor registering. Some judgement is needed on the exact policy, and reasonable people could disagree: - Failing fast once all executors are lost is the simplest behavior and is what a client most likely wants. - Alternatively, if the intent is to *wait* for executors to come back (a plausible design for a long-lived cluster), then that wait should be **bounded** by a configurable timeout, after which the job fails — and ideally the job should be visibly reported as blocked-on-resources in the meantime rather than looking indistinguishable from a job that is simply slow. Either way, an unbounded silent hang is not a good outcome: the caller has no signal and no way to distinguish it from progress. ## Additional context Found while building an HA chaos-testing harness (#PR_NUM) that drives the scheduler's fault-tolerance paths on a real multi-process cluster. This scenario is deliberately a **hang detector** rather than a correctness test. Lower severity than the two sibling issues (the `FetchFailed` and `Shared`-wrapping misclassifications), since total executor loss is a more extreme condition than a single executor dying — but the failure mode (silent unbounded hang) is worse than a clean error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
