andygrove opened a new pull request, #2159: URL: https://github.com/apache/datafusion-ballista/pull/2159
# Which issue does this PR close? Part of #2158. Contributes toward #2144. # Rationale for this change Under pull-based scheduling the executor's only heartbeat to the scheduler is `poll_work` — the scheduler records a heartbeat on every poll. The poll loop, however, blocked on `free_vcores.acquire().await` *before* polling, waiting for a free vcore to bind new work. When every vcore is held by a task running longer than the scheduler's `executor_timeout` (e.g. a long stage-1 scan on a wide TPC-H query such as q9/q18), the loop blocked on `acquire` indefinitely and never polled. The scheduler then stopped receiving heartbeats, marked the healthy-but-busy executor dead, and reset its in-flight (and completed) tasks. When a stage's tasks consistently exceed the timeout this livelocks: `reset -> re-dispatch -> reset`, never converging — the executor goes idle and the job hangs. This is one of the root causes behind the fleet-wide heartbeat timeouts reported in #2144. It was originally found and fixed in the Spice AI fork; this ports the fix to current `main` (the loop has since been refactored to use `free_vcores`/`num_free_vcores`, so the fix is adapted accordingly). # What changes are included in this PR? - Cap the vcore wait at a new `HEARTBEAT_POLL_INTERVAL` (5s, well below the scheduler's executor timeout) using `tokio::time::timeout`. - On timeout, poll the scheduler anyway, reporting `num_free_vcores: 0` (heartbeat only), so executor liveness no longer depends on vcore availability. - A freed vcore still wakes the wait immediately, so scheduling responsiveness is unchanged; when fully busy the loop now polls roughly every 5s instead of never. - Handle a closed semaphore (executor shutdown) explicitly. # Are there any user-facing changes? No. Behavior only changes for fully-busy executors, which now continue to heartbeat instead of being wrongly declared dead. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
