akshaychitneni opened a new pull request, #2730:
URL: https://github.com/apache/datafusion-ballista/pull/2730
Part of #1829. Follows #2633 (design doc step 5: recovery, part 2).
# Rationale for this change
Since #2633, a running query that loses cache data fails cleanly and the
retry
rebuilds the cache. This PR rebuilds the cache and resumes the query
instead.
# What changes are included in this PR?
- **Detection:** a failed fetch is cache data when it matches a cache
location
(executor, build stage, partition), with no protocol change. If the
job's own
input on that executor also matches, both are recovered. On executor
loss, the
cache inputs left incomplete are reported in a new
`ExecutorLostOutcome::lost_inputs` field.
- **Rebuild and resume:** the reading stage rolls back and waits
(`ExternalInputLost`). The scheduler drops the lost build and then
starts a
rebuild, joins one in flight, or reuses a newer build.
`ExternalInputReady`
then re-resolves the stage.
- **Pipelined jobs:** a cache lost before the job is placed is rebuilt,
then the
job is placed reading the rebuild. Once placed, the job fails cleanly,
because
its tasks and its results to the client are already streaming.
- **Guard:** a job that reads the lost cache more than once, directly or
through
another cache built from it, fails cleanly, so a query never sees two
versions
of a cache.
- **Edge cases:** rebuilds run off the event loop under the job's
admission. A
rebuild that fails or writes a different schema fails the job. A stage
out of
attempts names the lost data, so the retry rebuilds it.
- **Unchanged:** adaptive-planner jobs still fail cleanly; that graph
cannot
resume a rolled-back stage.
- **Tests and docs:** graph, flow and pipelined-graph tests, chaos
Scenario L,
and a new k8s Scenario M that kills the pod holding a cache mid-read.
Updated
the design doc, tuning guide and 56.0.0 upgrade guide.
# Are there any user-facing changes?
Yes. A static-planner query that loses `DataFrame::cache()` data now
rebuilds it
and finishes, as does a pipelined query that loses it before it starts.
`ExecutorLostOutcome` gains `lost_inputs` (see the upgrade guide).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
