andygrove opened a new pull request, #2026:
URL: https://github.com/apache/datafusion-ballista/pull/2026
# Which issue does this PR close?
Closes #ISSUE_HARNESS.
This harness reproduces the following, each of which is filed separately:
- #ISSUE_FETCH — shuffle-fetch failures lose their type, so the map-stage
resubmit never fires
- #ISSUE_RETRY — retryable IO errors are misclassified when wrapped in
`DataFusionError::Shared`
- #ISSUE_HANG — killing every executor hangs the job instead of failing it
# Rationale for this change
Ballista has a substantial high-availability state machine in
`ballista/scheduler/src/state/execution_graph.rs` (`update_task_status`):
failures are classified
`retryable` / `count_to_failures`, tasks retry up to `task_max_failures`, a
`FetchPartitionError`
rolls back the running stage and resubmits the map stage, and a lost
executor resets its running
tasks and removes the shuffle output it produced so map stages re-run.
None of it was tested end to end. Every existing test of these paths
fabricates `TaskStatus`
protobuf messages by hand (`execution_graph.rs` tests,
`scheduler/src/test_utils.rs`). No test ever
drove the HA state machine from a real query on a real cluster, and no test
ever killed an executor.
Running the state machine for real immediately surfaced three bugs, two of
which mean Ballista
**fails queries that it is designed to recover from**. See the linked issues.
There is an existing chaos operator (`ChaosExec`), but it is unusable as a
regression harness: it is
wired only into the AQE planner (`state/aqe/planner.rs`), so it cannot
inject faults with AQE off; it
wraps a *randomly chosen* plan node; and it fires *probabilistically*. No
test uses the
`ballista.testing.chaos_execution.*` keys today.
# What changes are included in this PR?
A new non-published workspace crate, `chaos-testing` (`ballista-chaos`).
**No production crate is
modified** — the only change outside `chaos-testing/` is adding the member
to the workspace
`Cargo.toml`.
- **Fault injection via UDFs, not the planner.** `chaos_fail(guard, mode,
budget_dir)` and
`chaos_delay(guard, ms)` are pass-through scalar UDFs spliced into the
query text, so the identical
injection works with AQE on and AQE off with zero planner wiring. `mode`
selects which of Ballista's
three failure classifications is exercised: `io` (retryable), `exec`
(non-retryable), `panic`
(caught by the executor's `catch_unwind`, non-retryable).
- **Determinism without RNG.** The `guard` predicate over fixed data selects
*which* partitions fault;
a filesystem **token budget** bounds *how many* attempts fault,
cluster-wide, across task retries and
executor restarts (consuming a token is `fs::remove_file`, which is atomic
across processes). Budget 1
means exactly one attempt faults anywhere, so a retry *must* succeed;
budget ≫ `task_max_failures`
means retries *must* exhaust.
- **A real multi-process cluster.** `TestCluster` spawns real
`chaos-scheduler` / `chaos-executor`
binaries as OS processes (they inject the UDFs via the existing
`override_function_registry` /
`override_session_builder` hooks, following
`examples/custom-executor.rs`), so executors can be
`SIGKILL`ed and restarted. It tunes `executor_timeout_seconds` / heartbeat
down from the production
defaults of 180s/60s, without which a killed executor would not be noticed
for three minutes.
- **Kills land at a known point.** The scheduler's REST API is polled
(`/api/job/{id}/stages`) so a kill
happens while a stage genuinely has running tasks, rather than after a
guessed `sleep`.
- **Seven scenarios, each run under both AQE settings**: retryable fault
recovered; retries exhausted;
panicking task; executor killed mid-stage; executor killed after writing
shuffle output; executor
killed and restarted; every executor killed.
The load-bearing assertion in every recovery scenario is **result equality
against a chaos-free baseline
run**, not merely "it did not error" — re-running a stage is exactly where
duplicated or dropped
partitions would appear, and only a correctness check catches that.
# Current test status
`cargo test -p ballista-chaos -- --test-threads=1` → **11 passed, 5 failed,
0 ignored.**
The 5 failures are the three bugs above, reproduced end to end. They are
**left failing on purpose** and
are not suppressed: a scenario that fails because Ballista is broken is the
point of this harness. Each
is documented in `chaos-testing/README.md` with its root cause. Once the
linked issues are fixed, these
scenarios become the regression tests for them.
Passing: baseline correctness (both AQE settings), retry-on-IO-fault with
AQE off, retry exhaustion,
panicking task (job fails cleanly, executor survives), executor killed after
shuffle write, and
executor kill + restart.
# Are there any user-facing changes?
No. The crate is test-only and `publish = false`; no production crate is
touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
