andygrove opened a new pull request, #2308: URL: https://github.com/apache/datafusion-ballista/pull/2308
# Which issue does this PR close? N/A. This is a documentation-only change and does not have a tracking issue. # Rationale for this change Ballista blocks between query stages: a stage runs to completion and materializes its output to local disk before any downstream stage starts. Other DataFusion-based distributed engines, notably [DataFusion Distributed](https://datafusion-contrib.github.io/datafusion-distributed/) and [Sail](https://github.com/lakehq/sail), stream data between stages instead, and they are faster on interactive workloads as a result. We had nothing written down explaining why Ballista chose the blocking model, so the question keeps coming up and the answer lives in people's heads. The existing docs cover the mechanics of the shuffle writer in the tuning guide, but not the design rationale. That gap cuts both ways. Users evaluating Ballista against a pipelined engine have no way to tell whether the blocking shuffle is a deliberate trade-off or just something nobody has gotten around to fixing. Contributors proposing changes to the exchange layer have no shared statement of which properties the current design is protecting. Worth noting that DataFusion Distributed's own documentation already points at Ballista for large batch and ETL workloads that benefit from materializing intermediate results, so this completes a comparison they started rather than opening a new one. # What changes are included in this PR? A new contributors-guide page, `docs/source/contributors-guide/shuffle.md`, covering: - **The barrier, concretely.** The three places it is implemented: shuffle-write summaries not existing until files close, `UnresolvedStage::resolvable()` requiring every input stage to be complete, and `ShuffleReaderExec` fetching concrete `PartitionLocation`s. - **Why the barrier is there.** Five properties that follow from materializing stage output: producer and consumer tasks never compete for slots, so a stage can be far wider than the cluster; a lost executor costs re-run tasks rather than the whole query; the filesystem absorbs producer/consumer skew so shuffles larger than cluster memory stay safe; completed stages give the adaptive planner exact statistics; and executors can join or leave between stages. - **What it costs.** Straggler stall, write amplification, interactive latency, the local-disk requirement, and the absence of any path to streaming. - **How pipelined engines differ**, including a note that pipelining does not rule out adaptive query execution. DataFusion Distributed does AQE without a barrier by sampling in-flight batches, so the real distinction is exact versus sampled statistics. Empty-stage elimination is the one rule that genuinely needs a completed stage. - **A decision table** for when each model wins. - **Directions that do not require abandoning the model**, framed as open questions rather than commitments: partial-input early start, hybrid exchange, and a remote shuffle service (#1539). Supporting changes: - A user-facing FAQ entry, "Why does Ballista write shuffle data to disk instead of streaming it between stages?", with the short version and links onward. - A pointer from the Shuffle section of the architecture page. - A toctree entry in `docs/source/index.rst`. # Are there any user-facing changes? Documentation only. No code or API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
