milenkovicm opened a new pull request, #2744: URL: https://github.com/apache/datafusion-ballista/pull/2744
# Which issue does this PR close? Closes #. # Rationale for this change #2713 added a user guide page for pipelined execution, but the README never says that Ballista has more than one execution mode. Someone landing on the repository has no way to learn that pipelined execution exists, or how it differs from the default. # What changes are included in this PR? Adds an "Execution Modes" section to `README.md`, placed after "Architecture". It covers: - **Staged execution (default):** runs one stage at a time, the way Apache Spark does. Shuffle files let a failed task be re-run, which suits long-running jobs and shared clusters. - **Pipelined execution (experimental):** runs the way Trino does. Every task is scheduled at once and stages stream record batches over Arrow Flight, without writing intermediate data to disk. Results arrive sooner, but a failed task fails the whole job. It suits short, latency-sensitive queries. - How to choose a mode with `ballista.execution.mode`, plus a link to the Pipelined Execution guide. # Are there any user-facing changes? Documentation only. No code, API, or wire-format changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
