andygrove opened a new pull request, #2011:
URL: https://github.com/apache/datafusion-ballista/pull/2011
# Which issue does this PR close?
<!-- No dedicated issue; this refreshes the stale benchmark charts in the
README. -->
This does not close a specific issue — it refreshes the performance section
of the top-level README with current numbers and adds reproducible tooling for
regenerating the charts.
# Rationale for this change
The README's Performance section quoted a "2.9x" overall speedup with a
chart set that no longer reflects current behavior, and there was no committed,
reproducible way to regenerate those charts. This PR replaces the stale numbers
with a fresh, fair Spark-vs-Ballista TPC-H run and lands the chart generator so
future refreshes are a one-command operation.
# What changes are included in this PR?
- Add `benchmarks/generate-comparison.py`, a chart generator (adapted from
Apache DataFusion Comet's script, ASF header preserved) that reads two JSON
result files and emits the four `tpch_*.png` charts the README references. It
accepts both the Spark/Comet result format (keyed by query) and Ballista's
`tpch` binary format (`{"queries": [...]}`) via a small `normalize_result`
loader, plus a unit test for that loader.
- Document the regeneration steps in `benchmarks/README.md`.
- Commit the two raw SF100 result JSONs under `benchmarks/results/` for
provenance.
- Regenerate the four `docs/source/_static/images/tpch_*.png` charts and
update the README Performance prose.
Benchmark configuration for the published numbers: TPC-H scale factor 100,
two-node cluster (2 executors × 8 cores) reading Parquet from node-local disk,
3 iterations, identical SQLBench-H query set for both engines.
- **Apache Spark 3.5.3** (vanilla, Comet disabled): 369.6 s
- **Ballista 54.0.0-rc2** (`prefer_hash_join=true`, static planner): 244.7 s
- **Overall speedup: 1.5x** (sum of median-of-three runtimes across all 22
queries)
Row counts match between the two engines on 21 of 22 queries; they differ
only on Q15 (Spark 0 rows, Ballista 1), which is noted in the README and does
not change the overall speedup.
# Are there any user-facing changes?
Documentation only — the README Performance section and its chart images are
updated. No code or API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
