andygrove opened a new pull request, #5374:
URL: https://github.com/apache/datafusion-comet/pull/5374
## Which issue does this PR close?
Part of #1636.
## Rationale for this change
Comet has around 35 micro benchmark suites, but running them is a manual,
one-suite-at-a-time
exercise (`SPARK_GENERATE_BENCHMARK_FILES=1 make benchmark-<class>`), and
the results are written
to `spark/benchmarks`, which is git ignored. That makes it tedious to
produce a full set of numbers
and impossible to publish them without force adding files.
Running the suites on a laptop also produces noisy numbers because of
thermal throttling,
efficiency cores, and background load. A dedicated EC2 instance gives
repeatable results, but the
existing EC2 guide only covers TPC-H and TPC-DS.
## What changes are included in this PR?
- `benchmarks/micro/run.py`, a dependency-free Python entry point with
`setup`, `run`, `collect`,
`publish` and `all` subcommands. It installs the prerequisites, clones or
updates the repository
at a chosen ref, builds in release mode, runs each suite in its own JVM
with per-suite logs,
timings and timeouts, continues past failures, and copies the results into
`benchmarks/results/micro` together with a `RUN-INFO.md` recording the
instance type, CPU,
memory, JDK, Rust and Comet commit that produced them. The script can be
downloaded on its own
before the repository is cloned.
- The three TPC suites and `CometC2RIsolatedBench` are excluded from the
default set, with the
reason reported by `run --list`. The default set is the 35 suites that
generate their own data.
- `benchmarks/results/micro` as the tracked location for published results.
`spark/benchmarks`
stays ignored and is used as scratch output.
- A new contributor guide, `benchmarking_micro_ec2.md`, covering instance
creation on
`m7i.xlarge`, manual installation of every prerequisite (git, make, gcc,
cmake,
protobuf-compiler, JDK 17, Rust, and a note that Maven is provided by
`./mvnw`), running,
collecting, publishing, guidance on getting comparable numbers, and
troubleshooting.
- `BENCH_HEAP` in the `Makefile`, defaulting to the previous `20g`, so that
the `benchmark-%`
target can also be run by hand on an instance with less memory.
- A RAT exclusion for the result text files.
## How are these changes tested?
The runner was exercised locally against a release build: suite listing and
`--only` / `--skip`
filtering, a dry run of the Maven invocation, a real run of
`CometArithmeticBenchmark`, and the
collect and publish steps including the generated `RUN-INFO.md` and the git
and `gh` commands.
Documentation changes were checked with prettier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
