Copilot commented on code in PR #17921: URL: https://github.com/apache/iceberg/pull/17921#discussion_r3911447251
########## site/docs/benchmarks.md: ########## @@ -20,113 +20,213 @@ title: "Benchmarks" ## Available Benchmarks and how to run them -Benchmarks are located under `<project-name>/jmh`. It is generally favorable to only run the tests of interest rather than running all available benchmarks. -Also note that JMH benchmarks run within the same JVM as the system-under-test, so results might vary between runs. +Benchmarks are located under `<module>/src/jmh`. It is generally better to run only the benchmarks you are investigating instead of the full suite. +Iceberg benchmarks use JMH forks to run measurements in a new JVM for isolation and reproducibility. +Other work on the same host can still compete for CPU, memory, or I/O and produce inconsistent results. +Run benchmarks on an otherwise idle host, or preferably a dedicated benchmark host, for more reliable comparisons. ## Running Benchmarks on GitHub -It is possible to run one or more Benchmarks via the **JMH Benchmarks** GH action on your own fork of the Iceberg repo. This GH action takes the following inputs: -* The repository name where those benchmarks should be run against, such as `apache/iceberg` or `<user>/iceberg` -* The branch name to run benchmarks against, such as `main` or `my-cool-feature-branch` +It is possible to run one or more benchmarks via the **JMH Benchmarks** GitHub Actions workflow on your own fork of the Iceberg repository. This workflow takes the following inputs: + +* The repository name to run against, such as `apache/iceberg` or `<user>/iceberg` +* The branch name to benchmark, such as `main` or `my-feature-branch` * The Spark version to use, such as `4.1` (default: `4.1`) * The Scala version to use, such as `2.13` (default: `2.13`) -* A list of comma-separated double-quoted Benchmark names, such as `"IcebergSourceFlatParquetDataReadBenchmark", "IcebergSourceFlatParquetDataFilterBenchmark", "IcebergSourceNestedListParquetDataWriteBenchmark"` +* A comma-separated list of double-quoted benchmark names, such as `"IcebergSourceFlatParquetDataReadBenchmark", "IcebergSourceFlatParquetDataFilterBenchmark", "IcebergSourceNestedListParquetDataWriteBenchmark"` Benchmark results will be uploaded once **all** benchmarks are done. -It is worth noting that the GH runners have limited resources so the benchmark results should rather be seen as an indicator to guide developers in understanding code changes. -It is likely that there is variability in results across different runs, therefore the benchmark results shouldn't be used to form assumptions around production choices. +GitHub-hosted runners have limited and shared resources, so treat these results as directional signals for understanding code changes rather than as production-grade measurements. ## Running Benchmarks locally -Below are the existing benchmarks shown with the actual commands on how to run them locally. - -### IcebergSourceNestedListParquetDataWriteBenchmark -A benchmark that evaluates the performance of writing nested Parquet data using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=IcebergSourceNestedListParquetDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-list-parquet-data-write-benchmark-result.txt` - -### SparkParquetReadersNestedDataBenchmark -A benchmark that evaluates the performance of reading nested Parquet data using Iceberg and Spark Parquet readers. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=SparkParquetReadersNestedDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-readers-nested-data-benchmark-result.txt` - -### SparkParquetWritersFlatDataBenchmark -A benchmark that evaluates the performance of writing Parquet data with a flat schema using Iceberg and Spark Parquet writers. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=SparkParquetWritersFlatDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-writers-flat-data-benchmark-result.txt` - -### IcebergSourceFlatORCDataReadBenchmark -A benchmark that evaluates the performance of reading ORC data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=IcebergSourceFlatORCDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-orc-data-read-benchmark-result.txt` - -### SparkParquetReadersFlatDataBenchmark -A benchmark that evaluates the performance of reading Parquet data with a flat schema using Iceberg and Spark Parquet readers. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=SparkParquetReadersFlatDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-readers-flat-data-benchmark-result.txt` - -### VectorizedReadDictionaryEncodedFlatParquetDataBenchmark -A benchmark to compare performance of reading Parquet dictionary encoded data with a flat schema using vectorized Iceberg read path and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=VectorizedReadDictionaryEncodedFlatParquetDataBenchmark -PjmhOutputPath=benchmark/vectorized-read-dict-encoded-flat-parquet-data-result.txt` - -### IcebergSourceNestedListORCDataWriteBenchmark -A benchmark that evaluates the performance of writing nested Parquet data using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=IcebergSourceNestedListORCDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-list-orc-data-write-benchmark-result.txt` - -### VectorizedReadFlatParquetDataBenchmark -A benchmark to compare performance of reading Parquet data with a flat schema using vectorized Iceberg read path and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: - -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=VectorizedReadFlatParquetDataBenchmark -PjmhOutputPath=benchmark/vectorized-read-flat-parquet-data-result.txt` - -### IcebergSourceFlatParquetDataWriteBenchmark -A benchmark that evaluates the performance of writing Parquet data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: +JMH writes human-readable output to `build/reports/jmh/human-readable-output.txt` and JSON output to `build/reports/jmh/results.json` by default. Override them with `-PjmhOutputPath=<path>` and `-PjmhJsonOutputPath=<path>` if needed. You can share the JSON output with others and view it in the [JMH Visualizer](https://jmh.morethan.io/). -`./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 :iceberg-spark:iceberg-spark-4.1_2.13:jmh -PjmhIncludeRegex=IcebergSourceFlatParquetDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-parquet-data-write-benchmark-result.txt` +Core and data benchmarks can be run directly. Spark and Flink benchmarks use versioned Gradle modules. +Spark `4.1` benchmarks use `2.13` module names, while older Spark modules follow the configured `scalaVersion`. Review Comment: The Spark/Scala module naming note is incomplete: Spark 4.0 modules are also hard-coded to `_2.13` in `settings.gradle`, so this currently reads as if only Spark 4.1 is special. This can mislead users into thinking `scalaVersion` affects Spark 4.0/4.1 module names. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
