villebro commented on code in PR #2549: URL: https://github.com/apache/datafusion-ballista/pull/2549#discussion_r4198856618
########## docs/source/contributors-guide/benchmarking.md: ########## @@ -165,34 +204,36 @@ arithmetic (for example `date '1998-12-01' - interval '90' day` instead of ## Results -**Times in seconds; lower is better.** Every engine: mean of 2 iterations, -including the cold first iteration. - -| Query | Ballista (s) | Spark 4.1.3 (s) | Spark 4.1.3 + Comet 1.1.0-rc2 (s) | -| --------: | -----------: | --------------: | --------------------------------: | -| 1 | 13.94 | 71.45 | 10.66 | -| 2 | 28.35 | 37.98 | 21.66 | -| 3 | 25.29 | 30.61 | 16.05 | -| 4 | 8.80 | 23.86 | 8.11 | -| 5 | 63.94 | 57.39 | 38.32 | -| 6 | 4.68 | 1.73 | 0.99 | -| 7 | 45.85 | 25.34 | 16.60 | -| 8 | 28.93 | 58.17 | 46.04 | -| 9 | 37.64 | 74.90 | 55.78 | -| 10 | 38.18 | 35.31 | 21.31 | -| 11 | 15.87 | 31.66 | 14.79 | -| 12 | 10.18 | 13.24 | 5.33 | -| 13 | 9.78 | 22.73 | 11.31 | -| 14 | 8.88 | 7.25 | 2.50 | -| 15 | 10.61 | 22.45 | 11.85 | -| 16 | 14.50 | 24.34 | 8.94 | -| 17 | 19.56 | 81.36 | 25.14 | -| 18 | 54.43 | 134.11 | 46.42 | -| 19 | 10.65 | 13.27 | 9.07 | -| 20 | 20.40 | 16.08 | 6.57 | -| 21 | 91.31 | 98.04 | 68.43 | -| 22 | 9.78 | 16.90 | 9.22 | -| **Total** | **571.55** | **898.17** | **455.09** | +**Times in seconds; lower is better.** Ballista, Spark, and Spark + Comet: +mean of 2 iterations, including the cold first iteration. Trino: mean of 3 +iterations after an untimed warm-up run (see +[Trino configuration](#trino-configuration)). Review Comment: While measuring warm performance for Trino is probably better due to JIT overhead, we should try to avoid penalizing Ballista, Spark and Spark + Comet by having their results cover both cold and warm runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
