kaivalnp commented on PR #15285: URL: https://github.com/apache/lucene/pull/15285#issuecomment-3398444020
> It's hard for human eyes + brain to scan all those numbers and confirm there's no real difference Haha true :) I fed the raw data to an LLM and asked it to report percentage differences: | Benchmark | Baseline Score (ops/μs) | Candidate Score (ops/μs) | % Difference | |-----------|------------------------|-------------------------|--------------| | floatCosineVector | 8.913 | 8.646 | -3.00% | | floatDotProductVector | 12.561 | 12.249 | -2.48% | | binaryHalfByteDotProductBothPackedVector | 11.652 | 11.498 | -1.32% | | l2NormalizeVector | 12.349 | 12.207 | -1.15% | | binaryDotProductUint8Scalar | 2.307 | 2.288 | -0.82% | | binarySquareScalar | 2.431 | 2.413 | -0.74% | | binaryHalfByteSquareBothPackedVector | 12.614 | 12.562 | -0.41% | | floatSquareScalar | 3.181 | 3.171 | -0.31% | | floatCosineScalar | 1.419 | 1.415 | -0.28% | | binaryHalfByteSquareVector | 18.526 | 18.487 | -0.21% | | binaryHalfByteDotProductScalar | 2.378 | 2.376 | -0.08% | | binarySquareUint8Scalar | 2.422 | 2.420 | -0.08% | | binaryHalfByteSquareSinglePackedScalar | 2.022 | 2.021 | -0.05% | | floatDotProductScalar | 3.734 | 3.733 | -0.03% | | binaryDotProductVector | 8.040 | 8.039 | -0.01% | | binarySquareVector | 6.710 | 6.709 | -0.01% | | binaryCosineScalar | 0.841 | 0.841 | 0.00% | | binaryDotProductUint8Vector | 8.040 | 8.040 | 0.00% | | binaryHalfByteDotProductSinglePackedVector | 2.627 | 2.627 | 0.00% | | binarySquareUint8Vector | 6.709 | 6.709 | 0.00% | | l2Normalize | 3.016 | 3.017 | 0.03% | | binaryHalfByteDotProductSinglePackedScalar | 2.446 | 2.449 | 0.12% | | binaryHalfByteDotProductBothPackedScalar | 2.368 | 2.376 | 0.34% | | binaryHalfByteSquareScalar | 2.465 | 2.474 | 0.36% | | binaryDotProductScalar | 2.289 | 2.298 | 0.39% | | binaryHalfByteDotProductVector | 20.677 | 20.785 | 0.52% | | binaryHalfByteSquareSinglePackedVector | 2.590 | 2.609 | 0.73% | | floatSquareVector | 12.370 | 12.483 | 0.91% | | binaryCosineVector | 4.778 | 4.860 | 1.72% | | binaryHalfByteSquareBothPackedScalar | 1.642 | 1.696 | 3.29% | Side note: I found this cool visualizer (https://jmh.morethan.io), which takes the JSON output of JMH (add `-rf json` to the command line), and can compare multiple runs too! For example, I re-ran a subset of functions and recorded their output in https://gist.github.com/kaivalnp/0424bd84326aebdecd10f8144fb46c73 Now we can visualize the results at: https://jmh.morethan.io/?gist=0424bd84326aebdecd10f8144fb46c73 Also found this GH action that automatically runs and compares JMH output: https://github.com/benchmark-action/github-action-benchmark, might be interesting to add to Lucene! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
