[
https://issues.apache.org/jira/browse/GEODE-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448314#comment-17448314
]
Donal Evans commented on GEODE-9340:
------------------------------------
Failures of this benchmark are due to the test being overly sensitive to minor
changes in performance, particularly those related to memory. The failure
threshold for benchmarks should probably be adjusted on a per-test basis so
that overly sensitive tests don't produce false positives.
> Benchmark instability in PartitionedPutLongBenchmark
> ----------------------------------------------------
>
> Key: GEODE-9340
> URL: https://issues.apache.org/jira/browse/GEODE-9340
> Project: Geode
> Issue Type: Bug
> Components: benchmarks
> Affects Versions: 1.15.0
> Reporter: Sarah Abbey
> Priority: Major
> Labels: blocks-1.15.0, pull-request-available
>
> PartitionedPutLongBenchmark failed in CI
> (https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/benchmark-base/builds/6):
> {code:java}
> This is ITERATION 1 of benchmarking against baseline.
> P2pPartitionedGetBenchmark avg ops/sec
> Baseline: 825011.38 Test: 835847.67 Difference: +1.3%
> avg latency
> Baseline: 871392.31 Test: 859444.66 Difference: -1.4%
> P2pPartitionedPutBenchmark avg ops/sec
> Baseline: 123838.43 Test: 122686.30 Difference: -0.9%
> avg latency
> Baseline: 6015719.73 Test: 6119472.19 Difference: +1.7%
> P2pPartitionedPutBytesBenchmark avg ops/sec
> Baseline: 174887.77 Test: 171040.93 Difference: -2.2%
> avg latency
> Baseline: 4145337.60 Test: 4236159.60 Difference: +2.2%
> PartitionedFunctionExecutionBenchmark avg ops/sec
> Baseline: 248635.36 Test: 261498.94 Difference: +5.2%
> avg latency
> Baseline: 867122.63 Test: 824550.34 Difference: -4.9%
> PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec
> Baseline: 280071.19 Test: 275305.31 Difference: -1.7%
> avg latency
> Baseline: 1026643.12 Test: 1044307.43 Difference: +1.7%
> PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec
> Baseline: 301416.23 Test: 304317.30 Difference: +1.0%
> avg latency
> Baseline: 1908390.88 Test: 1890040.46 Difference: -1.0%
> PartitionedGetBenchmark avg ops/sec
> Baseline: 790800.52 Test: 784514.74 Difference: -0.8%
> avg latency
> Baseline: 908357.58 Test: 915790.96 Difference: +0.8%
> PartitionedGetLongBenchmark avg ops/sec
> Baseline: 1020821.32 Test: 996529.93 Difference: -2.4%
> avg latency
> Baseline: 703761.09 Test: 720744.36 Difference: +2.4%
> PartitionedGetStringBenchmark avg ops/sec
> Baseline: 1028992.93 Test: 1010447.47 Difference: -1.8%
> avg latency
> Baseline: 698009.55 Test: 710765.29 Difference: +1.8%
> PartitionedIndexedQueryBenchmark avg ops/sec
> Baseline: 30868.78 Test: 31478.90 Difference: +2.0%
> avg latency
> Baseline: 18670093.21 Test: 18278083.16 Difference: -2.1%
> PartitionedNonIndexedQueryBenchmark avg ops/sec
> Baseline: 99.45 Test: 101.97 Difference: +2.5%
> avg latency
> Baseline: 723415530.75 Test: 705653061.86 Difference: -2.5%
> PartitionedPutAllBenchmark avg ops/sec
> Baseline: 7921.61 Test: 7816.66 Difference: -1.3%
> avg latency
> Baseline: 18172638.37 Test: 18416169.28 Difference: +1.3%
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1379.53 Test: 1169.16 Difference: -15.2%
> avg latency
> Baseline: 105140260.44 Test: 123722914.94 Difference: +17.7%
> PartitionedPutBenchmark avg ops/sec
> Baseline: 474986.11 Test: 467924.19 Difference: -1.5%
> avg latency
> Baseline: 1514276.31 Test: 1536263.99 Difference: +1.5%
> PartitionedPutBytesBenchmark avg ops/sec
> Baseline: 457550.69 Test: 456011.33 Difference: -0.3%
> avg latency
> Baseline: 1570713.84 Test: 1575841.02 Difference: +0.3%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 418221.79 Test: 389221.70 Difference: -6.9%
> avg latency
> Baseline: 1717869.66 Test: 1849602.96 Difference: +7.7%
> PartitionedPutStringBenchmark avg ops/sec
> Baseline: 410007.93 Test: 390442.31 Difference: -4.8%
> avg latency
> Baseline: 1754915.95 Test: 1841243.74 Difference: +4.9%
> ReplicatedFunctionExecutionBenchmark avg ops/sec
> Baseline: 478780.89 Test: 466855.62 Difference: -2.5%
> avg latency
> Baseline: 449616.85 Test: 461217.05 Difference: +2.6%
> ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec
> Baseline: 504979.76 Test: 501025.09 Difference: -0.8%
> avg latency
> Baseline: 2279046.16 Test: 2296973.89 Difference: +0.8%
> ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec
> Baseline: 523232.66 Test: 517943.28 Difference: -1.0%
> avg latency
> Baseline: 1374146.10 Test: 1387969.18 Difference: +1.0%
> ReplicatedGetBenchmark avg ops/sec
> Baseline: 799001.77 Test: 788077.23 Difference: -1.4%
> avg latency
> Baseline: 899092.08 Test: 911932.02 Difference: +1.4%
> ReplicatedGetLongBenchmark avg ops/sec
> Baseline: 1038120.92 Test: 1025267.47 Difference: -1.2%
> avg latency
> Baseline: 691970.73 Test: 700641.58 Difference: +1.3%
> ReplicatedIndexedQueryBenchmark avg ops/sec
> Baseline: 32088.81 Test: 31641.88 Difference: -1.4%
> avg latency
> Baseline: 17960402.47 Test: 18222972.89 Difference: +1.5%
> ReplicatedNonIndexedQueryBenchmark avg ops/sec
> Baseline: 49.31 Test: 48.15 Difference: -2.4%
> avg latency
> Baseline: 1456350672.39 Test: 1490270040.10 Difference: +2.3%
> ReplicatedPutAllBenchmark avg ops/sec
> Baseline: 821.20 Test: 793.66 Difference: -3.4%
> avg latency
> Baseline: 175498155.76 Test: 181712150.69 Difference: +3.5%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 855.75 Test: 812.41 Difference: -5.1%
> avg latency
> Baseline: 168343905.55 Test: 177510321.30 Difference: +5.4%
> ReplicatedPutBenchmark avg ops/sec
> Baseline: 485135.10 Test: 481879.66 Difference: -0.7%
> avg latency
> Baseline: 1482039.07 Test: 1491885.72 Difference: +0.7%
> ReplicatedPutLongBenchmark avg ops/sec
> Baseline: 454212.92 Test: 443628.38 Difference: -2.3%
> avg latency
> Baseline: 1583988.26 Test: 1620588.33 Difference: +2.3%
> This is ITERATION 2 of benchmarking against baseline.
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1359.77 Test: 1263.03 Difference: -7.1%
> avg latency
> Baseline: 106472640.40 Test: 114522255.46 Difference: +7.6%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 426518.85 Test: 400573.57 Difference: -6.1%
> avg latency
> Baseline: 1686679.27 Test: 1793189.27 Difference: +6.3%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 847.77 Test: 802.20 Difference: -5.4%
> avg latency
> Baseline: 170110197.18 Test: 179656637.86 Difference: +5.6%
> This is ITERATION 3 of benchmarking against baseline.
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1308.38 Test: 1371.30 Difference: +4.8%
> avg latency
> Baseline: 110742328.38 Test: 105657848.59 Difference: -4.6%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 422675.38 Test: 393034.80 Difference: -7.0%
> avg latency
> Baseline: 1701984.68 Test: 1830333.62 Difference: +7.5%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 882.11 Test: 868.74 Difference: -1.5%
> avg latency
> Baseline: 163447771.46 Test: 166032265.56 Difference: +1.6%
> This is ITERATION 4 of benchmarking against baseline.
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 421497.23 Test: 394141.73 Difference: -6.5%
> avg latency
> Baseline: 1705094.48 Test: 1823187.55 Difference: +6.9%
> This is ITERATION 5 of benchmarking against baseline.
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 410979.61 Test: 387125.91 Difference: -5.8%
> avg latency
> Baseline: 1749420.68 Test: 1854964.43 Difference: +6.0%
> {code}
> Potentially, there is some instability in this benchmark since it passed in
> the runs immediately before and after.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
