[
https://issues.apache.org/jira/browse/GEODE-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378246#comment-17378246
]
Hale Bales commented on GEODE-9340:
-----------------------------------
this PR was created to resolve this issue by ignoring the performance
regression and moving the baseline to version 1.13.3. The jetty performance
degradation occurred after 9.4.34, so we moved the baseline to use a newer
version that also has the performance issue. This will result in fewer CI
failures, and follows our protocol of changing the benchmark baseline when a
new version is released.
I was never able to track down the exact cause of the failure. It seems to be
somewhere in jetty that we don't have access to through profiling. Given that
it is a small performance regression that is only detectable by our most
sensitive tests, ignoring the issue is a reasonable solution that has been
approved in conversations with other developers.
> Benchmark instability in PartitionedPutLongBenchmark
> ----------------------------------------------------
>
> Key: GEODE-9340
> URL: https://issues.apache.org/jira/browse/GEODE-9340
> Project: Geode
> Issue Type: Bug
> Components: benchmarks
> Affects Versions: 1.15.0
> Reporter: Sarah Abbey
> Assignee: Hale Bales
> Priority: Major
> Labels: pull-request-available
>
> PartitionedPutLongBenchmark failed in CI
> (https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/benchmark-base/builds/6):
> {code:java}
> This is ITERATION 1 of benchmarking against baseline.
> P2pPartitionedGetBenchmark avg ops/sec
> Baseline: 825011.38 Test: 835847.67 Difference: +1.3%
> avg latency
> Baseline: 871392.31 Test: 859444.66 Difference: -1.4%
> P2pPartitionedPutBenchmark avg ops/sec
> Baseline: 123838.43 Test: 122686.30 Difference: -0.9%
> avg latency
> Baseline: 6015719.73 Test: 6119472.19 Difference: +1.7%
> P2pPartitionedPutBytesBenchmark avg ops/sec
> Baseline: 174887.77 Test: 171040.93 Difference: -2.2%
> avg latency
> Baseline: 4145337.60 Test: 4236159.60 Difference: +2.2%
> PartitionedFunctionExecutionBenchmark avg ops/sec
> Baseline: 248635.36 Test: 261498.94 Difference: +5.2%
> avg latency
> Baseline: 867122.63 Test: 824550.34 Difference: -4.9%
> PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec
> Baseline: 280071.19 Test: 275305.31 Difference: -1.7%
> avg latency
> Baseline: 1026643.12 Test: 1044307.43 Difference: +1.7%
> PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec
> Baseline: 301416.23 Test: 304317.30 Difference: +1.0%
> avg latency
> Baseline: 1908390.88 Test: 1890040.46 Difference: -1.0%
> PartitionedGetBenchmark avg ops/sec
> Baseline: 790800.52 Test: 784514.74 Difference: -0.8%
> avg latency
> Baseline: 908357.58 Test: 915790.96 Difference: +0.8%
> PartitionedGetLongBenchmark avg ops/sec
> Baseline: 1020821.32 Test: 996529.93 Difference: -2.4%
> avg latency
> Baseline: 703761.09 Test: 720744.36 Difference: +2.4%
> PartitionedGetStringBenchmark avg ops/sec
> Baseline: 1028992.93 Test: 1010447.47 Difference: -1.8%
> avg latency
> Baseline: 698009.55 Test: 710765.29 Difference: +1.8%
> PartitionedIndexedQueryBenchmark avg ops/sec
> Baseline: 30868.78 Test: 31478.90 Difference: +2.0%
> avg latency
> Baseline: 18670093.21 Test: 18278083.16 Difference: -2.1%
> PartitionedNonIndexedQueryBenchmark avg ops/sec
> Baseline: 99.45 Test: 101.97 Difference: +2.5%
> avg latency
> Baseline: 723415530.75 Test: 705653061.86 Difference: -2.5%
> PartitionedPutAllBenchmark avg ops/sec
> Baseline: 7921.61 Test: 7816.66 Difference: -1.3%
> avg latency
> Baseline: 18172638.37 Test: 18416169.28 Difference: +1.3%
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1379.53 Test: 1169.16 Difference: -15.2%
> avg latency
> Baseline: 105140260.44 Test: 123722914.94 Difference: +17.7%
> PartitionedPutBenchmark avg ops/sec
> Baseline: 474986.11 Test: 467924.19 Difference: -1.5%
> avg latency
> Baseline: 1514276.31 Test: 1536263.99 Difference: +1.5%
> PartitionedPutBytesBenchmark avg ops/sec
> Baseline: 457550.69 Test: 456011.33 Difference: -0.3%
> avg latency
> Baseline: 1570713.84 Test: 1575841.02 Difference: +0.3%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 418221.79 Test: 389221.70 Difference: -6.9%
> avg latency
> Baseline: 1717869.66 Test: 1849602.96 Difference: +7.7%
> PartitionedPutStringBenchmark avg ops/sec
> Baseline: 410007.93 Test: 390442.31 Difference: -4.8%
> avg latency
> Baseline: 1754915.95 Test: 1841243.74 Difference: +4.9%
> ReplicatedFunctionExecutionBenchmark avg ops/sec
> Baseline: 478780.89 Test: 466855.62 Difference: -2.5%
> avg latency
> Baseline: 449616.85 Test: 461217.05 Difference: +2.6%
> ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec
> Baseline: 504979.76 Test: 501025.09 Difference: -0.8%
> avg latency
> Baseline: 2279046.16 Test: 2296973.89 Difference: +0.8%
> ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec
> Baseline: 523232.66 Test: 517943.28 Difference: -1.0%
> avg latency
> Baseline: 1374146.10 Test: 1387969.18 Difference: +1.0%
> ReplicatedGetBenchmark avg ops/sec
> Baseline: 799001.77 Test: 788077.23 Difference: -1.4%
> avg latency
> Baseline: 899092.08 Test: 911932.02 Difference: +1.4%
> ReplicatedGetLongBenchmark avg ops/sec
> Baseline: 1038120.92 Test: 1025267.47 Difference: -1.2%
> avg latency
> Baseline: 691970.73 Test: 700641.58 Difference: +1.3%
> ReplicatedIndexedQueryBenchmark avg ops/sec
> Baseline: 32088.81 Test: 31641.88 Difference: -1.4%
> avg latency
> Baseline: 17960402.47 Test: 18222972.89 Difference: +1.5%
> ReplicatedNonIndexedQueryBenchmark avg ops/sec
> Baseline: 49.31 Test: 48.15 Difference: -2.4%
> avg latency
> Baseline: 1456350672.39 Test: 1490270040.10 Difference: +2.3%
> ReplicatedPutAllBenchmark avg ops/sec
> Baseline: 821.20 Test: 793.66 Difference: -3.4%
> avg latency
> Baseline: 175498155.76 Test: 181712150.69 Difference: +3.5%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 855.75 Test: 812.41 Difference: -5.1%
> avg latency
> Baseline: 168343905.55 Test: 177510321.30 Difference: +5.4%
> ReplicatedPutBenchmark avg ops/sec
> Baseline: 485135.10 Test: 481879.66 Difference: -0.7%
> avg latency
> Baseline: 1482039.07 Test: 1491885.72 Difference: +0.7%
> ReplicatedPutLongBenchmark avg ops/sec
> Baseline: 454212.92 Test: 443628.38 Difference: -2.3%
> avg latency
> Baseline: 1583988.26 Test: 1620588.33 Difference: +2.3%
> This is ITERATION 2 of benchmarking against baseline.
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1359.77 Test: 1263.03 Difference: -7.1%
> avg latency
> Baseline: 106472640.40 Test: 114522255.46 Difference: +7.6%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 426518.85 Test: 400573.57 Difference: -6.1%
> avg latency
> Baseline: 1686679.27 Test: 1793189.27 Difference: +6.3%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 847.77 Test: 802.20 Difference: -5.4%
> avg latency
> Baseline: 170110197.18 Test: 179656637.86 Difference: +5.6%
> This is ITERATION 3 of benchmarking against baseline.
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1308.38 Test: 1371.30 Difference: +4.8%
> avg latency
> Baseline: 110742328.38 Test: 105657848.59 Difference: -4.6%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 422675.38 Test: 393034.80 Difference: -7.0%
> avg latency
> Baseline: 1701984.68 Test: 1830333.62 Difference: +7.5%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 882.11 Test: 868.74 Difference: -1.5%
> avg latency
> Baseline: 163447771.46 Test: 166032265.56 Difference: +1.6%
> This is ITERATION 4 of benchmarking against baseline.
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 421497.23 Test: 394141.73 Difference: -6.5%
> avg latency
> Baseline: 1705094.48 Test: 1823187.55 Difference: +6.9%
> This is ITERATION 5 of benchmarking against baseline.
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 410979.61 Test: 387125.91 Difference: -5.8%
> avg latency
> Baseline: 1749420.68 Test: 1854964.43 Difference: +6.0%
> {code}
> Potentially, there is some instability in this benchmark since it passed in
> the runs immediately before and after.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
