This is the most recent fix on develop for the benchmark issues. Also tends
to indicate that the tests are flaky.
commit 278c2470a5cac2c332d13914b935f0618b820a91
Author: Helena Bales <hba...@pivotal.io>
Date: Mon Dec 16 11:16:55 2019 -0800
GEODE-7554: Add retry mechanism for failed tests (#4461)
* GEODE-7554: Add retry mechanism for failed tests
There are still some flaky benchmarks. Retry tests that have failed, up
to 5 times, to determine if the failure is legitimate or just a flaky
test. If the test fails 5 times in a row, we know that it is a
legitimate failure.
On Fri, Dec 27, 2019 at 1:44 PM Dan Smith <dsm...@pivotal.io> wrote:
> On Fri, Dec 27, 2019 at 1:28 PM Owen Nichols <onich...@pivotal.io> wrote:
>
>> It looks like the Benchmark <
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-release-1-11-0-main/jobs/Benchmark/builds/11>
>> job in the release pipeline has failed for the last 4 runs. At various
>> times it has flagged PartitionedGet, PartitionedPutAllLong,
>> PartitionedFunctionExecution, PartitionedNonIndexedQuery, and
>> ReplicatedPutAllBenchmark (2x), but there is no discernible pattern, so
>> maybe this is random noise?
>>
>> Random noise is what it looks like to me. Those same tests you mention
> look better than 1.10 in some runs and worse in others. We make the
> benchmark job on develop not block other jobs recently because it was too
> noisy.
>
