Yes, I'm still working on JMXMBeanReconnectDUnitTest. The test fails intermittently due to a couple product bugs that the test found. I've filed separate tickets for these bugs and I expect the test's flakiness to be fixed when the bugs have been addressed.
-Kirk On Tue, Jun 30, 2020 at 2:57 PM Alexander Murmann <amurm...@apache.org> wrote: > Hi everyone, > > Just like Mark did two weeks ago, I'd like to bring some attention to our > mass test runs. These now run in the pipeline > < > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run > > > once a week. The results should typically be available on Tuesdays. > > *Context* > Some context since we only recently started running these in our pipeline: > In the past we've seen the number of flaky tests, in particular among our > DUnite tests, grow over time. This would eventually lead to death by > thousand paper cuts when trying to merge a PR. The increased noise in our > test suite also gives room for new signals about actual bugs to hide in. In > past efforts to combat the flakiness of our test suite we have found that > running the tests many times against a non-broken SHA gave us great data to > identify which tests contribute most flakiness to the test suite. > > *How does this week look?* > This week > < > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/8 > > > doesn't look much different than last week. We got a 78% pass-rate > > The test with the single highest fail rate > remains org.apache.geode.management.JMXMBeanReconnectDUnitTest with a fail > rate of 4.5%. I believe Kirk had been working on this for a prolonged time. > Kirk, is this still on your radar? > > The other big issue remains a cluster of issues around WAN Gateway that > caused 18 of the total 44 failures we saw in our 200 runs > I believe the WAN test issues got introduced fairly recently by GEODE-7458. > > If we got just the JMXMBean issue and the WAN tests addressed, our fail > rate should go back to well above 90% which would be a big win >
