@Kirk, I share your concern. If the builds are RED then we should
address them. Ideally there should be no flaky tests, but given the
hyper-threaded nature of the beast, things can go wrong that are really
hard to reproduce. Arguably, even those scenarios should be addressed,
but if we wait for the perfect alignment of the planets to have a
complete green run, then we would never release.
That said. I've accepted, that a clean run of all categories, other than
flaky, to mean we had a "good" run. It would be awesome to collect some
metrics (counts,etc.. ), on a weekly,monthly basis, about failing test
(per category).
This way, we could hopefully identify "burning" failing tests, that need
to be addressed with urgency.
--Udo
On 5/25/17 09:00, Anthony Baker wrote:
This is topic is very relevant given that we are working towards a 1.2.0
release. In order to release we need—at the very minimum--a clean pass through
distributedTest.
Recently we’ve had some instability that has contributed to failing tests.
Anthony
On May 24, 2017, at 4:54 PM, Kirk Lund <kl...@apache.org> wrote:
If other people are ignoring the results of the flakyTest target and that's
the reason that people do not want to add FlakyTest category to tests that
aren't currently marked with FlakyTest then here's my proposal... we delete
FlakyTest category (well actually just rename it) and replace it with
TestsThatFailMoreFrequentlyAndNeedToRunInFreshJVM category and we educate
all geode developers to NOT ignore the results. If the test keeps failing
when run in isolation under
"TestsThatFailMoreFrequentlyAndNeedToRunInFreshJVM" then that means it
needs to be fixed asap and not ignored. Thoughts?
On Wed, May 24, 2017 at 4:50 PM, Kirk Lund <kl...@apache.org> wrote:
Geode Nightly build is staying consistently RED... each night the tests
that fail vary some. Do we still want to not add FlakyTest category to
these tests and just live with a RED nightly build?
What do you all want to do? Any ideas what to do to get it consistently
GREEN?
The only way I know to get it GREEN is either a) we drop everything we're
working on and fix flaky tests or b) we add FlakyTest category to flaky
tests. The flakyTest target still runs the FlakyTest tests but it does so
with a fresh JVM. In my opinion we should NOT ignore the results of
flakyTest (and I personally do not ignore these results). I think it's a
useful way to run DUnit tests that have special needs.
