Good point Jordan re: flaky test being either implying API instability or blocker to ability to beta test.
On Thu, May 28, 2020 at 12:56 PM Jordan West <jw...@apache.org> wrote: > > On Wed, May 27, 2020 at 5:13 PM Ekaterina Dimitrova < > > ekaterina.dimitr...@datastax.com> wrote: > > > - No flaky tests according to Jenkins or CircleCI? Also, some people run > > > the free tier, others take advantage of premium CircleCI. What should > be > > > the framework? > > > While I agree that we should use the Apache infrastructure as the canonical > infrastructure, failures in both (or any) environment matter when it comes > to flaky tests. > > On Wed, May 27, 2020 at 5:23 PM Joshua McKenzie <jmcken...@apache.org> > wrote: > > > > > At least for me, what I learned in the past is we'd drive to a green test > > board and immediately transition it as a milestone, so flaky tests would > > reappear like a disappointing game of whack-a-mole. They seem > frustratingly > > ever-present. > > > > > Agreed. Having multiple successive green runs would be a better bar than > one on a single platform imo. > > > > I'd personally advocate for us taking the following stance on flaky tests > > from this point in the cycle forward: > > > > - Default posture to label fix version as beta > > - *excepting* on case-by-case basis, if flake could imply product > defect > > that would greatly impair beta testing we leave alpha > > > > I would be in favor of tightening this further to flakes that imply > interface changes or major defects (e.g. corruption, data loss, etc). To do > so would require evaluation of the flaky test, however, which I think is in > sync with our "start in alpha and make exceptions to move to beta". The > difference would be that we better define and widen what flaky tests can be > punted to beta and my guess is we could already evaluate all outstanding > flaky test tickets by that bar. > > Jordan >
