> On Wed, May 27, 2020 at 5:13 PM Ekaterina Dimitrova < > ekaterina.dimitr...@datastax.com> wrote:
> - No flaky tests according to Jenkins or CircleCI? Also, some people run > > the free tier, others take advantage of premium CircleCI. What should be > > the framework? While I agree that we should use the Apache infrastructure as the canonical infrastructure, failures in both (or any) environment matter when it comes to flaky tests. On Wed, May 27, 2020 at 5:23 PM Joshua McKenzie <jmcken...@apache.org> wrote: > > At least for me, what I learned in the past is we'd drive to a green test > board and immediately transition it as a milestone, so flaky tests would > reappear like a disappointing game of whack-a-mole. They seem frustratingly > ever-present. > > Agreed. Having multiple successive green runs would be a better bar than one on a single platform imo. > I'd personally advocate for us taking the following stance on flaky tests > from this point in the cycle forward: > > - Default posture to label fix version as beta > - *excepting* on case-by-case basis, if flake could imply product defect > that would greatly impair beta testing we leave alpha > I would be in favor of tightening this further to flakes that imply interface changes or major defects (e.g. corruption, data loss, etc). To do so would require evaluation of the flaky test, however, which I think is in sync with our "start in alpha and make exceptions to move to beta". The difference would be that we better define and widen what flaky tests can be punted to beta and my guess is we could already evaluate all outstanding flaky test tickets by that bar. Jordan
