[DISCUSS] reduce PR checks to JDK11 only
Currently every PR commit triggers both JDK8 and JDK11 versions of each test job. I propose that we can eliminate the JDK8 version of each check. In the extremely rare case where a code change breaks on Java 8 but works fine on Java 11, it would still be caught by the main pipeline (just as Windows failures are caught only in the main pipeline). The only tangible effect today of running both JDK8 and JDK11 tests in PR pipeline is twice the chance to encounter possible flaky failures (usually unrelated to the commit itself).
[DISCUSS] run Windows tests concurrently with Linux tests in main pipeline
Currently we wait to run the Windows tests for a commit until all the Linux tests have passed. This results in unnecessarily long feedback time to know if a commit passed on Windows. The only benefit of staggering is a small efficiency in resource usage when a breaking commit is merged to develop, but optimizing for failure seems unproductive when our intent is to maintain a healthy CI pipeline. On the other hand if we want to embrace staggering to optimize resource usage on failure, then we should also split up JDK8 and JDK11 into two stages rather than run them concurrently...
[DISCUSS] is it time to make Windows tests gating?
For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all. Now seems like a good time to review whether to: a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows b) keep as-is c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only" One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).
Re: [DISCUSS] is it time to make Windows tests gating?
They are also as red as the day is long... On Wed, May 15, 2019, 18:28 Owen Nichols wrote: > For a very long time we’ve had Windows tests in the main pipeline (hidden > away, not in the default view), but the pipeline proceeds to publish > regardless of whether Windows tests fail or even run at all. > > Now seems like a good time to review whether to: > a) treat Windows tests as first-class tests and prevent the pipeline from > proceeding if any test fails on Windows > b) keep as-is > c) change Windows tests to trigger only once a week rather than on every > commit, if they are going to remain "informational only" > > One disadvantage to making Windows tests gating is that they currently > take much longer to run (around 5 hours, vs 2 hours for Linux tests).
