> I have to apologise here. CircleCI did not uncover these problems, apparently > due to some way it resolves dependencies,
I double checked your CircleCI run for the trunk branch, and the problem doesn’t have to do with “resolves dependencies”, the problem lies with our CI being too complex and doesn’t natively support multi-branch commits. Right now you need to opt-in to 2 builds to run the single jvm-dtest upgrade test build (missed in your CI); this should not be opt-in (see my previous comment about this), and it really shouldn’t be 2 approvals for a single build… Enabling “upgrade tests” does not run all the upgrade tests… you need to approve 2 other builds to run the full set of upgrade tests (see problem above). I see in the build you ran the upgrade tests, which only touches the python-dtest upgrade tests Lastly, you need to hack the circleci configuration to support multi-branch CI, if you do not it will run against w/e is already committed to 2.2, 3.0, 3.11, and 4.0. Multi-branch commits are very normal for our project, but doing CI properly in these cases is way too hard (you can not do multi-branch tests in Jenkins https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test/build <https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test/build>; no support to run against your other branches). > On Nov 1, 2021, at 3:03 PM, David Capwell <dcapw...@apple.com.INVALID> wrote: > >> How do we define what "releasable trunk" means? > > One thing I would love is for us to adopt a “run all tests needed to release > before commit” mentality, and to link a successful run in JIRA when closing > (we talked about this once in slack). If we look at CircleCI we currently do > not run all the tests needed to sign off; below are the tests disabled in the > “pre-commit” workflows (see > https://github.com/apache/cassandra/blob/trunk/.circleci/config-2_1.yml#L381): > > start_utests_long > start_utests_compression > start_utests_stress > start_utests_fqltool > start_utests_system_keyspace_directory > start_jvm_upgrade_dtest > start_upgrade_tests > > Given the configuration right now we have to opt-in to upgrade tests, but we > can’t release if those are broken (same for compression/fqltool/cdc (not > covered in circle)). > >> On Oct 30, 2021, at 6:24 AM, bened...@apache.org wrote: >> >>> How do we define what "releasable trunk" means? >> >> For me, the major criteria is ensuring that work is not merged that is known >> to require follow-up work, or could reasonably have been known to require >> follow-up work if better QA practices had been followed. >> >> So, a big part of this is ensuring we continue to exceed our targets for >> improved QA. For me this means trying to weave tools like Harry and the >> Simulator into our development workflow early on, but we’ll see how well >> these tools gain broader adoption. This also means focus in general on >> possible negative effects of a change. >> >> I think we could do with producing guidance documentation for how to >> approach QA, where we can record our best practices and evolve them as we >> discover flaws or pitfalls, either for ergonomics or for bug discovery. >> >>> What are the benefits of having a releasable trunk as defined here? >> >> If we want to have any hope of meeting reasonable release cadences _and_ the >> high project quality we expect today, then I think a ~shippable trunk policy >> is an absolute necessity. >> >> I don’t think means guaranteeing there are no failing tests (though ideally >> this would also happen), but about ensuring our best practices are followed >> for every merge. 4.0 took so long to release because of the amount of hidden >> work that was created by merging work that didn’t meet the standard for >> release. >> >> Historically we have also had significant pressure to backport features to >> earlier versions due to the cost and risk of upgrading. If we maintain >> broader version compatibility for upgrade, and reduce the risk of adopting >> newer versions, then this pressure is also reduced significantly. Though >> perhaps we will stick to our guns here anyway, as there seems to be renewed >> pressure to limit work in GA releases to bug fixes exclusively. It remains >> to be seen if this holds. >> >>> What are the costs? >> >> I think the costs are quite low, perhaps even negative. Hidden work produced >> by merges that break things can be much more costly than getting the work >> right first time, as attribution is much more challenging. >> >> One cost that is created, however, is for version compatibility as we cannot >> say “well, this is a minor version bump so we don’t need to support >> downgrade”. But I think we should be investing in this anyway for operator >> simplicity and confidence, so I actually see this as a benefit as well. >> >>> Full disclosure: running face-first into 60+ failing tests on trunk >> >> I have to apologise here. CircleCI did not uncover these problems, >> apparently due to some way it resolves dependencies, and so I am responsible >> for a significant number of these and have been quite sick since. >> >> I think a push to eliminate flaky tests will probably help here in future, >> though, and perhaps the project needs to have some (low) threshold of flaky >> or failing tests at which point we block merges to force a correction. >> >> >> From: Joshua McKenzie <jmcken...@apache.org> >> Date: Saturday, 30 October 2021 at 14:00 >> To: dev@cassandra.apache.org <dev@cassandra.apache.org> >> Subject: [DISCUSS] Releasable trunk and quality >> We as a project have gone back and forth on the topic of quality and the >> notion of a releasable trunk for quite a few years. If people are >> interested, I'd like to rekindle this discussion a bit and see if we're >> happy with where we are as a project or if we think there's steps we should >> take to change the quality bar going forward. The following questions have >> been rattling around for me for awhile: >> >> 1. How do we define what "releasable trunk" means? All reviewed by M >> committers? Passing N% of tests? Passing all tests plus some other metrics >> (manual testing, raising the number of reviewers, test coverage, usage in >> dev or QA environments, etc)? Something else entirely? >> >> 2. With a definition settled upon in #1, what steps, if any, do we need to >> take to get from where we are to having *and keeping* that releasable >> trunk? Anything to codify there? >> >> 3. What are the benefits of having a releasable trunk as defined here? What >> are the costs? Is it worth pursuing? What are the alternatives (for >> instance: a freeze before a release + stabilization focus by the community >> i.e. 4.0 push or the tock in tick-tock)? >> >> Given the large volumes of work coming down the pike with CEP's, this seems >> like a good time to at least check in on this topic as a community. >> >> Full disclosure: running face-first into 60+ failing tests on trunk when >> going through the commit process for denylisting this week brought this >> topic back up for me (reminds me of when I went to merge CDC back in 3.6 >> and those test failures riled me up... I sense a pattern ;)) >> >> Looking forward to hearing what people think. >> >> ~Josh > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org >
