> How do we define what "releasable trunk" means? One thing I would love is for us to adopt a “run all tests needed to release before commit” mentality, and to link a successful run in JIRA when closing (we talked about this once in slack). If we look at CircleCI we currently do not run all the tests needed to sign off; below are the tests disabled in the “pre-commit” workflows (see https://github.com/apache/cassandra/blob/trunk/.circleci/config-2_1.yml#L381):
start_utests_long start_utests_compression start_utests_stress start_utests_fqltool start_utests_system_keyspace_directory start_jvm_upgrade_dtest start_upgrade_tests Given the configuration right now we have to opt-in to upgrade tests, but we can’t release if those are broken (same for compression/fqltool/cdc (not covered in circle)). > On Oct 30, 2021, at 6:24 AM, bened...@apache.org wrote: > >> How do we define what "releasable trunk" means? > > For me, the major criteria is ensuring that work is not merged that is known > to require follow-up work, or could reasonably have been known to require > follow-up work if better QA practices had been followed. > > So, a big part of this is ensuring we continue to exceed our targets for > improved QA. For me this means trying to weave tools like Harry and the > Simulator into our development workflow early on, but we’ll see how well > these tools gain broader adoption. This also means focus in general on > possible negative effects of a change. > > I think we could do with producing guidance documentation for how to approach > QA, where we can record our best practices and evolve them as we discover > flaws or pitfalls, either for ergonomics or for bug discovery. > >> What are the benefits of having a releasable trunk as defined here? > > If we want to have any hope of meeting reasonable release cadences _and_ the > high project quality we expect today, then I think a ~shippable trunk policy > is an absolute necessity. > > I don’t think means guaranteeing there are no failing tests (though ideally > this would also happen), but about ensuring our best practices are followed > for every merge. 4.0 took so long to release because of the amount of hidden > work that was created by merging work that didn’t meet the standard for > release. > > Historically we have also had significant pressure to backport features to > earlier versions due to the cost and risk of upgrading. If we maintain > broader version compatibility for upgrade, and reduce the risk of adopting > newer versions, then this pressure is also reduced significantly. Though > perhaps we will stick to our guns here anyway, as there seems to be renewed > pressure to limit work in GA releases to bug fixes exclusively. It remains to > be seen if this holds. > >> What are the costs? > > I think the costs are quite low, perhaps even negative. Hidden work produced > by merges that break things can be much more costly than getting the work > right first time, as attribution is much more challenging. > > One cost that is created, however, is for version compatibility as we cannot > say “well, this is a minor version bump so we don’t need to support > downgrade”. But I think we should be investing in this anyway for operator > simplicity and confidence, so I actually see this as a benefit as well. > >> Full disclosure: running face-first into 60+ failing tests on trunk > > I have to apologise here. CircleCI did not uncover these problems, apparently > due to some way it resolves dependencies, and so I am responsible for a > significant number of these and have been quite sick since. > > I think a push to eliminate flaky tests will probably help here in future, > though, and perhaps the project needs to have some (low) threshold of flaky > or failing tests at which point we block merges to force a correction. > > > From: Joshua McKenzie <jmcken...@apache.org> > Date: Saturday, 30 October 2021 at 14:00 > To: dev@cassandra.apache.org <dev@cassandra.apache.org> > Subject: [DISCUSS] Releasable trunk and quality > We as a project have gone back and forth on the topic of quality and the > notion of a releasable trunk for quite a few years. If people are > interested, I'd like to rekindle this discussion a bit and see if we're > happy with where we are as a project or if we think there's steps we should > take to change the quality bar going forward. The following questions have > been rattling around for me for awhile: > > 1. How do we define what "releasable trunk" means? All reviewed by M > committers? Passing N% of tests? Passing all tests plus some other metrics > (manual testing, raising the number of reviewers, test coverage, usage in > dev or QA environments, etc)? Something else entirely? > > 2. With a definition settled upon in #1, what steps, if any, do we need to > take to get from where we are to having *and keeping* that releasable > trunk? Anything to codify there? > > 3. What are the benefits of having a releasable trunk as defined here? What > are the costs? Is it worth pursuing? What are the alternatives (for > instance: a freeze before a release + stabilization focus by the community > i.e. 4.0 push or the tock in tick-tock)? > > Given the large volumes of work coming down the pike with CEP's, this seems > like a good time to at least check in on this topic as a community. > > Full disclosure: running face-first into 60+ failing tests on trunk when > going through the commit process for denylisting this week brought this > topic back up for me (reminds me of when I went to merge CDC back in 3.6 > and those test failures riled me up... I sense a pattern ;)) > > Looking forward to hearing what people think. > > ~Josh --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
