+1 to creating such a module. It would definitely be useful and valuable. +1 to having it live in the side car. Though implementing it will of course need changes in the server as well to support the features the sidecar needs.
Not sure we would get to the case of only a single restart on the new version being required for all major upgrades, but it is a good end goal to have. I would also like us to get to the point where it’s possible to do upgrades/changes in a way that it s possible to rollback from them, but that is probably beyond the scope of the initial implementation here. -Jeremiah Jordan On Jun 11, 2025 at 8:54:02 AM, Josh McKenzie <jmcken...@apache.org> wrote: > +1 that this is something that would be very valuable to our users. > > As to where it lives, my gut reaction is to have the coordination, > stop/start, etc live inside the sidecar and lean on the DB for the durable > distributed lock to make the process atomic. Ultimately I'd like to see us > move to the sidecar as control plane (see CEP-1 > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224>), > so having this functionality inside the sidecar would fit nicely with some > of the other things we discussed. i.e.: > > 1. Stop nodes safely > 2. Start nodes safely > 3. Rolling restart a cluster safely w/respect to racks and availability > 4. Plan and execute an upgrade > > c) Performing capability limitation during an upgrade > > +1 to this as well; we never did get to consensus re: the capabilities > framework did we? Think that kind of fizzled / got derailed on whether we > should use TCM as a durable delivery mechanism for capabilities or not > (ponymail > link <https://lists.apache.org/thread/9oztm7o1czky3tywp3mxskb529nrn2wv>). > > On Wed, Jun 11, 2025, at 2:34 AM, Paulo Motta wrote: > > Hi, > > One of the most important operational features of Cassandra is how easy it > is (or should be) to do an in-place upgrade. The in-place upgrade procedure > essentially consists of rolling-restarting the cluster while updating the > jar to the new version, while following additional upgrade instructions > from NEWS.txt. In practice, as new features are added and existing features > are extended, the upgrade procedure gets more complex, placing more burden > on operators to ensure a smooth upgrade process. > > For example, updating the storage_compatibility_mode from 4.0 to 5.0 > requires 3 cluster-wide restarts[1]. Another example is that upgrading to > Cassandra 6.0 prohibits operations like schema changes, node replacement, > bootstrap, decommission, move, assassinate before all the nodes are > migrated to CMS[2]. I don't want to focus on these particular examples, > this is just to illustrate that a lot of manual steps and caution is > required to perform in-place upgrades safely and smoothly. > > In order to improve this, I would like to propose extending Cassandra to > allow an operator to register an upgrade intent with the goals of: > a) Tracking the upgrade progress in a system table > b) Verifying the correctness and improving the safety of the upgrade > process > c) Performing capability limitation during an upgrade > d) Perform pre and post upgrade actions automatically, when registered in > the upgrade plan by the operator > > While there is upgrade awareness in the server, it is mostly reactive and > scattered across different modules (as far as I last seen). A potential > side goal of this effort is to centralize upgrade handling code from > different features in the same module, allowing different features to > specify upgrade pre/post actions/conditions more uniformly. This would > allow for example, developers to specify upgrade constraints via testable > code instead of notes in NEWS.txt, with the hope they will be read by a > careful operator. > The upgrade plan would be registered in a system table and tracked by an > upgrade manager module, that would prevent certain operations (ie. range > movements/schema changes) when an upgrade plan is active or emit > errors/warnings when anomalies are encountered. A few safety/usability > improvements can be enabled when the upgrade plan is registered in the > server, among others: > a) A node could fail startup if it tries to start in a version different > from the one specified in the currently active upgrade plan. > b) If a latency degradation or other SLO degradation is detected while an > upgrade plan is active, then warnings could be emitted allowing operators > to more easily detect upgrade issues. > c) When the upgrade is determined to be completed successfully, nodes can > coordinate running upgrade-sstables or other post-operations according to a > policy specified in the upgrade plan (ie. by rack/dc). > > To give an example of what the API would look like, a user wishing upgrade > to upgrade a cluster from version 4.1 to 5.0 would register the upgrade > intent via an API, ie.: nodetool upgradeplan create --target 5.0.4 > --disable-schema-changes --post-action upgrade-sstables --post-action > upgrade-storage-compatibility-mode. It would not be possible to create > another upgrade plan if there's a current in progress. > > The ultimate goal is that the upgrade process to any version will be as > simple as registering an upgrade plan, and performing a cluster rolling > restart in the desired target version. Any additional actions would be > autonomously coordinated by the servers based on the upgrade progress and > according to the preferences specified in the upgrade plan. > > A related, and probably broader, topic is upgrading features. A couple of > examples that come to mind are upgrading Paxos[2] or migrating to > incremental repair[3]. Like version upgrades, these feature upgrades > require a series of steps to be executed on a determined order and > sometimes global coordination. While this suggestion focuses on version > upgrades, it can potentially be extended to track feature upgrades. > I would appreciate your feedback on this draft suggestion to check if it > makes sense before elaborating it on a more detailed proposal, as well as > pointers to other efforts or past proposals that might be related to this. > Thanks, > Paulo > > [1] - https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L15-L21 > [2] - > https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L142C1-L148C19 > [3] - https://lists.apache.org/thread/06bl99mt502k7lowd5ont9jtnf5p0t05 > > >
