Re: Downgradability

[Benedict]

Forget downgradeability for a moment: we should not be breaking format compatibility without good reason. Bumping a major version isn’t enough of a reason.  Can somebody explain to me why this is being fought tooth and nail, when the work involved is absolutely minimal? Regarding tests: what more do you want, than running our upgrade suite backwards? On 23 Feb 2023, at 09:45, Benjamin Lerer <ble...@apache.org> wrote:  Can somebody explain to me what is so burdensome, that we seem to be spending longer debating it than it would take to implement the necessary changes? I believe that we all agree on the outcome. Everybody wants downgradability. The issue is on the path to get there. As far as I am concerned, I would like to see a proper solution and as Jeff suggested the equivalent of the upgrade tests as gatekeepers. Having everybody trying to enforce it on his own way will only lead to a poor result in my opinion with some addoc code that might not really guarantee real downgradability in the end. We have rushed in the past to get feature outs and pay the price for it. I simply prefer that we take the time to do things right. Thanks to Scott and you, downgradability got a much better visibility so no matter what approach we pick, I am convinced that we will get there. Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev <dev@cassandra.apache.org> a écrit : Broken downgrading can be fixed (I think) by modifying the SearializationHeader.toHeader() method where it currently throws an UnknownColumnException.  If we can, instead of throwing the exception, create a dropped column for the unexpected column then I think the code will work. I realise that to do this in the wild is not possible as it is a change to released code, but we could handle it going forward. On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo <henrik.i...@datastax.com> wrote: ... ok apparently shift+enter  sends messages now? I was just saying if at least the file format AND system/tables - anything written to disk - can be protected with a switch, then it allows for quick downgrade by shutting down the entire cluster and restarting with the downgraded binary. It's a start. To be able to do that live in a distributed system needs to consider much more: gossip, streaming, drivers, and ultimately all features, because we don't' want an application developer to use a shiny new thing that a) may not be available on all nodes, or b) may disappear if the cluster has to be downgraded later. henrik On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo <henrik.i...@datastax.com> wrote: Just this once I'm going to be really brief :-) Just wanted to share for reference how Mongodb implemented downgradeability around their 4.4 version: https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/ Jeff you're right. Ultimately this is about more than file formats. However, ideally if at least the On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jji...@gmail.com> wrote: I'm not even convinced even 8110 addresses this - just writing sstables in old versions won't help if we ever add things like new types or new types of collections without other control abilities. Claude's other email in another thread a few hours ago talks about some of these surprises - "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added to system/local.  This means that 3.1 system can not read the table as it has no definition for it.  I tried marking the column for deletion in the metadata and in the serialization header.  The later got past the column not found problem, but I suspect that it just means that data columns after broadcast_port shifted and so incorrectly read." - this is a harder problem to solve than just versioning sstables and network protocols.  Stepping back a bit, we have downgrade ability listed as a goal, but it's not (as far as I can tell) universally enforced, nor is it clear at which point we will be able to concretely say "this release can be downgraded to X".   Until we actually define and agree that this is a real goal with a concrete version where downgrade-ability becomes real, it feels like things are somewhat arbitrarily enforced, which is probably very frustrating for people trying to commit work/tickets. - Jeff On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <djo...@apache.org> wrote: I’m a big fan of maintaining backward compatibility. Downgradability implies that we could potentially roll back an upgrade at any time. While I don’t think we need to retain the ability to downgrade in perpetuity it would be a good objective to maintain strict backward compatibility and therefore downgradability until a certain point. This would imply versioning metadata and extending it in such a way that prior version(s) could continue functioning. This can certainly be expensive to implement and might bloat on-disk storage. However, we could always offer an option for the operator to optimize the on-disk structures for the current version then we can rewrite them in the latest version. This optimizes the storage and opens up new functionality. This means new features that can work with old on-disk structures will be available while others that strictly require new versions of the data structures will be unavailable until the operator migrates to the new version. This migration IMO should be irreversible. Beyond this point the operator will lose the ability to downgrade which is ok. Dinesh On Feb 20, 2023, at 10:40 AM, Jake Luciani <jak...@gmail.com> wrote:  There has been progress on  https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928 Which is similar to what datastax does for DSE. Would this be an acceptable solution? Jake  On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cclive1...@gmail.com> wrote: It seems “An alternative solution is to implement/complete CASSANDRA-8110” can give us more options if it is finished😉 Branimir Lambov <blam...@apache.org>于2023年2月20日 周一下午11:03写道： Hi everyone, There has been a discussion lately about changes to the sstable format in the context of being able to abort a cluster upgrade, and the fact that changes to sstables can prevent downgraded nodes from reading any data written during their temporary operation with the new version. Most of the discussion is in CASSANDRA-18134, and is spreading into CASSANDRA-14277 and CASSANDRA-17698, none of which is a good place to discuss the topic seriously. Downgradability is a worthy goal and is listed in the current roadmap. I would like to open a discussion here on how it would be achieved. My understanding of what has been suggested so far translates to: - avoid changes to sstable formats; - if there are changes, implement them in a way that is backwards-compatible, e.g. by duplicating data, so that a new version is presented in a component or portion of a component that legacy nodes will not try to read; - if the latter is not feasible, make sure the changes are only applied if a feature flag has been enabled. To me this approach introduces several risks: - it bloats file and parsing complexity; - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF ticket once this requirement is in place); - it needs care to avoid risky solutions to address technical issues with the format versioning (e.g. staying on n-versions for 5.0 and needing a bump for a 4.1 bugfix might require porting over support for new features); - it requires separate and uncoordinated solutions to the problem and switching mechanisms for each individual change. An alternative solution is to implement/complete CASSANDRA-8110, which provides a method of writing sstables for a target version. During upgrades, a node could be set to produce sstables corresponding to the older version, and there is a very straightforward way to implement modifications to formats like the tickets above to conform to its requirements.  What do people think should be the way forward? Regards, Branimir -- you are the apple of my eye ! -- http://twitter.com/tjake -- Henrik Ingo c. +358 40 569 7354  w. www.datastax.com       -- Henrik Ingo c. +358 40 569 7354  w. www.datastax.com