Sean Christopherson <[email protected]> writes: > On Fri, Aug 21, 2026, Pratyush Yadav wrote: >> Hi Sean, >> >> On Tue, Aug 18 2026, Sean Christopherson wrote: >> >> Does this idea of "backwards compatibility" sound acceptable to you, at >> >> least at a high level? >> > >> > No. >> > >> > It's probably fine for Google and other large companies that tightly >> > control their >> > kernels and use cases, and have the resources to juggle the resulting >> > complexity, >> > e.g. have kernel engineers on staff to track feature and dependencies, >> > coordinate >> > and plan kernel upgrades, etc. >> > >> > It's not acceptable for upstream, where downstream consumers often run a >> > distro >> > kernel, have much more varied use cases, and don't always have a horde of >> > kernel >> > engineers on staff to help them thread the needle you describe above. And >> > if >> > supporting live update as a general feature for all users of the kernel >> > isn't >> > being factored into design considerations, then that needs to change, >> > otherwise >> > this is all dead in the water. >> > >> > I also don't see the point. Maintaining a rigid save/restore ABI is >> > annoying, >> > but it's not _hard_ (or at least, not _that_ hard), especially if there's >> > a set >> > of well-documented best known practices that subsystems can follow, e.g. >> > so that >> > individual subsystems don't need to learn painful lessons first-hand. I >> > genuinely >> > believe that maintaining the version hell you describe above would be more >> > costly >> > in the long run than simply committing to full backwards compatibility >> > within a >> > given subsystem. I can imagine that enumerating what subsystems' >> > information is >> > in the payload will require a different scheme, but for a given subsystem, >> > I don't >> > see any reason to aim for anything less than full backwards compatibility. >> >> Let's say for argument's sake that we commit for a fully stable >> backwards compatible ABI. Even then, you have to deal with multiple ABI >> versions. >> >> Live update's ABI is more complex compared to KVM's save/restore ABI. >> For the KVM save/restore uAPI, you are largely describing architectural >> state like CPU registers, etc. These things don't evolve as fast and >> more or less stay the same. > > Right, because nothing meaningful has changed in any architecture in the 20+ > years since KVM has provided save/restore support, whereas guest_memfd looks > nothing like it did when it was introduced three years ago. > >> Live update needs to describe the state of kernel objects. These are >> more complex > > LOL, you might be the first to claim x86 virtualization isn't all that > complex. > >> and evolve faster. > > The speed at which things change doesn't automatically mean we shouldn't > strive > for backwards compatibility. Yes, providing backwards compatibility requires > additional care and planning, and over time *might* lead to an ABI that is > difficult to maintain. But IMO, that just makes it all the more important to > get the design right the first time, not that we shouldn't even try because > it's > hard. > >> For example, say you merge guest_memfd preservation today. Some time >> later, someone comes up with a more efficient data structure to track >> the folios in the file. You _have_ to make a backwards-incompatible ABI >> change to use this data structure. > > Only if those details bleed into the ABI/contract. I actually have a concrete > KVM (well, virtualization) example for this. > > Intel's VMX architecture disallows direct memory accesses to the VMCS, and > instead > requires software to access the VMCS via dedicated ISA, using architectural > encoding > numbers to reference VMCS fields. I.e. VMX decouples how data is stored in > memory > (the data structures) from the ABI/contract with software (VMCS field > encodings). > > This allows Intel to optimize the data structures to be more efficient and > performant > for each microarchitecture based on the features and properties of each > uarch, all > without breaking backwards/forwards compatibility with software. > > My favorite esoteric example is AR_BYTES packing. For Haswell, Intel added an > optimization in ucode to allow saving/loading segment register state in a > single > uop (IIRC). The optimization was especially valuable for virtualization as it > shaved cycles off the VM-Enter/VM-Exit hot paths. A key piece of the > optimization > was it required the AR_BYTES metadata to be stored in 16 bits, but existing > CPUs > stored AR_BYTES using 32 bits in an "unpacked" format. > > Fortunately, because the in-memory representation was decoupled from the > contract > with software, Intel could pack AR_BYTES into 16 bits for Haswell+ and > pack/unpack > the data on VMWRITE/VMREAD, so that the format presented to software remained > unchanged. > > Does VMX's decoupling of the in-memory represntation of a VMCS have downsides? > Absolutely. Most notably, it incurs extra complexity (in software and > hardware) > to achieve comparable performance to directly accessible data structures > (AMD's > VMCB and Hyper-V's eVMCS) for nested virtualization. And I'm sure it has > placed > contraints on Intel's designs, and obviously introduces complexity into the > overall system by adding a layer of indirection. But IMO the VMX architecture > has been a huge win overall for Intel. > >> Or say you add a new memory backend (like the HugeTLB patches in >> flight). That likely will need a different ABI to describe the state of >> the guest_memfd. >> >> So you will end up with multiple ABI versions that aren't always >> backwards compatible. > > No, you end up with *features* that aren't backwards compatible. I can't > imagine > anyone will argue that we should never add new features because then we can't > rollback to an older kernel. > > But adding a new feature shouldn't break the existing ABI. E.g. adding > support > for HugeTLB in guest_memfd shouldn't prevent rolling back to an older kernel > when > the HugeTLB functionality isn't being used. > > Using AMD's VMCB and Intel's VMCS as examples, literally every major new > AMD/Intel > uarch extends the VMC{B,S} in some way, but without fail it's always done in > a way > that is backwards compatible with existing software. I.e. AMD and Intel ship > new > features, but existing software continues to work, and VMs continue to be > migratable > across CPU generations[*], with the obvious restriction that migrating a VM > using > a feature introduced on generation N to a generation N-1 CPU isn't a smart > idea. > > [*] There are exceptions. E.g. Intel removed MPX, and so VMs with MPX can't > be > migrated to newer CPUs. Migrating between CPUs with different MAXPHYADDR > is > sketchy (and simply not done by some CSPs) because neither AMD nor Intel > virtualizes MAXPHYADDR. But those exceptions are absolutely Big Deals > that > undergo significant scrutiny, from all parties involved. > >> If you refuse that idea too, then KVM live update will be dead in the >> water for a different reason. It will be damn near useless because it >> can't keep up with an evolving subsystem. >> >> Now once you get multiple ABI versions and you can seamlessly go from >> old to new one, say you have a version that was superseded 5 years ago. >> It would be completely reasonable to say that this version is old enough >> and no one should be going from a 5 year old kernel to a modern one. > > LOL, Google literally does this. Granted, the extreme cases only happen for > stragglers, and I think we do force VMs to bounce through a "middle" kernel in > those cases, but I doubt Google is the only company that runs frankenkernels > for an absurd number of years for a variety of reasons. E.g. 4.4 LTS was > officially > supported for 6 years, and I'll bet dollars to donuts people ran it for much > longer > than that. > >> So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented. > > When there are provably no users, or we can convince existing userspace to > migrate > to an alternative. > >> I think we are better off formalizing this deprecation period from the >> get go. > > Why? What does it buy us? Because all I see is potential abuse and an excuse > for not spending time getting the designs right. > >> A somewhat tangential example is BPF kfuncs. My BPF program that works >> in kernel X might not work in kernel Y because the kfunc has changed or >> been removed. >> >> The argument they make in kfuncs.rst is that kfuncs "provide a kernel >> <-> kernel API, and thus are not bound by any of the strict stability >> restrictions associated with kernel <-> user UAPIs". > > BPF's documentation isn't arguing anything, it's merely reiterating Linux's > long-standing policy that there is no such thing as a stable kernel ABI (in > upstream). > >> For LUO as well, this is a kernel -> kernel API. > > Stating the obvious, I disagree with this. As I said before, if this is the > stance LUO wants to take, then so be it, but my NAK stands. > >> Users can also still do a regular kexec or reboot. They just won't get the >> performance optimization of LUO. > > The amount of time, energy, and money poured into minimizing VM downtime on > live > migration suggests the overwhelming majority of LUO's targeted users aren't > going > to take kindly to this stance. > >> Regardless of if you agree with the last bit about deprecating old >> versions, ABIs evolving with the subsystem is a ground reality of live >> update and it would be foolish to think we can do with only >> backwards-compatible ABI changes forever. > > I never said the ABI is immutable, I said it needs to be backwards/forwards > compatible.
Thanks Sean, Pratyush. @loganodell has recently sent the RFC[1] on backward compatibility for liveupdate. This discussion is important so I propose to move the backward compatibility discussion on [1]. For guest_memfd preservation, I will sent v5 soon with the following changes: 1. Suggestions from across the patches 2. tentative Plan for future Guest_memfd preservation 3. tentative Plan for VM preservation. 4. Backward compatibility built on top of RFC [1] What do you think? [1]: https://lore.kernel.org/all/[email protected]/
