Peter Xu <[email protected]> writes: > On Mon, Jan 22, 2024 at 12:39:06PM -0300, Fabiano Rosas wrote: >> [email protected] writes: >> >> > From: Peter Xu <[email protected]> >> > >> > The current article is not extremely easy to follow, and may contain too >> > much information for someone looking for solutions on VMSD compatibility >> > issues. Meanwhile, VMSD versioning is not discussed. >> > >> > I'm not yet sure whether we should just obsolete VMSD versioning; it's >> > still used quite a lot. And I had a feeling that for simple use cases >> > where backward migration is not strongly required, device developers can >> > still consider using it. So in this patch I decided to keep it (anyway, we >> > can't drop it in the near future because of massive existing users), and we >> > can still allow user to use it in contexts where forward-only migration >> > might be enough. >> > >> > This doc patch does below changes: >> > >> > - Rename the page from "Backward compatibility" to "Migration >> > compatibility", to avoid using "backward" as a word (because we'll want >> > to identify "forward" / "backward" migrations in the new doc) >> > >> > - Add a TOC for this page for better indexing >> > >> > - A new section to explain what is forward/backward migration >> > >> > - A new small section for VMSD just to make things complete >> > >> > - Explain the two ways to make VMSD compatible with old qemu binaries >> > >> > For this one, I added a small section on how to use VMSD versioning for >> > new fields just to start such documents. Rename the old "How backwards >> > compatibility works" section to "machine type based (forward+backward >> > migration)" to be the 2nd solution (I called it machine type based >> > solution). When at it, I provided a summary and a TODO for the 2nd >> > solution. >> > >> > - A new section to explain which solution to choose >> > >> > - Moved the other two existing sections into "Extended readings", because >> > they can be even further away to most device developers >> > >> > Signed-off-by: Peter Xu <[email protected]> >> > --- >> > docs/devel/migration/compatibility.rst | 140 ++++++++++++++++++++++++- >> > 1 file changed, 137 insertions(+), 3 deletions(-) >> > >> > diff --git a/docs/devel/migration/compatibility.rst >> > b/docs/devel/migration/compatibility.rst >> > index 5a5417ef06..ea9da201ef 100644 >> > --- a/docs/devel/migration/compatibility.rst >> > +++ b/docs/devel/migration/compatibility.rst >> > @@ -1,8 +1,139 @@ >> > -Backwards compatibility >> > ======================= >> > +Migration compatibility >> > +======================= >> > + >> > +Migration is a hard topic sometimes. One of the major reason is that it >> > +has a strict compatibility requirement - a migration (live or not) can >> > +happen between two different versions of QEMUs, so QEMU needs to make sure >> >> S/QEMUs/QEMU/ > > I'll fix all these up that you suggested as English issues. > >> >> > +the migration can work across different versions of QEMU binaries. >> > + >> > +This document majorly discusses the compatibility requirement of forward / >> >> s/majorly/mainly/ >> >> > +backward migrations that QEMU need to maintain, and what QEMU developers >> >> s/need/needs/ >> >> > +should do to achieve such compatibility requirements across different QEMU >> >> maybe s/achieve/maintain/ ? >> >> > +versions. >> > + >> > +.. contents:: >> > + >> > +Types of migrations (forward / backward) >> > +======================================== >> > + >> > +Let's firstly define the terms **forward migration** and **backward >> > +migration**. >> > + >> > +.. note:: >> > + >> > + To simplify the use case, we always discuss between two consecutive >> > + versions of QEMU major releases (between QEMU version *N* and QEMU >> > + version *N-1*). But logically it applies to the case where the two >> > + QEMU binaries involved contains more than one major version >> > difference. >> > + >> > +.. _forward_migration: >> > + >> > +**Forward migration**: can be seen as the use case where a VM cluster can >> > +upgrade its nodes to a newer version of QEMU (version *N*) from an older >> > +version of QEMU (version *N-1*). >> > + >> > +.. _backward_migration: >> > + >> > +**Backward migration**: can be seen as the use case where a VM cluster >> > +would like to migrate from a newer version of QEMU (version *N*) back to >> > an >> > +even older version of QEMU (version *N-1*). >> >> I'd drop the VM cluster part from these. Define the terms in a more >> strict manner (QEMU versions, n/n-1, that's it). Then the parts below >> could be second paragraphs further detailing the use-cases of the two >> types of migration. > > OK. > >> >> > + >> > +A forward migration is more common, where system upgrades are needed. In >> > +this case, the upgrade can be done seamlessly by live migrating the old >> > VMs >> > +to the new VMs with the new binaries. >> >> I got a bit confused whether this was describing migration to a >> different host or within the same host. I suggest we spell it out some >> more: > > It can be same-host or across-host. > >> >> "live migrating an existing VM that uses an old QEMU binary to another >> VM using the newly updated QEMU binary." > > Even though I don't see a major difference on how it was reworded.. but I > can use your version here. > >> >> > + >> > +A backward migration can be less common OTOH, because downgrade is less >> > +common than upgrade for whatever reasons. However for a production level >> > +system setup, this should also be allowed, because a cluster can contain >> > +different versions of QEMU binary. It should be always allowed to migrate >> > +between old and new hosts as long as the machine type is supported across >> > +all the relevant hosts / nodes. >> > + >> > +VMState description data structure (VMSD) >> > +========================================= >> > + >> > +VMSD (or in the complete form, **VMStateDescription**) is the data >> > +structure that QEMU uses to describe data to be migrated for devices. >> > +Each device should provide its own VMSD structure to describe what it >> > needs >> > +to be migrated when a VM live migration is requested. >> > + >> > +Device VMSD compatibility >> > +========================= >> > + >> > +Then if the VMSD structures need changing, how does the device maintain >> > +compatibilty? >> >> s/compatibilty/compatibility/ >> >> Hm, there's a logical jump here from the _device_ changing to this >> requiring a change in the VMSD. Also the device is not the thing that >> maintains compatibility. Again we might need to spell it out: >> >> "When a device data needs to change from one QEMU version to another, >> how to maintain compatibility?" ... or something like this > > OK. > >> >> > >> > -How backwards compatibility works >> > ---------------------------------- >> > +Here we only discuss VMSD-based migrations. If one device is not using >> > +VMSD to migrate its device data, it's considered part of "advanced users", >> > +then this document may not apply anymore. If you're writting a new >> > device, >> > +please always consider starting with VMSD-based migration model. >> > + >> > +Consider the case where a device can start to support a new feature in the >> > +current release, where it wasn't supported before. The new feature may >> >> s/, where it/ that/ >> >> > +require some new device states to be migrated (which can be new VMSD >> > fields >> > +to be added, or new subsections). The same question needs to be answered >> > +when one would like to modify an existing VMSD fields / layouts to fix a >> >> s/an // >> >> > +bug, and so on. >> > + >> > +Depending on the goal, the solution to this problem may vary. >> > + >> > +If one would like to provide a full support of migration between whatever >> > +versions, one can try to implement it using :ref:`machine_type_compat` >> > +solution. If one would like to provide a fundamental upgrade-only >> > +compatibility, one could consider to use the simpler >> > +:ref:`vmsd_versioning_compat` solution. >> >> Should this whole paragraph be put before mentioning VMSD? Otherwise >> here we are talking about machine type compat after having said: "Here >> we only discuss VMSD-based migrations". > > Note that even with the machine compat properties solution we mentioned > below, the device will still need some VMSD entry changes, and VMSD is > still the core of the problem. > > The difference is instead of using VMSD versioning, one may need to link > the VMSD entry to the machine compat properties via either: > > - VMStateDescription.needed(), as a sub-vmsd or, > > - VMStateField.field_exists() as a vmsd field. > > That's why I want to keep VMSD a separate section, because it's always the > core of the problem, no matter for either of the two solutions below. > >> >> All in all this could be: >> >> up top: >> Migration compatibility strategies >> ================================== >> >> When a device data needs to change from one QEMU version to another, how >> to maintain compatibility? >> >> Depending on the goal, the solution to this problem may vary. >> >> If one would like to provide a full support of migration between >> arbitrary versions, one can try to implement it using >> :ref:`machine_type_compat` solution. If one would like to provide a >> fundamental upgrade-only compatibility, one could consider to use the >> simpler :ref:`vmsd_versioning_compat` solution. >> >> VMState description data structure (VMSD) >> ----------------------------------------- >> >> VMSD (or in the complete form, **VMStateDescription**) is the data ... >> ... >> >> Machine type properties >> ----------------------- >> >> The QEMU machine type is versioned (-machine q35 resolves is equivalent >> to pc-q35-9.0) and has properties that can be set to different values >> across machine versions. Those can be used to enable/disable different >> parts of the code and can be used to help migration compatibility. > > Some statement on machine types may be good, but I'm afraid then we'll also > need to describe more on machine compat properties, which is IMHO even more > important. That part is actually more or less covered by the old document > below (which is pretty long that Juan drafted just a while ago). That's > why I didn't mention much on machine types here, leaving that for the rest. > > PS: I added a "TODO" though right below for a possible future rewrite [1]. > The current explanation is, IMHO, too much, and not as clear. > > The major goal of this doc patch is adding vmsd versioning into the > picutre, and describe both solutions we can use, while in Juan's old > version there's no mention of VMSD versioning. The ultimate goal is in the > future we can share the doc to other developers when asking similar > questions. > > Since we're at it, I would also like to know how you think about whether we > should still suggest people using VMSD versioning, as we know that it won't > work for backward migrations. > > My current thoughts is it is still fine, as it's easier to use, and it > should still be applicable to the cases where a strict migration semantics > are not required. However it's hard to justify which device needs that > strictness.
I'd prefer if we kept things strict. However I don't think we can do that without having enough testing and specially, clear recipes on how to add compatibility back once it gets lost. Think of that recent thread were we discussed an old powerpc issue. How come we can see the fix today in the code but cannot tell which problem it was trying to solve? That's bonkers. Ideally every type of breakage would have a mapping into why it breaks and how to fix it. So with testing to catch the issue early and a clear step-by-step on how to identify and fix compatibility, then we could require strict compatibility for every device. > > For example, any device to be used in migration-test must be forward + > backward migration compatible at least, because you just added the n-1 > regression tests to cover both directions. Said that, only a few devices > are involved because currently our migration-test qemu cmdline is pretty > simple. We might want to make a distinction between migration core vs. device state testing. I see n-1 testing more like migration core testing. It's bad to break migration, but it's really bad to break migration for everyone because we refactored something deep within migration/. I also wouldn't mind if we had some simple way for device developers to add migration tests that cover their code. Currently it's infeasible to edit migration-test with new command lines for every device of interest. Maybe we could have a little framework that takes a command line and spits a migration stream? Something really self-contained, behind the device's CONFIG in meson. > > Fundamentally, IMHO it's because QEMU as a project is used both in > enterprise and personal emulations. I think it might be too strict to > always request backward migration capability if we know some device / arch > is only used for personal, or educational, purposes. Do we need migration support tiers? =) If a machine supports KVM, that's already a good indication that we will want to have migration working both ways. If the machine supports only TCG, then migration will probably be a nice to have for development, but no one would be relying on it in production. It might be ok to only support forward migration. Could that be a first filter to reason about this? That might narrow the list of devices enough. > The other reason is so far using machine compat properties can sometimes be > complicated, and I'm not sure whether it's worthwhile we keep requesting > that to device developers. Maybe some day we can provide some VMSD macros > to make that even easier, then we can obsolete VMSD versionings. But I > haven't really thought it through. > >> >> Migration compatibility solutions >> ================================= >> >> VMSD versioning (forward migration only) >> ---------------------------------------- >> ... >> >> Machine type based (forward+backward migration) >> ----------------------------------------------- >> ... >> >> > + >> > +Solutions >> > +========= >> > + >> > +.. _vmsd_versioning_compat: >> > + >> > +VMSD versioning (forward migration only) >> > +---------------------------------------- >> > + >> > +This is normally the simplest way to support cross-version QEMU live >> > +migration. The trade-off is backward migration will not be supported. It >> > +means migrations from new QEMU binaries to old QEMU binaries can fail. >> > It's >> > +because even if the new QEMU can understand the old version of VMSD by >> > +proper versioning of the VMSD fields, the old QEMU will not be able to >> > +understand the new version of VMSD layout. Then when someone migrates a >> > VM >> > +using the new VMSD to an older version of QEMU, the old QEMU will not >> > +accept the new migration stream, reporting that the VSMD version too new. >> >> But we still have _some_ form of backward compat for _some_ kinds of >> problems by using dummy fields in the VMSD for instance, right? > > Just to make sure we're on the same page: UNUSED is definitely needed for > forward migration, since if without those the old stream will still contain > the removed field, which on the new binary will be wrongly recognized as > the "next" field if ever existed, or "extra / unknown" fields, probably > messing up the next device state. > > For backward, I assume it only works if all-zero is a valid state first on > that field being removed on the old binary. Otherwise it might still fail > in weird ways, afaiu. For that, I'd say that shouldn't be stated as > supported either as a general approach. > >> >> > + >> > +Please have a look at **include/migration/vmstate.h** for more information >> > +on how to use VMSD versioning. >> > + >> > +Taking an example of adding a new field for migration. The change will >> > +need to at least contain two parts: >> > + >> > + - Boost existing VMSD version. >> > + >> > + - Add the new VMSD field with the boosted version, with specific >> > + **VMSTATE_\*_V()** macros. For example, **VMSTATE_UINT8_V()** will >> > + define an uint8 typed VMSD field with version specified. >> > + >> > +.. _machine_type_compat: >> > + >> > +Machine type based (forward+backward migration) >> > +----------------------------------------------- >> > + >> > +QEMU developers can leverage machine type compatibile properties to >> > provide >> >> compatible >> >> > +a fully migratable device / protocol, so the migration behavior will be >> > +defined by the machine type, no matter which QEMU binary will be used. >> > One >> > +can reference the entries defined in **hw_compat_\*** global properties >> > for >> > +examples. >> > + >> > +Comparing to VMSD versioning approach above, this may require more code >> > +changes, but provide a higher level of compatibility that is bound to the >> > +machine type being used. To be explicit, since the migration behavior is >> > +bound to machine type, it will support both forward migration and backward >> > +migration as long as the machine type is supported. >> > + >> > +.. note:: >> > + >> > + Currently this section is pretty long. TODO: rewrite this section to >> > + make it easier for QEMU developers to understand. > > [1] > >> > >> > When we do migration, we have two QEMU processes: the source and the >> > target. There are two cases, they are the same version or they are >> > @@ -217,6 +348,9 @@ machine types to have the right value:: >> > ... >> > }; >> > >> > +Extended readings >> > +================= >> > + >> > A device with different features on both sides >> > ---------------------------------------------- >>
