David Woodhouse <[email protected]> writes: > On Wed, 2026-07-22 at 17:24 -0700, Ackerley Tng wrote: >> David Woodhouse <[email protected]> writes: >> >> > For dedicated hosting environments, there are a few reasons to prefer >> > using external PFNMAP memory for guests, rather than kernel-managed >> > memory: >> > >> > • No struct page overheads >> > >> > • Easily managed in 1GiB pages (or larger chunks); no fragmentation/THP >> > >> > • Faster kexec/KHO live update (every millisecond spent faffing around >> > with memory management is an extra millisecond of steal time taken >> > from guests) >> > >> >> Also, no need to split/merge pages when converting from private to >> shared if the host mappings can't depend on the refcounts in struct >> pages :). >> >> > Today, there is only one implementation of guest_memfd: the internal >> > shmem-based page cache in virt/kvm/guest_memfd.c. This series allows >> > for other code to provide "a guest_memfd". This is an early path- >> > finding proof of concept tested with AMD SEV-SNP as well as non-CoCo >> > guests on x86 and arm64. I'm planning to build a memory-based file >> > system which gives files for both guest_memfd as well as memory-based >> > storage to allow things like userspace VM and orchestrator state >> > to be passed over KHO/kexec. >> > >> >> Thanks for raising this, good to know that there is more interest! >> >> Vishal and I have been working on guest_memfd with HugeTLB for a while >> now. I'll be restarting upstream work on HugeTLB after the conversions >> series merges. >> >> Vishal, Frank and I brought up having guest_memfd with HugeTLB with >> VM_PFNMAP mappings at a guest_memfd biweekly meeting a while ago. The >> consensus was that guest_memfd HugeTLB must support GUP and so VM_PFNMAP >> can't be the only way that guest_memfd supports HugeTLB. >> >> Your proposal is true non-page-struct memory, which has been on the >> discussion guest_memfd biweekly discussion backlog for a while >> now. Google is definitely also interested in this. >> >> Frank also described some use cases [1] other than HugeTLB and the >> regular PAGE_SIZE pages from the buddy that is currently supported. >> >> I submitted a proposal for LPC 2026 to discuss the interface. What I had >> in mind was to have the KVM_GUEST_MEMFD_CREATE ioctl take a "template" >> fd. guest_memfd will then read off what it needs from the "template" fd >> (ops, like you have here), and the "template" fd could be VFIO [2] or >> HugeTLB. >> >> You have a provider_fd = open("/dev/gmem_provider"), how about passing >> that provider_fd to guest_memfd, where /dev/gmem_provider could >> represent the page-less memory, like: >> >> ioctl(kvm_fd, KVM_GUEST_MEMFD_CREATE, provider_fd); >> >> It would be cool if the fallback could be set up flexibly, like: >> >> ioctl(kvm_fd, KVM_GUEST_MEMFD_CREATE, >> [provider_fd, fallback_provider_fd, ...]); >> >> I think guest_memfd should be the wrapper around other memory providers >> because then the CoCo shared/private logic and special page table >> handling for both host and stage 2 page tables is centralized in >> guest_memfd, and also, KVM doesn't have to understand other, different >> kinds of fds other than guest_memfd. > > Yeah. My approach was to make those "other kinds of fds" essentially > the same — they are "a guest_memfd", and KVM didn't have to understand > anything else (after a little bit of rework to have the generic ops so > that other code can create its own guest_memfds). As opposed to the > approach in Xu Yilun's TDISP series, which did teach KVM about a new > kind of memslot. > > Likewise, I used dma-buf so that IOMMUFD didn't have to understand > anything new either. > > I like your approach, inverting the wrapping of guest_memfd/dma-buf. > The provider then gives a plain dma-buf (not associated with any KVM as > it is in my current code). Userspace can use that *directly* with the > IOMMU,
A wrinkle we will have to iron out is kind of the opposite problem, to make sure that while guest_memfd thinks a range of offsets (PAGE_SIZE or larger) is CoCo-private, the PFNs in that range must not be mapped anywhere else in the host, or IOMMU. One of the options I was thinking of was to seal off the template fd, perhaps with an inode flag like S_IMMUTABLE, but not even allow reads, and unlike S_IMMUTABLE, the (new) flag must not be changeable by userspace. Another option would be to just close the template fd so it'll not be used, and the PFNs behind that template fd would just be completely in guest_memfd ownership. All non-guest_memfd paths to access the PFNs must be closed off, or perhaps more precisely, any path that is not checked for CoCo-shared/private status before building page table entries must be closed off. For CoCo-shared memory as tracked by guest_memfd (or non-CoCo, which would use SHARED), the PFNs can still be mmap()-ed and actually set up in userspace page tables. Then, to use CoCo-shared memory for IO, IOMMUFD would also have to take the guest_memfd (which IOMMUFD probably has to do, for CoCo-private memory for Confidential IO, since CoCo-private memory can't be mapped to userspace). Perhaps through IOMMU_IOAS_MAP_FILE? In the process of mapping, IOMMUFD would tell guest_memfd "I have some of your PFNs mapped.", and when guest_memfd gets a private->shared conversion request, guest_memfd would get IOMMUFD to drop the mappings. (perhaps some kind of MMU notifier setup) Do you plan to use this for non-CoCo? > and passes it to your KVM_GUEST_MEMFD_CREATE in order to get a > guest_memfd for use with a given KVM. I think a lot of the warts in my > series fall away if we do it that way round. > > I'll have a go at reworking it that way round... and see what different > warts it ends up with. The interesting part will be getting actual PFNs > out of the dma-buf (which is already under active discussion). > If we go with the above, IOMMUFD would get PFNs from guest_memfd instead of dma-buf. > I'm not sure I understand the 'fallback provider' part. If one provider > doesn't have a page for a given address, KVM would iterate over them > all until it finds one that does? If each provider provides a .get_pfn() op, guest_memfd's kvm_gmem_get_pfn() would try the first provider's .get_pfn() and if that errors out, try the next one. guest_memfd today uses a standard struct address_space to track the pages that it owns. guest_memfd will have to use something else that can track both pages and PFNs, and probably which provider it came from. The more you probe me, the more complicated it's sounding. :) Still worth pursuing if it's good to have though. > Is that about falling back from one > THP pool to another? Is this something that the generic KVM code needs > to know about, or could it be handled *within* a provider, which can > chain to the next pool if it can't find a page? Handling that within a provider could be nice too, so instead of guest_memfd trying .get_pfn()s in series, the provider will try a hard-coded pair/list of PFN sources and then deal with the complexity on behalf guest_memfd? The flipside is for every combination of providers, the owner of the virtualization service would have to maintain a custom kernel module for the managing pages? Maybe the flipside isn't too bad, actually, to provide more flexibility. > Wouldn't we want that > done transparently for both KVM and IOMMU consumers? > >> >> [1] >> https://lore.kernel.org/all/CAPTztWajm_JLpp9BjRcX=h72r25ELrXeGkOXVachybBxLJGS=g...@mail.gmail.com/ >> [2] >> https://lore.kernel.org/all/CAEvNRgFpJWQ5M5sQhGpQUV3GbBq9N+MQhhaxdxa=d8ky94s...@mail.gmail.com/ >> >> > This series lifts guest_memfd's operations into a small ops struct and >> > converts KVM's own gmem to use it, so it becomes one implementation >> > among peers rather than the only game in town. It also adapts the >> > SEV-SNP fast paths to page-less operation, plumbs iommufd to map a >> > guest_memfd-backing dma-buf as CPU RAM (not MMIO), and adds the >> > provider->KVM revocation primitive that makes overcommit-style reclaim >> > possible. >> > >> > It's largely AI-built at the moment. My main design concerns are around >> > how we tell that a given file is "a guest_memfd", and how we plug into >> > IOMMUFD. Presenting a dma-buf was the less intrusive choice because that >> > path already has support for taking pages away, but I'm less convinced >> > that it's the cleaner choice in the long term. >> > >> >> IIUC from this series, you're creating a dmabuf fd out of a guest_memfd, >> what does "taking pages away" mean, in this case, from where/who? > > Taken away from the guest. Ballooning pages out, either for traditional > memory overcommit, or for the models where a guest donates some of its > pages to the system to run a "child" which is properly isolated from > the guest. > > The physical page is reclaimed and needs to be taken out of the KVM and > IOMMU stage2 mappings. Will the VMM be doing a PUNCH_HOLE on the dma-buf fd or guest_memfd to actually reclaim the page, or do you have something else in mind? > If KVM wants to touch it again it must call the > provider to get a new PFN for that range. This is roughly equivalent to > the MMU notifier path on userspace VMAs in the normal memslot case. > IOMMUFD has no equivalent of that for userspace addresses — > IOMMU_IOAS_MAP pins the pages so they *can't* go away, which is why I > used dma-buf for the IOMMUFD interface. > > I've actually been fleshing out the overcommit path a bit more, > experimenting with adding async-pf support to guest_memfd in > https://git.infradead.org/?p=users/dwmw2/linux.git;a=commitdiff;h=894dd0fcb34 > (the sample provider just pretends its pages are slow to populate, so > the guest gets to run something else while a "reclaimed" page comes > back).
