(+ the other Arm maintainers)
On 31/10/2025 13:01, [email protected] wrote:
Hello,
Hi,
Before answering to the rest, would you be able to configure your e-mail
client to quote with '>' and avoid top-posting? Otherwise, it will
become quite difficult to follow the conversation after a few round.
I have seen no such performance issue with nested KVM. For Xen, if this
can be relaxed from |vmalls12e1| to |vmalle1|, this would still be a
huge performance improvement. I used Ftrace to get execution time of
each of these handler functions:
handle_vmalls12e1is() min-max = 1464441 - 9495486 us
To clarify, Xen is using the local TLB version. So it should be
vmalls12e1. But it looks like KVM will treat it the same way and I
wonder whether this could be optimized? (I don't know much about the KVM
implementation though).
So, to summarize using HCR_EL2.FB (which Xen already enables?) and then
using vmalle1 instead of vmalls12e1 should resolve the issue-2 for vCPUs
switching on pCPUs.
I don't think HCR_EL2.FB would matter here.
Coming back to issue-1, what do you think about creating a batch version
of hypercall XENMEM_remove_from_physmap (other batch versions exist such
as for XENMEM_add_to_physmap) and doing the TLB invalidation only once
per this hypercall?
Before going into batching, do you have any data showing how often
XENMEM_remove_from_physmap is called in your setup? Similar, I would be
interested to know the number of TLBs flush within one hypercalls and
whether the regions unmapped were contiguous.
In your previous e-mail you wrote:
> During the creation of domu, first the domu memory is mapped onto
dom0 domain, images are copied into it, and it is then unmapped. During
unmapping, the TLB translations are invalidated one by one for each page
being unmapped in XENMEM_remove_from_physmap hypercall. Here is the code
snippet where the decision to flush TLBs is being made during removal of
mapping.
Don't we map only the memory that is needed to copy the binaries? If
not, then I would suggest to look at that first.
I am asking because even with batching, we may still send a few TLBs
because:
* We need to avoid long-running operations, so the hypercall may
restart. So we will have to flush at mininum before every restart
* The current way we handle batching is we will process one item at
the time. As this may free memory (either leaf or intermediate
page-tables), we will need to flush the TLBs first to prevent the domain
accessing the wrong memory. This could be solved by keeping track of the
list of memory to free. But this is going to require some work and I am
not entirely sure this is worth it at the moment.
I just realized that ripas2e1 is a range TLBI
instruction which is only supported after Armv8.4 indicated
by ID_AA64ISAR0_EL1.TLB == 2. So, on older architectures, full stage-2
invalidation would be required. For an architecture independent
solution, creating a batch version seems to be a better way.
I don't think we necessarily need a full stage-2 invalidation for
processor not supporting range TLBI. We could use a series of TLBI
IPAS2E1IS which I think is what TBLI range is meant to replace (so long
the addresses are contiguous in the given space).
On the KVM side, it would be worth looking at whether the implementation
can be optimized. Is this really walking block by block? Can it skip
over large hole (e.g. if we know a level 1 entry doesn't exist, then we
can increment by 1GB).
Cheers,
--
Julien Grall
