Pau has performed some performance tests by booting 1000 mirage unikernels test VMs and shutting them down. We've noticed on the flamegraphs that a lot of time is spent in Xenctrl `domain_getinfolist`, 17.7% of overall time (This needs to be multiplied by 16 because Dom0 100% usage = 16 vCPUs) In particular time is spent in camlXenctrl___getlist_339 as can be seen from this flamegraph, and it also creates a very deep call stack: https://cdn.jsdelivr.net/gh/edwintorok/xen@xenctrl-coverletter/docs/tmp/perf-merge-boot.svg?x=948.9&y=2213
After some algorithmic improvements to the code now the function barely shows up at all on a flamegraph, taking only 0.02%. The function is called camlXenctrl___getlist_343, but that is just due to the changed arguments, still the same function: https://cdn.jsdelivr.net/gh/edwintorok/xen@xenctrl-coverletter/docs/tmp/perf-xen-boot-1150.svg?x=1188.0&y=1941&s=infolist It was calling the Xen hypercall ~500*1000 times for 1000 VMs, and instead it is now calling it "only" 1000 times. I would suggest to try to take this in 4.17 given the massive improvement in scalability (number of VMs on a Xen host). There are further improvements possible here, but they'll be in xenopsd (part of XAPI) to avoid calling domain_getinfolist and just use domain_getinfo: the only reason it needs use infolist is that it does the lookup by VM UUID and not by domid, but it could have a small cache of UUID->domid mappings and then call just domain_getinfo (or get the mapping from xenstore if not in the cache), but it looks like that improvement is not even needed if this function barely registers on a flamegraph now. P.S.: the mirage test VM is a very old PV version, at some point we'll repeat the test with a Solo5 based PVH one. Edwin Török (2): xenctrl.ml: make domain_getinfolist tail recursive xenctrl: use larger chunksize in domain_getinfolist tools/ocaml/libs/xc/xenctrl.ml | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) -- 2.34.1
