** Changed in: linux (Debian)
Status: Unknown => Confirmed
** Changed in: linux
Status: Unknown => Confirmed
** Changed in: linux
Importance: Unknown => Medium
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1922846
Title:
KVM guest time accounting is broken
Status in Linux:
Confirmed
Status in linux package in Ubuntu:
New
Status in linux-hwe package in Ubuntu:
New
Status in linux package in Debian:
Confirmed
Bug description:
The kernel keeps track of the time spent in userspace, kernel space
and for quite a while (since 2.6.24) also guest_time.
This is accessible to userspace
a) globally via /proc/stat [1]
b) per process via /proc/<PID>/stat
But since ~5.9 this accounting is broken by [3]
#An example of a busy guest reporting all zeros in a common userspace tool
ubuntu@node-horsea:~$ pidstat -p $(pidof qemu-system-x86_64) 5
Linux 5.11.0-13-generic (node-horsea) 04/07/21 _x86_64_
(12 CPU)
05:13:13 UID PID %usr %system %guest %wait %CPU CPU
Command
05:13:18 64055 8186 10.20 96.80 0.00 0.00 107.00 4
qemu-system-x86
05:13:23 64055 8186 5.00 51.00 0.00 0.00 56.00 5
qemu-system-x86
05:13:28 64055 8186 3.60 109.80 0.00 0.00 113.40 5
qemu-system-x86
05:13:33 64055 8186 1.80 14.20 0.00 0.00 16.00 5
qemu-system-x86
This is due to these stats staying zero:
$ cat /proc/stat | grep "cpu "
cpu 11345 154 10342 71506151 549 0 161 0 0 0
The Testcase for this is simple, run any KVM guest content for example even
the following without any prep is enough
$ sudo qemu-system-x86_64 -enable-kvm -nographic
That should execute at least a bit of time in the guest for startup code.
Surely, if you want anything for long term debugging you can run a full Guest
and do any busy loop in the guest.
This was reported upstream to the kernel in [4] and Debian [5] and
qemu [6].
AFAIK there are two suggested [8][9], but no accepted solution yet.
If it would be "only" the global counter (a) this might be prio low-
to-mid, because while still inhibiting monitoring you'd at least still
have the per guest counters (b). But since both are broken one has no
easy means anymore to check if there is massive host-kernel (%sys) or
host-userspace (%usr) overhead over the guest (%guest) anymore.
Therefore (maybe due to my old perf-engineer heart) IMHO this is
rather important to be fixed soon.
Best in the next possible update for Hirsute, but that might be hard.
But at lest for an Hirsute based HWE kernel to Focal I'd consider this
a hard blocker.
[1]: https://www.kernel.org/doc/Documentation/filesystems/proc.txt see "1.8
Miscellaneous kernel statistics in /proc/stat"
[2]: http://manpages.ubuntu.com/manpages/focal/man5/proc.5.html see "(43)
guest_time %lu"
[3]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87fa7f3e98a1310ef1ac1900e7ee7f9610a038bc
[4]: https://bugzilla.kernel.org/show_bug.cgi?id=209831
[5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371
[6]: https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg00732.html
[7]:
https://lore.kernel.org/kvm/1617011036-11734-1-git-send-email-wanpen...@tencent.com/
[8]: https://lore.kernel.org/kvm/20210206004218.312023-1-sea...@google.com/
ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: linux-image-5.11.0-13-generic 5.11.0-13.14
ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7
Uname: Linux 5.11.0-13-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq
crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu61
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: unknown
Date: Wed Apr 7 05:02:03 2021
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
PciMultimedia:
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-13-generic
root=UUID=c941b173-e6b5-485a-a02b-8d966b8d3c73 ro --- console=ttyS1,115200
RelatedPackageVersions:
linux-restricted-modules-5.11.0-13-generic N/A
linux-backports-modules-5.11.0-13-generic N/A
linux-firmware 1.196
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/22/2018
dmi.bios.release: 2.56
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.board.name: ProLiant DL360 Gen9
dmi.board.vendor: HP
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.ec.firmware.release: 2.60
dmi.modalias:
dmi:bvnHP:bvrP89:bd01/22/2018:br2.56:efr2.60:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.product.sku: 780018-S01
dmi.sys.vendor: HP
To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1922846/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
