I suspect the issue is caused by this commit:
commit c22d70a162d3cc177282c4487be4d54876ca55c8
Author: Roman Gushchin <roman.gushc...@linux.dev>
Date: Mon Jun 28 19:36:03 2021 -0700
writeback, cgroup: release dying cgwbs by switching attached inodes
It introduces a kworker "cleanup_offline_cgwbs_work", which leverages
inode_switch_wbs_work_fn to keep moving inodes from dying cgwb to their parent,
so it can reclaim more memory from wb structures.
Moving inodes from a wb needs to take its lock (wb->list_lock) which
also needs to be taken in inode_switch_wbs, that could be where the
contention happens.
I built a test kernel based on gcp kernel (6.8.0-1019) disabling this kworker
for "testing only":
- queue_work(system_unbound_wq, &cleanup_offline_cgwbs_work);
+ //queue_work(system_unbound_wq, &cleanup_offline_cgwbs_work);
https://launchpad.net/~gerald-yang-tw/+archive/ubuntu/400986-debug
If you would like to help testing, please add the above test PPA and install:
linux-gcp-6.8-headers-6.8.0-1019
linux-gcp-6.8-tools-6.8.0-1019
linux-headers-6.8.0-1019-gcp
linux-image-unsigned-6.8.0-1019-gcp
linux-modules-6.8.0-1019-gcp
linux-modules-extra-6.8.0-1019-gcp
Please note:
This test kernel is only used to confirm if this kworker is the cause of the
issue, not a fix.
Thanks,
Gerald
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-oem-6.5 in Ubuntu.
https://bugs.launchpad.net/bugs/2038492
Title:
workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 16 times,
consider switching to WQ_UNBOUND
Status in linux-oem-6.5 package in Ubuntu:
Confirmed
Bug description:
dmesg has multiple messages:
[ 852.580542] evict_inodes inode 00000000be24a21a, i_count = 1, was skipped!
[ 852.580543] evict_inodes inode 000000002aea522e, i_count = 1, was skipped!
[ 852.580544] evict_inodes inode 0000000048d75a5c, i_count = 1, was skipped!
[ 852.580545] evict_inodes inode 00000000d8a9e4c2, i_count = 1, was skipped!
[ 852.580546] evict_inodes inode 000000003d2c905c, i_count = 1, was skipped!
[ 852.580547] evict_inodes inode 00000000e5b1b232, i_count = 1, was skipped!
[ 852.580548] evict_inodes inode 0000000097383a6b, i_count = 1, was skipped!
[ 852.580549] evict_inodes inode 00000000ca8e2b44, i_count = 1, was skipped!
[ 1751.869281] workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 4
times, consider switching to WQ_UNBOUND
[ 1781.467278] workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 8
times, consider switching to WQ_UNBOUND
[ 1806.065364] workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 16
times, consider switching to WQ_UNBOUND
[ 1901.993975] evict_inodes inode 00000000abaa740e, i_count = 1, was skipped!
[ 1901.993981] evict_inodes inode 00000000515b9bf8, i_count = 1, was skipped!
[ 1901.993983] evict_inodes inode 000000001a69d536, i_count = 1, was skipped!
[ 1901.993984] evict_inodes inode 000000001403f675, i_count = 1, was skipped!
[ 1901.993985] evict_inodes inode 00000000757de21a, i_count = 1, was skipped!
[ 1901.993986] evict_inodes inode 000000000cee9028, i_count = 1, was skipped!
[ 1901.993987] evict_inodes inode 00000000d2827e77, i_count = 1, was skipped!
Description: Ubuntu 22.04.3 LTS
Release: 22.04
linux-oem-22.04d:
Installed: 6.5.0.1004.4
Candidate: 6.5.0.1004.4
Seems related to #2037214
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-oem-6.5/+bug/2038492/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
