[Expired for linux (Ubuntu) because there has been no activity for 60
days.]
** Changed in: linux (Ubuntu)
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1841841
Title:
disk stress test extremely slow progress after cpu offline/online
Status in linux package in Ubuntu:
Expired
Bug description:
One of the test systems we use for testing snap updates is:
Dell Inspiron 5759
Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
Advanced Micro Devices, Inc. [AMD/ATI] - 1002:6660, Intel - 8086:1916
We picked this one, some time ago from a pool of hardware to use for
one of the test systems to test amd64/i386 snaps in beta. One of the
tests in the test suite does some disk stress testing using:
bonnie++ -d /var/lib/snapd/hostfs/writable -u root -r 8000
Looking back through our results, I see that it has often taken from
20-40 min. to complete the test, and sometimes hit our timeout
threshold. More recently, it seems to always just timeout, but as far
back as our history goes with testing on this platform, it's always
seemed bad to some extent. I tried the exact same test on another
similar system, and it completed in only 11min. so we suspected a bad
disk. After replacing the disk with a new one, it still has the same
problem.
We only see this problem on the core18 i386 images though, not on
core16, and not on core18-amd64.
Looking into it some more, I discovered that it was easily reproducible
outside the test suite, but *only* if I tried running it after the full test
suite without rebooting. If I reboot the system, and run the bonnie++ command
above without the rest of the tests, it completes in about 13 min.
I divided up the other tests and narrowed it down to a cpu offlining test -
when that test runs, followed by this disk stress test, that's when it gets so
horribly slow.
This was all done using checkbox-snappy as the test suite, but if you
have another way to get bonnie++ on here, it can be reproduced without
it (or you can use bonnie++ from checkbox-snappy easily by doing 'sudo
/snap/bin/checkbox-snappy.shell')
Here's the basic instructions to reproduce (on core18-i386):
$ sudo bash -c "echo 0 > /sys/devices/system/cpu/cpu1/online"
$ sudo bash -c "echo 0 > /sys/devices/system/cpu/cpu2/online"
$ sudo bash -c "echo 0 > /sys/devices/system/cpu/cpu3/online"
$ sudo bash -c "echo 1 > /sys/devices/system/cpu/cpu1/online"
$ sudo bash -c "echo 1 > /sys/devices/system/cpu/cpu2/online"
$ sudo bash -c "echo 1 > /sys/devices/system/cpu/cpu3/online"
$ sudo /snap/bin/checkbox-snappy.shell
# bonnie++ -d /var/lib/snapd/hostfs/writable -u root -r 8000
Nothing particularly interesting in dmesg during all of this other than the
NOHZ messages during cpu offline:
[ 189.085026] smpboot: CPU 1 is now offline
[ 199.371529] smpboot: CPU 2 is now offline
[ 203.291835] NOHZ: local_softirq_pending 282
[ 203.293167] NOHZ: local_softirq_pending 282
[ 203.294581] smpboot: CPU 3 is now offline
[ 247.521569] x86: Booting SMP configuration:
[ 247.521576] smpboot: Booting Node 0 Processor 1 APIC 0x2
[ 247.522892] Initializing CPU#1
[ 252.127965] smpboot: Booting Node 0 Processor 2 APIC 0x1
[ 252.129533] Initializing CPU#2
[ 255.632004] smpboot: Booting Node 0 Processor 3 APIC 0x3
[ 255.633565] Initializing CPU#3
History I have for the past few kernels:
4.15.0-52.56-226 - passed - 41m
4.15.0-54.58-241 - passed - 36m
4.15.0-55.60-246 - passed after some timeouts/retries, took 41m though
4.15.0-55.60-253 - passed in 20min
4.15.0-56.62-263 - fails - seems to take much much longer from this point on
*These results were from that test as part of the full test run,
including the cpu offlining test. When the test is run by itself on
the same system, it takes only around 13min
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841841/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
