On Wed, Nov 08, 2017 at 11:56:02AM -0000, ChristianEhrhardt wrote: > Torkoal (our Jenkins node) was idle atm and Ryan reported he had seen the > issues there before, so trying there as well. > This is LTS + HWE - Kernel 4.10.0-38-generic, qemu: 1:2.5+dfsg-5ubuntu10 > > I thought about your case since you seem just to start a lot of them and > reboot, > this shouldn't be so much different to: > $ uvt-simplestreams-libvirt --verbose sync --source > http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=artful > $ for i in {1..30}; do uvt-kvm create --log-console-output --password=ubuntu > artful-${i}-bug1730717 release=artful arch=amd64 label=daily; done > $ for i in {1..30}; do uvt-kvm wait --insecure artful-${i}-bug1730717; done > $ for i in {1..30}; do uvt-kvm ssh --insecure artful-${i}-bug1730717 "sudo > reboot"; done > $ sudo grep "soft lockup" /var/log/libvirt/qemu/artful-*-bug1730717.log
Sounds like it's similar, but maybe you have to put the system under load - you might need more instances, or maybe start a whole bunch first and get them to run something memory intensive before running that same test again. In the cloud there will be buildds and tests running on the compute nodes too, as well as these 'empty' instances that I use to reproduce the problem. > Waiting for your feedback if you can trigger the same issue on a non- > busy openstack system (could after all be some openstack magic at work > that makes it behave differently). I don't have access to a non busy cloud I'm afraid. ANYWAY! My results are in. I created an image by booting the stock artful cloud image and installing the mainline kernel v4.14-rc8 (39dae59d66acd86d1de24294bd2f343fd5e7a625) packages, on lcy01 (the busy cloud that exhibits this problem). I started 34 (17 × 2 in two runs - that's all I could squeeze in before I hit my quota) instances, and they were all good. This isn't definitive proof, but it looks like that kernel might be good. Cheers, -- Iain Lane [ i...@orangesquash.org.uk ] Debian Developer [ la...@debian.org ] Ubuntu Developer [ la...@ubuntu.com ] -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1730717 Title: Some VMs fail to reboot with "watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]" Status in linux package in Ubuntu: Triaged Status in qemu-kvm package in Ubuntu: New Status in linux source package in Artful: Triaged Status in qemu-kvm source package in Artful: New Status in linux source package in Bionic: Triaged Status in qemu-kvm source package in Bionic: New Bug description: This is impacting us for ubuntu autopkgtests. Eventually the whole region ends up dying because each worker is hit by this bug in turn and backs off until the next reset (6 hourly). 17.10 (and bionic) guests are sometimes failing to reboot. When this happens, you see the following in the console [[0;32m OK [0m] Reached target Shutdown. [ 191.698969] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1] [ 219.698438] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1] [ 226.702150] INFO: rcu_sched detected stalls on CPUs/tasks: [ 226.704958] »(detected by 0, t=15002 jiffies, g=5347, c=5346, q=187) [ 226.706093] All QSes seen, last rcu_sched kthread activity 15002 (4294949060-4294934058), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 226.708202] rcu_sched kthread starved for 15002 jiffies! g5347 c5346 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 One host that exhibits this behaviour was: Linux klock 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux guest running: Linux version 4.13.0-16-generic (buildd@lcy01-02) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu2)) #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 (Ubuntu 4.13.0-16.19-generic 4.13.4) The affected cloud region is running the xenial/Ocata cloud archive, so the version of qemu-kvm in there may also be relevant. Here's how I reproduced it in lcy01: $ for n in {1..30}; do nova boot --flavor m1.small --image ubuntu/ubuntu-artful-17.10-amd64-server-20171026.1-disk1.img --key-name testbed-`hostname` --nic net-name=net_ues_proposed_migration laney-test${n}; done $ <ssh to each instance> sudo reboot # wait a minute or so for the instances to all reboot $ for n in {1..30}; do echo "=== ${n} ==="; nova console-log laney-test${n} | tail; done On bad instances you'll see the "soft lockup" message - on good it'll reboot as normal. We've seen good and bad instances on multiple compute hosts - it doesn't feel to me like a host problem but rather a race condition somewhere that's somehow either triggered or triggered much more often by what lcy01 is running. I always saw this on the first reboot - never on first boot, and never on n>1th boot. (But if it's a race then that might not mean much.) I'll attach a bad and a good console-log for reference. If you're at Canonical then see internal rt #107135 for some other details. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1730717/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
