I tried the following command:
ubuntu@myserver:~$ lxd-benchmark launch --count 900 --parallel 24 ubuntu:18.04 Test environment: Server backend: lxd Server version: 3.0.2 Kernel: Linux Kernel architecture: x86_64 Kernel version: 4.15.0-36-generic Storage backend: zfs Storage version: 0.7.5-1ubuntu16.3 Container backend: lxc Container version: 3.0.2 Test variables: Container count: 900 Container mode: unprivileged Startup mode: normal startup Image: ubuntu:18.04 Batches: 37 Batch size: 24 Remainder: 12 [Oct 3 21:46:55.617] Found image in local store: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a [Oct 3 21:46:55.617] Batch processing start [Oct 3 21:47:09.310] Processed 24 containers in 13.693s (1.753/s) [Oct 3 21:47:26.739] Processed 48 containers in 31.122s (1.542/s) [Oct 3 21:48:06.052] Processed 96 containers in 70.435s (1.363/s) [Oct 3 21:49:27.340] Processed 192 containers in 151.723s (1.265/s) ^C I interrupted the benchmark because it got stuck. Note: 1. I am running Ubuntu 18.04. 2. With the updated kernel 4.15.0-36. I did not try the proposed kernel. 3. With LXD 3.0.2 (from bionic/proposed) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781601 Title: Stress-testing LXD causes kernel hung in cgroups (cgroup_destroy css_killed_work_fn) Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: Confirmed Bug description: I have been stress-testing LXD and managed to cause the following kernel hung: [10271.564074] INFO: task systemd:1 blocked for more than 120 seconds. [10271.570397] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.577212] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.585071] systemd D 0 1 0 0x00000000 [10271.585073] Call Trace: [10271.585080] __schedule+0x297/0x8b0 [10271.585086] schedule+0x2c/0x80 [10271.585090] schedule_preempt_disabled+0xe/0x10 [10271.585095] __mutex_lock.isra.2+0x18c/0x4d0 [10271.585100] __mutex_lock_slowpath+0x13/0x20 [10271.585101] ? __mutex_lock_slowpath+0x13/0x20 [10271.585102] mutex_lock+0x2f/0x40 [10271.585106] proc_cgroup_show+0x4c/0x2a0 [10271.585108] proc_single_show+0x56/0x80 [10271.585111] seq_read+0xe5/0x430 [10271.585114] __vfs_read+0x1b/0x40 [10271.585115] vfs_read+0x8e/0x130 [10271.585117] SyS_read+0x55/0xc0 [10271.585120] do_syscall_64+0x73/0x130 [10271.585121] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [10271.585123] RIP: 0033:0x7fd47634d0b4 [10271.585124] RSP: 002b:00007ffe9a186700 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [10271.585125] RAX: ffffffffffffffda RBX: 0000000000000021 RCX: 00007fd47634d0b4 [10271.585126] RDX: 0000000000000400 RSI: 000055f70a08c0c0 RDI: 0000000000000021 [10271.585127] RBP: 000055f70a08c0c0 R08: 0000000000000000 R09: 0000000000000000 [10271.585128] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400 [10271.585129] R13: 00007fd4766252a0 R14: 000055f70a0298e0 R15: 00000000000007ff [10271.585221] INFO: task lxcfs:84510 blocked for more than 120 seconds. [10271.591687] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.598531] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.606401] lxcfs D 0 84510 1 0x00000000 [10271.606407] Call Trace: [10271.606416] __schedule+0x297/0x8b0 [10271.606418] schedule+0x2c/0x80 [10271.606420] schedule_preempt_disabled+0xe/0x10 [10271.606421] __mutex_lock.isra.2+0x18c/0x4d0 [10271.606423] __mutex_lock_slowpath+0x13/0x20 [10271.606424] ? __mutex_lock_slowpath+0x13/0x20 [10271.606425] mutex_lock+0x2f/0x40 [10271.606427] proc_cgroup_show+0x4c/0x2a0 [10271.606429] proc_single_show+0x56/0x80 [10271.606432] seq_read+0xe5/0x430 [10271.606434] __vfs_read+0x1b/0x40 [10271.606436] vfs_read+0x8e/0x130 [10271.606437] SyS_read+0x55/0xc0 [10271.606440] do_syscall_64+0x73/0x130 [10271.606441] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [10271.606443] RIP: 0033:0x7f1a50e700b4 [10271.606443] RSP: 002b:00007f1a2b7fd870 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [10271.606445] RAX: ffffffffffffffda RBX: 0000000000000013 RCX: 00007f1a50e700b4 [10271.606446] RDX: 0000000000000400 RSI: 00007f1a04000f90 RDI: 0000000000000013 [10271.606446] RBP: 00007f1a04000f90 R08: 0000000000000001 R09: 0000000000000000 [10271.606447] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400 [10271.606448] R13: 00007f1a511482a0 R14: 0000000000000000 R15: 00007f1a38003a40 [10271.606450] INFO: task lxcfs:84676 blocked for more than 120 seconds. [10271.612911] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.619722] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.627577] lxcfs D 0 84676 1 0x00000000 [10271.627579] Call Trace: [10271.627583] __schedule+0x297/0x8b0 [10271.627589] schedule+0x2c/0x80 [10271.627594] schedule_preempt_disabled+0xe/0x10 [10271.627598] __mutex_lock.isra.2+0x18c/0x4d0 [10271.627600] __mutex_lock_slowpath+0x13/0x20 [10271.627601] ? __mutex_lock_slowpath+0x13/0x20 [10271.627602] mutex_lock+0x2f/0x40 [10271.627604] proc_cgroup_show+0x4c/0x2a0 [10271.627606] proc_single_show+0x56/0x80 [10271.627608] seq_read+0xe5/0x430 [10271.627610] __vfs_read+0x1b/0x40 [10271.627611] vfs_read+0x8e/0x130 [10271.627613] SyS_read+0x55/0xc0 [10271.627615] do_syscall_64+0x73/0x130 [10271.627617] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [10271.627618] RIP: 0033:0x7f1a50e700b4 [10271.627618] RSP: 002b:00007f1a297f9870 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [10271.627620] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007f1a50e700b4 [10271.627620] RDX: 0000000000000400 RSI: 00007f1a2c001e80 RDI: 0000000000000014 [10271.627621] RBP: 00007f1a2c001e80 R08: 0000000000000001 R09: 0000000000000000 [10271.627622] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400 [10271.627622] R13: 00007f1a511482a0 R14: 0000000000000000 R15: 00007f1a2c002b10 [10271.627625] INFO: task lxcfs:115507 blocked for more than 120 seconds. [10271.634180] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.641007] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.648870] lxcfs D 0 115507 1 0x00000000 [10271.648872] Call Trace: [10271.648876] __schedule+0x297/0x8b0 [10271.648883] schedule+0x2c/0x80 [10271.648887] schedule_preempt_disabled+0xe/0x10 [10271.648891] __mutex_lock.isra.2+0x18c/0x4d0 [10271.648895] __mutex_lock_slowpath+0x13/0x20 [10271.648896] ? __mutex_lock_slowpath+0x13/0x20 [10271.648897] mutex_lock+0x2f/0x40 [10271.648899] proc_cgroup_show+0x4c/0x2a0 [10271.648902] proc_single_show+0x56/0x80 [10271.648904] seq_read+0xe5/0x430 [10271.648906] __vfs_read+0x1b/0x40 [10271.648907] vfs_read+0x8e/0x130 [10271.648909] SyS_read+0x55/0xc0 [10271.648911] do_syscall_64+0x73/0x130 [10271.648913] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [10271.648914] RIP: 0033:0x7f1a50e700b4 [10271.648915] RSP: 002b:00007f1a29ffa870 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [10271.648916] RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00007f1a50e700b4 [10271.648917] RDX: 0000000000000400 RSI: 00007f19f0001820 RDI: 0000000000000015 [10271.648918] RBP: 00007f19f0001820 R08: 0000000000000001 R09: 0000000000000000 [10271.648925] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400 [10271.648928] R13: 00007f1a511482a0 R14: 0000000000000000 R15: 00007f19f00024d0 [10271.648964] INFO: task kworker/14:1:111280 blocked for more than 120 seconds. [10271.656131] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.662944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.670822] kworker/14:1 D 0 111280 2 0x80000000 [10271.670827] Workqueue: cgroup_destroy css_killed_work_fn [10271.670828] Call Trace: [10271.670832] __schedule+0x297/0x8b0 [10271.670834] schedule+0x2c/0x80 [10271.670836] schedule_preempt_disabled+0xe/0x10 [10271.670837] __mutex_lock.isra.2+0x18c/0x4d0 [10271.670841] ? wake_up_q+0x44/0x80 [10271.670842] __mutex_lock_slowpath+0x13/0x20 [10271.670843] ? __mutex_lock_slowpath+0x13/0x20 [10271.670844] mutex_lock+0x2f/0x40 [10271.670845] css_killed_work_fn+0x1f/0xe0 [10271.670848] process_one_work+0x1de/0x410 [10271.670849] worker_thread+0x32/0x410 [10271.670851] kthread+0x121/0x140 [10271.670852] ? process_one_work+0x410/0x410 [10271.670854] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.670856] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.670857] ret_from_fork+0x22/0x40 [10271.670859] INFO: task kworker/29:0:111571 blocked for more than 120 seconds. [10271.678018] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.684827] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.692681] kworker/29:0 D 0 111571 2 0x80000000 [10271.692686] Workqueue: cgroup_destroy css_release_work_fn [10271.692691] Call Trace: [10271.692694] __schedule+0x297/0x8b0 [10271.692697] ? ttwu_do_wakeup+0x1e/0x150 [10271.692699] schedule+0x2c/0x80 [10271.692700] schedule_preempt_disabled+0xe/0x10 [10271.692702] __mutex_lock.isra.2+0x18c/0x4d0 [10271.692709] ? wake_up_q+0x44/0x80 [10271.692714] __mutex_lock_slowpath+0x13/0x20 [10271.692719] ? __mutex_lock_slowpath+0x13/0x20 [10271.692723] mutex_lock+0x2f/0x40 [10271.692728] css_release_work_fn+0x2b/0x180 [10271.692733] process_one_work+0x1de/0x410 [10271.692735] worker_thread+0x32/0x410 [10271.692737] kthread+0x121/0x140 [10271.692738] ? process_one_work+0x410/0x410 [10271.692740] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.692742] ? do_syscall_64+0x115/0x130 [10271.692743] ret_from_fork+0x22/0x40 [10271.692752] INFO: task kworker/2:1:128568 blocked for more than 120 seconds. [10271.699827] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.706653] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.714524] kworker/2:1 D 0 128568 2 0x80000000 [10271.714529] Workqueue: cgroup_destroy css_release_work_fn [10271.714530] Call Trace: [10271.714534] __schedule+0x297/0x8b0 [10271.714536] schedule+0x2c/0x80 [10271.714537] schedule_preempt_disabled+0xe/0x10 [10271.714539] __mutex_lock.isra.2+0x18c/0x4d0 [10271.714540] __mutex_lock_slowpath+0x13/0x20 [10271.714541] ? __mutex_lock_slowpath+0x13/0x20 [10271.714543] mutex_lock+0x2f/0x40 [10271.714544] css_release_work_fn+0x2b/0x180 [10271.714545] process_one_work+0x1de/0x410 [10271.714547] worker_thread+0x32/0x410 [10271.714548] kthread+0x121/0x140 [10271.714550] ? process_one_work+0x410/0x410 [10271.714551] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.714552] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.714554] ret_from_fork+0x22/0x40 [10271.714564] INFO: task kworker/43:4:46368 blocked for more than 120 seconds. [10271.721637] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.728456] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.736309] kworker/43:4 D 0 46368 2 0x80000000 [10271.736314] Workqueue: cgroup_destroy css_killed_work_fn [10271.736321] Call Trace: [10271.736328] __schedule+0x297/0x8b0 [10271.736330] schedule+0x2c/0x80 [10271.736332] schedule_preempt_disabled+0xe/0x10 [10271.736333] __mutex_lock.isra.2+0x18c/0x4d0 [10271.736336] ? wake_up_q+0x44/0x80 [10271.736337] __mutex_lock_slowpath+0x13/0x20 [10271.736338] ? __mutex_lock_slowpath+0x13/0x20 [10271.736339] mutex_lock+0x2f/0x40 [10271.736341] css_killed_work_fn+0x1f/0xe0 [10271.736342] process_one_work+0x1de/0x410 [10271.736344] worker_thread+0x32/0x410 [10271.736346] kthread+0x121/0x140 [10271.736352] ? process_one_work+0x410/0x410 [10271.736357] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.736362] ret_from_fork+0x22/0x40 [10271.736367] INFO: task kworker/13:3:49251 blocked for more than 120 seconds. [10271.743439] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.750250] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.758106] kworker/13:3 D 0 49251 2 0x80000000 [10271.758110] Workqueue: cgroup_destroy css_killed_work_fn [10271.758111] Call Trace: [10271.758119] __schedule+0x297/0x8b0 [10271.758122] schedule+0x2c/0x80 [10271.758123] schedule_preempt_disabled+0xe/0x10 [10271.758124] __mutex_lock.isra.2+0x18c/0x4d0 [10271.758127] ? wake_up_q+0x44/0x80 [10271.758128] __mutex_lock_slowpath+0x13/0x20 [10271.758129] ? __mutex_lock_slowpath+0x13/0x20 [10271.758130] mutex_lock+0x2f/0x40 [10271.758131] css_killed_work_fn+0x1f/0xe0 [10271.758133] process_one_work+0x1de/0x410 [10271.758134] worker_thread+0x32/0x410 [10271.758136] kthread+0x121/0x140 [10271.758143] ? process_one_work+0x410/0x410 [10271.758148] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.758153] ret_from_fork+0x22/0x40 [10271.758161] INFO: task kworker/38:1:21823 blocked for more than 120 seconds. [10271.765236] Tainted: P OE 4.15.0-20-generic #21-Ubuntu [10271.772047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [10271.779902] kworker/38:1 D 0 21823 2 0x80000000 [10271.779907] Workqueue: cgroup_destroy css_release_work_fn [10271.779911] Call Trace: [10271.779918] __schedule+0x297/0x8b0 [10271.779920] schedule+0x2c/0x80 [10271.779922] schedule_preempt_disabled+0xe/0x10 [10271.779923] __mutex_lock.isra.2+0x18c/0x4d0 [10271.779925] ? wake_up_q+0x44/0x80 [10271.779926] __mutex_lock_slowpath+0x13/0x20 [10271.779928] ? __mutex_lock_slowpath+0x13/0x20 [10271.779929] mutex_lock+0x2f/0x40 [10271.779930] css_release_work_fn+0x2b/0x180 [10271.779935] process_one_work+0x1de/0x410 [10271.779941] worker_thread+0x32/0x410 [10271.779946] kthread+0x121/0x140 [10271.779952] ? process_one_work+0x410/0x410 [10271.779955] ? kthread_create_worker_on_cpu+0x70/0x70 [10271.779959] ret_from_fork+0x22/0x40 root@myserver:~# To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781601/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
