I get this every couple of days on our CI - from processes which access
the GPU. nvidia-driver 396.24.02. Here are some dmesg warning from
various failures

[Mon Nov  5 18:43:29 2018] INFO: task kworker/4:2:25281 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/4:2     D ffff88026a5cbb68     0 25281      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff88026a5cbb68 0000000000000036 ffff880827175a00 
ffff88026a5cc000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff880827175a00
[Mon Nov  5 18:43:29 2018]  ffff88026a5cbb80 ffffffff81818105 7fffffffffffffff 
ffff88026a5cbc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3701>] ? kmem_cache_alloc+0x191/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/4:1:3562 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/4:1     D ffff880104db3b68     0  3562      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff880104db3b68 ffffffff81817b46 ffff88044e1f8f00 
ffff880104db4000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff88044e1f8f00
[Mon Nov  5 18:43:29 2018]  ffff880104db3b80 ffffffff81818105 7fffffffffffffff 
ffff880104db3c28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:3:16616 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:3     D ffff8805f97a3b68     0 16616      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff8805f97a3b68 ffffffff81817b46 ffff8805f5952d00 
ffff8805f97a4000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff8805f5952d00
[Mon Nov  5 18:43:29 2018]  ffff8805f97a3b80 ffffffff81818105 7fffffffffffffff 
ffff8805f97a3c28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3701>] ? kmem_cache_alloc+0x191/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:4:16617 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:4     D ffff8805f978bb68     0 16617      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff8805f978bb68 ffffffff810b73bf ffff8805f5951e00 
ffff8805f978c000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff8805f5951e00
[Mon Nov  5 18:43:29 2018]  ffff8805f978bb80 ffffffff81818105 7fffffffffffffff 
ffff8805f978bc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b8c6d>] ? 
check_preempt_wakeup+0x12d/0x210
[Mon Nov  5 18:43:29 2018]  [<ffffffff810aa577>] ? check_preempt_curr+0x87/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task py.test:8373 blocked for more than 120 
seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] py.test         D ffff880377c1bb68     0  8373   
8372 0x00000000
[Mon Nov  5 18:43:29 2018]  ffff880377c1bb68 0000000000000000 ffff88071774cb00 
ffff880377c1c000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff88071774cb00
[Mon Nov  5 18:43:29 2018]  ffff880377c1bb80 ffffffff81818105 7fffffffffffffff 
ffff880377c1bc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725100>] ? os_free_semaphore+0x10/0x10 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cc6008>] ? _nv007547rm+0xa8/0x150 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d5a7c0>] ? _nv001114rm+0x4c0/0x850 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d656f3>] ? rm_ioctl+0x73/0x100 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071d601>] ? nvidia_ioctl+0x91/0x6c0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071d601>] ? nvidia_ioctl+0x91/0x6c0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071db11>] ? nvidia_ioctl+0x5a1/0x6c0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0719083>] ? 
nvidia_frontend_unlocked_ioctl+0x43/0x50 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81219157>] ? do_vfs_ioctl+0x2e7/0x4d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c8ca7>] ? percpu_up_read+0x17/0x40
[Mon Nov  5 18:43:29 2018]  [<ffffffff81208131>] ? __sb_end_write+0x21/0x30
[Mon Nov  5 18:43:29 2018]  [<ffffffff81206195>] ? vfs_write+0x155/0x1a0
[Mon Nov  5 18:43:29 2018]  [<ffffffff812193b9>] ? SyS_ioctl+0x79/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181ba08>] ? 
entry_SYSCALL_64_fastpath+0x1c/0xbb
[Mon Nov  5 18:43:29 2018] INFO: task py.test:8603 blocked for more than 120 
seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] py.test         D ffff88011416bcc8     0  8603   
8372 0x00000000
[Mon Nov  5 18:43:29 2018]  ffff88011416bcc8 ffff88011416bdb0 ffff88008f51da00 
ffff88011416c000
[Mon Nov  5 18:43:29 2018]  ffff880825b56638 0000000000000002 0000000000000000 
ffff88008f51da00
[Mon Nov  5 18:43:29 2018]  ffff88011416bce0 ffffffff81818105 7fffffffffffffff 
ffff88011416bd88
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810fe25d>] ? futex_wait+0x1ad/0x270
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071d5d0>] nvidia_ioctl+0x60/0x6c0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0719083>] 
nvidia_frontend_unlocked_ioctl+0x43/0x50 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81219157>] do_vfs_ioctl+0x2e7/0x4d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b86>] ? __schedule+0x2d6/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff811009b1>] ? SyS_futex+0x71/0x150
[Mon Nov  5 18:43:29 2018]  [<ffffffff812193b9>] SyS_ioctl+0x79/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff810eeb21>] ? 
posix_get_monotonic_raw+0x11/0x20
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181ba08>] 
entry_SYSCALL_64_fastpath+0x1c/0xbb
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:0:8594 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:0     D ffff8804c115fb68     0  8594      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff8804c115fb68 ffff8804c115fb70 ffff880823618000 
ffff8804c1160000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff880823618000
[Mon Nov  5 18:43:29 2018]  ffff8804c115fb80 ffffffff81818105 7fffffffffffffff 
ffff8804c115fc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b475fc>] ? _nv007760rm+0x3c/0x180 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3701>] ? kmem_cache_alloc+0x191/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:2:14982 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:2     D ffff880101fb7b68     0 14982      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff880101fb7b68 ffff880101fb7b70 ffff8807025c6900 
ffff880101fb8000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff8807025c6900
[Mon Nov  5 18:43:29 2018]  ffff880101fb7b80 ffffffff81818105 7fffffffffffffff 
ffff880101fb7c28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b5b40>] ? update_curr+0x80/0x170
[Mon Nov  5 18:43:29 2018]  [<ffffffff8104f870>] ? 
native_smp_send_reschedule+0x50/0x70
[Mon Nov  5 18:43:29 2018]  [<ffffffff810a9b06>] ? resched_curr+0xb6/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/4:0:15088 blocked for more than 
120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] kworker/4:0     D ffff88010d7afb68     0 15088      
2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff88010d7afb68 ffff88010d7afb70 ffff880716fc0000 
ffff88010d7b0000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff880716fc0000
[Mon Nov  5 18:43:29 2018]  ffff88010d7afb80 ffffffff81818105 7fffffffffffffff 
ffff88010d7afc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b5b40>] ? update_curr+0x80/0x170
[Mon Nov  5 18:43:29 2018]  [<ffffffff8104f870>] ? 
native_smp_send_reschedule+0x50/0x70
[Mon Nov  5 18:43:29 2018]  [<ffffffff810a9b06>] ? resched_curr+0xb6/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? 
rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? 
os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? 
os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task ffmpeg:15818 blocked for more than 120 
seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic 
#153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[Mon Nov  5 18:43:29 2018] ffmpeg          D ffff88012f2cf958     0 15818  
15825 0x00000000
[Mon Nov  5 18:43:29 2018]  ffff88012f2cf958 ffffffff810b73bf ffff880114a2da00 
ffff88012f2d0000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 
ffff880114a2da00
[Mon Nov  5 18:43:29 2018]  ffff88012f2cf970 ffffffff81818105 7fffffffffffffff 
ffff88012f2cfa18
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b8c89>] ? 
check_preempt_wakeup+0x149/0x210
[Mon Nov  5 18:43:29 2018]  [<ffffffff810aa577>] ? check_preempt_curr+0x87/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725100>] ? os_free_semaphore+0x10/0x10 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d68960>] ? 
rm_get_gpu_uuid_raw+0x70/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff810ab555>] ? wake_up_process+0x15/0x20
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071aae2>] ? nv_open_device+0x4c2/0x700 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071b118>] ? nvidia_open+0x148/0x2f0 
[nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0719388>] ? 
nvidia_frontend_open+0x58/0xa0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff8120a884>] ? chrdev_open+0xb4/0x190
[Mon Nov  5 18:43:29 2018]  [<ffffffff81203d2a>] ? do_dentry_open+0x22a/0x320
[Mon Nov  5 18:43:29 2018]  [<ffffffff8120a7d0>] ? cdev_put+0x30/0x30
[Mon Nov  5 18:43:29 2018]  [<ffffffff81204df5>] ? vfs_open+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff81210d97>] ? may_open+0x57/0xe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff8121328d>] ? path_openat+0x1ad/0x12e0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81221d69>] ? atime_needs_update+0x69/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff8121602e>] ? do_filp_open+0x7e/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81223486>] ? __alloc_fd+0x46/0x180
[Mon Nov  5 18:43:29 2018]  [<ffffffff81205179>] ? do_sys_open+0x129/0x270
[Mon Nov  5 18:43:29 2018]  [<ffffffff812052de>] ? SyS_open+0x1e/0x20
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181ba08>] ? 
entry_SYSCALL_64_fastpath+0x1c/0xbb

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1767932

Title:
  nvidia-390 causes kernel hang

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-390/+bug/1767932/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to