** Tags removed: verification-needed-xenial ** Tags added: verification-done-xenial
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1765241 Title: virtio_scsi race can corrupt memory, panic kernel Status in linux package in Ubuntu: Confirmed Status in linux source package in Xenial: Fix Committed Bug description: There's a race in the virtio_scsi driver (for all kernels) That race is inadvertently avoided on most kernels due to a synchronize_rcu call coincidentally placed in one of the racing code paths By happenstance, the set of patches backported to the Ubuntu 4.4 kernel ended up without a synchronize_rcu in the relevant place. The issue first manifests with commit be2a20802abbde04ae09846406d7b670731d97d2 Author: Jan Kara <j...@suse.cz> Date: Wed Feb 8 08:05:56 2017 +0100 block: Move bdi_unregister() to del_gendisk() BugLink: http://bugs.launchpad.net/bugs/1659111 The race can cause a kernel panic due to corruption of a freelist pointer in a slab cache. The panics occur in arbitrary places as the failure occurs at an allocation after the corruption of the pointer. However, the most common failure observed has been within virtio_scsi itself during probe processing, e.g.: [ 3.111628] [<ffffffff811b0f32>] kfree_const+0x22/0x30 [ 3.112340] [<ffffffff813db534>] kobject_release+0x94/0x190 [ 3.113126] [<ffffffff813db3c7>] kobject_put+0x27/0x50 [ 3.113838] [<ffffffff8153dee7>] put_device+0x17/0x20 [ 3.114568] [<ffffffff815ac6b2>] __scsi_remove_device+0x92/0xe0 [ 3.115401] [<ffffffff815a928b>] scsi_probe_and_add_lun+0x95b/0xe80 [ 3.116287] [<ffffffff811f1083>] ? kmem_cache_alloc_trace+0x183/0x1f0 [ 3.117227] [<ffffffff8154eb0b>] ? __pm_runtime_resume+0x5b/0x80 [ 3.118048] [<ffffffff815a9eaa>] __scsi_scan_target+0x10a/0x690 [ 3.118879] [<ffffffff815aa59e>] scsi_scan_channel+0x7e/0xa0 [ 3.119653] [<ffffffff815aa743>] scsi_scan_host_selected+0xf3/0x160 [ 3.120506] [<ffffffff815aa83d>] do_scsi_scan_host+0x8d/0x90 [ 3.121295] [<ffffffff815aaa0c>] do_scan_async+0x1c/0x190 [ 3.122048] [<ffffffff810a5748>] async_run_entry_fn+0x48/0x150 [ 3.122846] [<ffffffff8109c6b5>] process_one_work+0x165/0x480 [ 3.123732] [<ffffffff8109ca1b>] worker_thread+0x4b/0x4d0 [ 3.124508] [<ffffffff8109c9d0>] ? process_one_work+0x480/0x480 Details on the race: CPU A: virtscsi_probe [...] __scsi_scan_target scsi_probe_and_add_lun [on return calls __scsi_remove_device, below] scsi_probe_lun [...] blk_execute_rq blk_execute_rq waits for the completion event, and then on wakeup returns up to scsi_probe_and_all_lun, which calls __scsi_remove_device. In order for the race to occur, the wakeup must occur on a CPU other than CPU B. After being woken up by the completion of the request, the call returns up the stack to scsi_probe_and_add_lun, which calls __scsi_remove_device: __scsi_remove_device blk_cleanup_queue [ no longer calls bdi_unregister ] scsi_target_reap(scsi_target(sdev)) scsi_target_reap_ref_put kref_put kref_sub scsi_target_reap_ref_release scsi_target_destroy ->target_destroy() = virtscsi_target_destroy kfree(tgt) <=== FREE TGT Note that the removal of the call to bdi_unregister in commit xenial be2a20802abbde block: Move bdi_unregister() to del_gendisk() and annotated above is the change that gates whether the issue manifests or not. The other code change from be2a20802abbde has no effect on the race. CPU B: vring_interrupt virtscsi_complete_cmd scsi_mq_done (via ->scsi_done()) scsi_mq_done blk_mq_complete_request __blk_mq_complete_request [...] blk_end_sync_rq complete [ wake up the task from CPU A ] After waking the CPU A task, execution returns up the stack, and then calls atomic_dec(&tgt->reqs) in virtscsi_complete_cmd immediately after returning from the call to ->scsi_done. If the activity on CPU A after it is woken up (starting at __scsi_remove_device) finishes before CPU B can call atomic_dec() in virtscsi_complete_cmd, then the atomic_dec() will modify a free list pointer in the freed slab object that contained tgt. This causes the system to panic on a subsequent allocation from the per-cpu slab cache. The call path on CPU B is significantly shorter than that on CPU A after wakeup, so it is likely that an external event delays CPU B. This could be either an interrupt within the VM or scheduling delays for the virtual cpu process on the hypervisor. Whatever the delay is, it is not the root cause but merely the triggering event. The virtscsi race window described above exists in all kernels that have been checked (4.4 upstream LTS, Ubuntu 4.13 and 4.15, and current mainline as of this writing). However, none of those kernels exhibit the panic in testing, only the Ubuntu 4.4 kernel after commit be2a20802abbde. The reason none of those kernels panic is they all have one thing in common: an incidental call to synchronize_rcu somewhere in the call path of CPU A after it is woken up. This causes CPU A to wait for CPU B's activity, as CPU A will not go on to free the "tgt" memory until after the RCU grace period passes, which requires waiting for CPU B's activity to finish. Note that the specific RCU sync call is different between some of those kernel versions, but all of them have one somewhere deep inside blk_cleanup_queue. The bdi_unregister function has one (in the call to bdi_remove_from_list), which is why removing that call opens the race window on the Ubuntu 4.4 kernel. Resolving the issue can be accomplished by adding an RCU sync to virtscsi_target_destroy prior to freeing the target. It is also possible to use a loop of the format: + while (atomic_read(&tgt->reqs)) + cpu_relax(); but this is higher risk as the loop is non-terminating in the case of other failure. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1765241/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
