Reported-by: [email protected] Closes: https://syzkaller.appspot.com/bug?extid=53706c567afab5131044 ("INFO: task hung in __rq_qos_throttle (2)", still open upstream, no fix, bisection failed, 97 crashes since 2026-08-08)
virtscsi_eh_timed_out() always returns SCSI_EH_RESET_TIMER, trusting the host to eventually answer every command. A privileged raw write to /sys/bus/pci/devices/*/config that clears PCI_COMMAND_MASTER on the disk breaks that assumption: the device can no longer DMA, so no completion -- real or error -- ever arrives, and the reset loop runs forever. Any submitter that then hits wbt's inflight limit blocks uninterruptibly in wbt_wait() with no way to recover. The syzbot C reproducer does exactly this: it opens the disk's PCI config sysfs file and pwrite()s the two bytes 02 00 at offset 4 (PCI_COMMAND), clearing PCI_COMMAND_MASTER while keeping PCI_COMMAND_MEMORY set. The hung task is a writeback worker stuck in wbt_wait() <- __rq_qos_throttle() <- blk_mq_get_new_requests(), on a disk named sdaN -- i.e. a SCSI-model disk, consistent with virtio_scsi rather than virtio-blk. Bound the resets: after VIRTSCSI_EH_RESET_LIMIT tries (~150s), let real SCSI EH run (abort -> device reset -> offline), which fails the command and unblocks rq_qos waiters. A host that is merely slow for longer than that now gets its commands aborted/offlined instead of waited on indefinitely -- a deliberate tradeoff. virtscsi_tmf(), used by both abort and device-reset handlers, waited on the same broken ctrl virtqueue unboundedly, which would just move the hang into the EH thread. Bound it too. On timeout the command is left queued on the ctrl vq (a device that answers late must still be able to find it), so it is not freed. Clearing cmd->comp so a late completion doesn't write through the now-invalid on-stack completion has to be serialized against virtscsi_complete_free(), which reads cmd->comp and calls complete() on it under ctrl_vq.vq_lock -- taking that same lock around the check-and-clear (and re-checking completion_done() inside it) closes the race instead of just shrinking it. The leak is one virtio_scsi_cmd per timed-out TMF, bounded by queue depth, on a path where the device is being offlined anyway. Verified with a QEMU virtio-scsi repro that reproduces the syzbot mechanism above (pwrite of 02 00 at PCI config offset 4 on the sda backing device, mid-writeback, via /sys/bus/pci/devices/*/config): unpatched, a request sits stuck past its own 30s timeout with zero EH activity even after 100+s; patched, eh_resets increments deterministically, SCSI EH aborts/resets/offlines the device at the expected ~150-190s mark, and both previously-stuck writers come back with -EIO instead of hanging. Full dmesg from the patched run was checked for WARN/BUG/lockdep output around the new ctrl_vq.vq_lock critical section in virtscsi_tmf(); none appeared. Signed-off-by: Nguyen Ngoc Thang <[email protected]> --- drivers/scsi/virtio_scsi.c | 51 ++++++++++++++++++++++++++++++++++---- 1 file changed, 46 insertions(+), 5 deletions(-) diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c index 35731b18c519..b4f20c487718 100644 --- a/drivers/scsi/virtio_scsi.c +++ b/drivers/scsi/virtio_scsi.c @@ -37,6 +37,11 @@ #define VIRTIO_SCSI_EVENT_LEN 8 #define VIRTIO_SCSI_VQ_BASE 2 +/* Max timer resets in virtscsi_eh_timed_out() before letting real EH run. */ +#define VIRTSCSI_EH_RESET_LIMIT 5 +/* How long to let the host answer an abort/reset TMF before giving up. */ +#define VIRTSCSI_TMF_TIMEOUT (10 * HZ) + static unsigned int virtscsi_poll_queues; module_param(virtscsi_poll_queues, uint, 0644); MODULE_PARM_DESC(virtscsi_poll_queues, @@ -46,6 +51,7 @@ MODULE_PARM_DESC(virtscsi_poll_queues, struct virtio_scsi_cmd { struct scsi_cmnd *sc; struct completion *comp; + unsigned int eh_resets; union { struct virtio_scsi_cmd_req cmd; struct virtio_scsi_cmd_req_pi cmd_pi; @@ -586,6 +592,7 @@ static enum scsi_qc_status virtscsi_queuecommand(struct Scsi_Host *shost, "cmd %p CDB: %#02x\n", sc, sc->cmnd[0]); cmd->sc = sc; + cmd->eh_resets = 0; BUG_ON(sc->cmd_len > VIRTIO_SCSI_CDB_SIZE); @@ -625,7 +632,35 @@ static int virtscsi_tmf(struct virtio_scsi *vscsi, struct virtio_scsi_cmd *cmd) sizeof cmd->req.tmf, sizeof cmd->resp.tmf, true) < 0) goto out; - wait_for_completion(&comp); + if (!wait_for_completion_timeout(&comp, VIRTSCSI_TMF_TIMEOUT)) { + unsigned long flags; + bool completed; + + /* + * No answer within the timeout. virtscsi_complete_free() + * reads cmd->comp and calls complete() on it under + * ctrl_vq.vq_lock, so take the same lock to decide, atomically + * with that path, whether the completion already happened. + * + * If it hasn't: clear cmd->comp so a completion that arrives + * after we drop the lock finds NULL and leaves this + * soon-to-be-invalid stack frame alone. cmd stays queued on + * the ctrl vq (a device that answers late must still be able + * to find it), so it is not freed here. + * + * If it has: the response landed (and complete() already ran) + * right as we timed out, so fall through and read it as if + * wait_for_completion_timeout() had succeeded. + */ + spin_lock_irqsave(&vscsi->ctrl_vq.vq_lock, flags); + completed = completion_done(&comp); + if (!completed) + cmd->comp = NULL; + spin_unlock_irqrestore(&vscsi->ctrl_vq.vq_lock, flags); + + if (!completed) + return FAILED; + } if (cmd->resp.tmf.response == VIRTIO_SCSI_S_OK || cmd->resp.tmf.response == VIRTIO_SCSI_S_FUNCTION_SUCCEEDED) ret = SUCCESS; @@ -783,13 +818,19 @@ static void virtscsi_commit_rqs(struct Scsi_Host *shost, u16 hwq) } /* - * The host guarantees to respond to each command, although I/O - * latencies might be higher than on bare metal. Reset the timer - * unconditionally to give the host a chance to perform EH. + * The host normally answers every command, so reset the timer and keep + * waiting. But if the transport is broken (e.g. bus mastering was turned + * off), no completion can ever arrive: give up after a few resets so SCSI + * EH fails the command instead of blocking its submitter forever. */ static enum scsi_timeout_action virtscsi_eh_timed_out(struct scsi_cmnd *scmnd) { - return SCSI_EH_RESET_TIMER; + struct virtio_scsi_cmd *cmd = scsi_cmd_priv(scmnd); + + if (++cmd->eh_resets < VIRTSCSI_EH_RESET_LIMIT) + return SCSI_EH_RESET_TIMER; + + return SCSI_EH_NOT_HANDLED; } static const struct scsi_host_template virtscsi_host_template = { -- 2.43.0
