On Mon, Jul 13, 2026 at 09:20:25PM +0800, Jinqian Yang wrote:
> virtnet_poll_cleantx() contains a do-while loop that cleans up
> transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check
> whether more buffers need processing. When the virtio backend stops
> responding during guest reboot, used->idx is never updated, so
> virtqueue_enable_cb_delayed() always returns false and the loop never
> terminates. Then it will block reboot process, and the guest will hang.
>
> The problem occurs during guest reboot under network traffic:
>
> 1. kernel_restart() -> device_shutdown() traverses the device list
> 2. virtio_dev_shutdown() calls virtio_break_device() which sets
> vq->broken = true
> 3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait
> for in-flight callbacks to complete
> 4. A virtio interrupt fires, softirq is deferred to ksoftirqd which
> calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx()
> 5. virtnet_poll_cleantx() enters the do-while loop and never exits
> because the QEMU backend has stopped updating used->idx, despite
> vq->broken having been set to true in step 2.
>
> Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is
> visible to the scheduler and does not trigger a hard lockup. However,
> the kthread never leaves the loop, so RCU detects it as a CPU stall
> and reports it periodically. Meanwhile, the reboot process remains
> blocked in device_shutdown() because virtio_dev_shutdown() cannot
> complete its synchronization step, and the guest hangs permanently.
>
> This can be reproduced on a guest with a virtio-net device: run iperf3
> traffic in the guest, then trigger reboot. The reboot occasionally hangs
> permanently with RCU stall on ksoftirqd.
>
> Observed on ARM64 KVM guest:
>
> CPU#1 RCU stall (ksoftirqd/1), repeated periodically:
> virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <-
> net_rx_action <- handle_softirqs <- run_ksoftirqd <-
> smpboot_thread_fn <- kthread
>
> Fix by adding a virtqueue_is_broken() check to the loop condition, so
> that the loop exits immediately when the device is broken, allowing
> the device shutdown to proceed.
>
> Signed-off-by: Jinqian Yang <[email protected]>
I'd expect lots of drivers have this issue? Wouldn't it make more sense
to check virtqueue_is_broken in
virtqueue_enable_cb_delayed/virtqueue_enable_cb? This way it works for
all drivers.
> ---
> drivers/net/virtio_net.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 7d2eeb9b1226..c8d2d420c31d 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -2970,7 +2970,8 @@ static void virtnet_poll_cleantx(struct receive_queue
> *rq, int budget)
> do {
> virtqueue_disable_cb(sq->vq);
> free_old_xmit(sq, txq, !!budget);
> - } while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
> + } while (!virtqueue_is_broken(sq->vq) &&
> + unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
>
> if (sq->vq->num_free >= MAX_SKB_FRAGS + 2)
> virtnet_tx_wake_queue(vi, sq);
> --
> 2.33.0
