kworker/u32:7(f2fs writeback thread) was trying to acquire
sbi->writepages, the mutex was already acquired by emulated;0.
kworker/u32:7 tried to flush plugged IO before sleep, during
flushing plug list, kworker/u32:7 got preempted in RCU read-side
critical section and got scheduled out waiting to be woken up by
the release of mutex. emulated;0 was blocked on blk_mq_get_tag
as there was no available tag and expected the previous IO
requests to be handled by UFS host to release tags, but UFS host
was blocked on synchronize_rcu waiting for ending of RCU grace
period. the deadlock was caused by writeback thread was stuck in
RCU critical section waiting for mutex. call blk_flush_plug() if
mutex_trylock(&sbi->writepages) failed, so that the plug list is
empty when blk_flush_plug() is called in sched_submit_work to
fix the deadlock.
Task name: kworker/u32:7 [affinity: 0xff] pid: 233 tgid: 233 cpu:
6 prio: 98 start: 0xffffff8099390040
state: 0x2[D] exit_state: 0x0 stack base: 0xffffffc083c48000
Last_enqueued_ts: 390.169395877 Last_sleep_ts: 389.756639574
Stack:
[<ffffffd3a0b1d254>] __switch_to+0x214
[<ffffffd3a0b1deb8>] __schedule+0xa30
[<ffffffd3a0b1e754>] preempt_schedule_notrace+0x68
[<ffffffd39f9ddf38>] rcu_is_watching[jt]+0x5c
[<ffffffd39f9995b8>] lock_acquire+0x68
[<ffffffd39ffe870c>] rcu_lock_acquire+0x34
[<ffffffd39ffe8764>] percpu_ref_put_many+0x20
[<ffffffd39ffe3a64>] blk_mq_dispatch_list+0x570
[<ffffffd39ffe3264>] blk_mq_flush_plug_list+0x13c
[<ffffffd39ffd43a8>] __blk_flush_plug+0x11c
[<ffffffd39f93e6c4>] sched_submit_work+0x78
[<ffffffd3a0b1e4d8>] schedule+0x38
[<ffffffd3a0b1e5c4>] schedule_preempt_disabled+0x18
[<ffffffd3a0b20c7c>] __mutex_lock_common+0xab8
[<ffffffd3a0b20084>] mutex_lock_nested+0x2c
[<ffffffd39feea72c>] f2fs_write_data_pages+0xd64
[<ffffffd39fbb3958>] do_writepages+0xd4
[<ffffffd39fd09fd8>] __writeback_single_inode+0x78
[<ffffffd39fd09958>] writeback_sb_inodes+0x2b8
[<ffffffd39fd09cc0>] __writeback_inodes_wb+0xa0
[<ffffffd39fd09054>] wb_writeback+0x188
[<ffffffd39fd066d0>] wb_workfn[jt]+0x438
[<ffffffd39f9198e0>] process_one_work+0x27c
[<ffffffd39f91baf8>] worker_thread+0x358
[<ffffffd39f924534>] kthread+0x150
[<ffffffd39f830e44>] ret_from_fork+0x10
Task name: kworker/u32:2 [affinity: 0xff] pid: 90 tgid: 90 cpu:
1 prio: 120 start: 0xffffff80adae8040
state: 0x2[D] exit_state: 0x0 stack base: 0xffffffc080a18000
Last_enqueued_ts: 389.899608637 Last_sleep_ts: 389.899665303
Stack:
[<ffffffd3a0b1d254>] __switch_to+0x214
[<ffffffd3a0b1deb8>] __schedule+0xa30
[<ffffffd3a0b1e4e8>] schedule+0x48
[<ffffffd39f9dfa64>] synchronize_rcu_expedited+0x928
[<ffffffd39f9deeb0>] synchronize_rcu[jt]+0x234
[<ffffffd39ffde5d4>] blk_mq_quiesce_tagset[jt]+0xa8
[<ffffffd3a061cd90>] ufshcd_devfreq_scale+0x90
[<ffffffd3a061ca6c>] ufshcd_devfreq_target+0x204
[<ffffffd3a06de244>] devfreq_set_target+0xb8
[<ffffffd3a06de14c>] devfreq_update_target[jt]+0xd8
[<ffffffd3a06de5e8>] devfreq_monitor+0x38
[<ffffffd39f9198e0>] process_one_work+0x27c
[<ffffffd39f91baf8>] worker_thread+0x358
[<ffffffd39f924534>] kthread+0x150
[<ffffffd39f830e44>] ret_from_fork+0x10
Task name: emulated;0 [affinity: 0xff] pid: 5245 tgid: 4922 cpu:
6 prio: 120 start: 0xffffff89e2e60040
state: 0x2[D] exit_state: 0x0 stack base: 0xffffffc0d42f8000
Last_enqueued_ts: 390.169395877 Last_sleep_ts: 390.178459731
Stack:
[<ffffffd3a0b1d254>] __switch_to+0x214
[<ffffffd3a0b1deb8>] __schedule+0xa30
[<ffffffd3a0b1e4e8>] schedule+0x48
[<ffffffd3a0b1e8f4>] io_schedule+0x38
[<ffffffd39ffeb110>] blk_mq_get_tag+0x1a4
[<ffffffd39ffdedc8>] __blk_mq_alloc_requests+0x358
[<ffffffd39ffe409c>] blk_mq_submit_bio+0x50c
[<ffffffd39ffd4cb8>] __submit_bio[jt]+0x164
[<ffffffd39ffd2e7c>] submit_bio_noacct_nocheck+0x14c
[<ffffffd39ffd3394>] submit_bio_noacct+0x330
[<ffffffd39ffd3814>] submit_bio+0x1f4
[<ffffffd39fee2ba8>] f2fs_submit_write_bio+0x88
[<ffffffd39fee413c>] __submit_merged_bio[jt]+0xbc
[<ffffffd39fee3c4c>] f2fs_submit_page_write+0x400
[<ffffffd39ff04aac>] do_write_page+0x180
[<ffffffd39ff04ca0>] f2fs_outplace_write_data+0x78
[<ffffffd39fee88d4>] f2fs_do_write_data_page+0x390
[<ffffffd39fee8df0>] f2fs_write_single_data_page+0x1e0
[<ffffffd39feea288>] f2fs_write_data_pages+0x8c0
[<ffffffd39fbb3958>] do_writepages+0xd4
[<ffffffd39fba153c>] __filemap_fdatawrite_range+0x94
[<ffffffd39fbb0370>] generic_fadvise+0x1d8
[<ffffffd39feabaa0>] f2fs_file_fadvise+0x124
[<ffffffd39fbb0570>] __arm64_sys_fadvise64_64+0x70
[<ffffffd39f847e7c>] invoke_syscall+0x58
[<ffffffd39f847da4>] el0_svc_common[jt]+0xb8
[<ffffffd39f847d18>] do_el0_svc+0x1c
[<ffffffd3a0b14fac>] el0_svc+0x40
[<ffffffd3a0b14ef8>] el0t_64_sync_handler[jt]+0xd0
[<ffffffd39f8116a0>] ret_to_user[jt]+0x0
Signed-off-by: Xiaosen He <[email protected]>
---
fs/f2fs/data.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 338df7a2aea6..c8e81f63fe73 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -3585,7 +3585,13 @@ static int __f2fs_write_data_pages(struct address_space
*mapping,
}
if (__should_serialize_io(inode, wbc)) {
+ if (!mutex_trylock(&sbi->writepages))
+ blk_flush_plug(((struct task_struct *)current)->plug,
true);
+ else
+ goto set_locked;
+
mutex_lock(&sbi->writepages);
+set_locked:
locked = true;
}
--
2.34.1
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
