From: Hui Zhu <[email protected]> Add bpf_proactive_reclaim(), a sleepable kfunc which performs one proactive reclaim pass on a given memory cgroup, similar to a write to memory.reclaim but without retrying until the target is reached.
The kfunc is restricted to BPF_PROG_TYPE_SYSCALL so that reclaim always runs in a clean process context. Generic sleepable programs may execute with filesystem locks held or in NOFS/NOIO contexts, where the reclaim path could deadlock in filesystem shrinkers. A SYSCALL program can still drive reclaim asynchronously through bpf_wq or task_work callbacks, which run in process context and keep the SYSCALL program type, so they can call the kfunc too. The kfunc refuses to reclaim if the calling task is already in a reclaim context, as a nested reclaim would corrupt the outer reclaim state. The reclaim target of a single call is clamped to MEMCG_CHARGE_BATCH. try_to_free_mem_cgroup_pages() takes nr_to_reclaim as a lower bound, so nothing else limits how long one call scans. An unbounded call stalls other work on a shared workqueue, and since lru_lock is held with interrupts disabled the contention delays IPI handling and can cause CSD lock stalls. MEMCG_CHARGE_BATCH is the bound already used by high_work_func(), so each call becomes a bounded unit of work. Reclaiming more than one batch is left to the BPF program. This is documented in the kfunc rather than enforced: call the kfunc once per bpf_wq callback and requeue the same work item for the next batch instead of looping inside a callback, and give each target memcg its own bpf_wq item. Keeping the policy in BPF lets a program decide when to reclaim and when to stop, e.g. once the target cgroup is dying. Signed-off-by: Hui Zhu <[email protected]> --- mm/bpf_memcontrol.c | 95 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 93 insertions(+), 2 deletions(-) diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c index 716df49d7647..92f35ba66309 100644 --- a/mm/bpf_memcontrol.c +++ b/mm/bpf_memcontrol.c @@ -6,6 +6,7 @@ */ #include <linux/memcontrol.h> +#include <linux/swap.h> #include <linux/bpf.h> __bpf_kfunc_start_defs(); @@ -159,6 +160,74 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg) mem_cgroup_flush_stats(memcg); } +/* + * Reclaim must not recurse: try_to_free_mem_cgroup_pages() overwrites + * current->reclaim_state, so a nested call would corrupt the outer + * reclaim state. Reclaim windows are marked with PF_MEMALLOC; + * reclaim_state is also checked because it is installed slightly + * before PF_MEMALLOC. + */ +static bool bpf_in_reclaim_context(void) +{ + return (current->flags & PF_MEMALLOC) || current->reclaim_state; +} + +/** + * bpf_proactive_reclaim - proactively reclaim memory from a memory + * cgroup + * @memcg: the target memory cgroup to reclaim from + * @size: the amount of memory to reclaim, in bytes, clamped to + * MEMCG_CHARGE_BATCH (64 pages) + * + * Trigger one proactive reclaim pass on @memcg, similar to a write to + * memory.reclaim, but without retrying until @size is reached. + * + * @size is clamped so that one call is a bounded unit of work, matching + * the memory.high workqueue fallback in high_work_func(). To reclaim + * more, call this kfunc repeatedly instead of passing a larger @size. + * + * This kfunc is restricted to BPF_PROG_TYPE_SYSCALL to ensure it runs + * in a clean process context. The SYSCALL program can schedule the + * actual reclaim work via bpf_wq or timers, which also execute in + * safe process context (workqueue, task_work). + * + * When reclaim is driven from a bpf_wq, call this kfunc once per + * callback and requeue the same work item for the next batch rather + * than looping inside the callback: a long-running callback stalls + * other work on the shared workqueue, and because lru_lock is held with + * interrupts disabled the resulting contention also delays IPI + * handling. Give each target memcg its own bpf_wq item, so that + * reclaiming one memcg neither serializes behind nor piles up on top of + * another. Deciding whether to submit the next batch is up to the BPF + * program, which can stop at any point, e.g. once the target cgroup is + * dying. + * + * Must not be called with a filesystem lock held: the reclaim path + * may deadlock on it via filesystem shrinkers. + * + * Return: The amount of memory reclaimed, in bytes, or 0 if @size is + * smaller than a page or the task is already in a reclaim context. + */ +__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg, + unsigned long size) +{ + unsigned long nr_reclaimed; + unsigned long nr_pages; + + if (size < PAGE_SIZE || unlikely(bpf_in_reclaim_context())) + return 0; + + nr_pages = min(size / PAGE_SIZE, (unsigned long)MEMCG_CHARGE_BATCH); + + nr_reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages, + GFP_KERNEL, + MEMCG_RECLAIM_MAY_SWAP | + MEMCG_RECLAIM_PROACTIVE, + NULL); + + return nr_reclaimed * PAGE_SIZE; +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(bpf_memcontrol_kfuncs) @@ -171,22 +240,44 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_memory_events) BTF_ID_FLAGS(func, bpf_mem_cgroup_usage) BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state) BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE) - BTF_KFUNCS_END(bpf_memcontrol_kfuncs) +/* + * Proactive reclaim needs a clean process context, so it is restricted + * to BPF_PROG_TYPE_SYSCALL. The bpf_wq and task_work callbacks that a + * SYSCALL program schedules run as the same program type, so they can + * still invoke it; generic sleepable programs (e.g. fentry on reclaim + * paths, inode_rmdir) cannot. + */ +BTF_KFUNCS_START(bpf_memcontrol_reclaim_kfuncs) +BTF_ID_FLAGS(func, bpf_proactive_reclaim, KF_SLEEPABLE) +BTF_KFUNCS_END(bpf_memcontrol_reclaim_kfuncs) + static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = { .owner = THIS_MODULE, .set = &bpf_memcontrol_kfuncs, }; +static const struct btf_kfunc_id_set bpf_memcontrol_reclaim_kfunc_set = { + .owner = THIS_MODULE, + .set = &bpf_memcontrol_reclaim_kfuncs, +}; + static int __init bpf_memcontrol_init(void) { int err; err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &bpf_memcontrol_kfunc_set); - if (err) + if (err) { pr_warn("error while registering bpf memcontrol kfuncs: %d", err); + return err; + } + + err = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, + &bpf_memcontrol_reclaim_kfunc_set); + if (err) + pr_warn("error registering bpf reclaim kfuncs: %d", err); return err; } -- 2.53.0
