On 6/24/26 11:28 PM, Pedro Falcato wrote: > On Mon, Jun 15, 2026 at 08:05:57PM +0900, Harry Yoo (Oracle) wrote: >> Teach kfree_rcu_sheaf() how to handle the !allow_spin case. Try to get >> an empty sheaf from pcs->spare or the barn even when spinning is not >> allowed. Unlike __pcs_replace_full_main(), try harder to allocate >> an empty sheaf because the fallback path will be more expensive than >> kfree_nolock(). >> >> When trylock fails or the kernel observes non-NULL pcs->rcu_free after >> lock acquisition, free the sheaf instead of putting it to the barn. >> This is rare and not worth complicating the code. >> >> Since call_rcu() cannot be called in an unknown context, >> kfree_rcu_sheaf() fails when the rcu sheaf becomes full. >> >> Signed-off-by: Harry Yoo (Oracle) <[email protected]> >> --- >> mm/slab.h | 2 +- >> mm/slab_common.c | 2 +- >> mm/slub.c | 39 ++++++++++++++++++++++++++++++--------- >> 3 files changed, 32 insertions(+), 11 deletions(-) >> >> diff --git a/mm/slab.h b/mm/slab.h >> index 509f330654b8..b1bd33a16544 100644 >> --- a/mm/slab.h >> +++ b/mm/slab.h >> @@ -429,7 +429,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache >> *s) >> return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT)); >> } >> >> -bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj); >> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin); >> void flush_all_rcu_sheaves(void); >> void flush_rcu_sheaves_on_cache(struct kmem_cache *s); >> >> diff --git a/mm/slab_common.c b/mm/slab_common.c >> index b6426d7ceec9..bc1a8ec938d9 100644 >> --- a/mm/slab_common.c >> +++ b/mm/slab_common.c >> @@ -1605,7 +1605,7 @@ static bool kfree_rcu_sheaf(void *obj) >> >> s = slab->slab_cache; >> if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id())) >> - return __kfree_rcu_sheaf(s, obj); >> + return __kfree_rcu_sheaf(s, obj, /* allow_spin = */ true); > > Since this is stacked on top of the slab alloc flags work, could it pass slab > alloc flags instead of allow_spin?
Good point.
Hmm, I thought we'll eventually introduce SLAB_FREE_ flags rather than
reusing SLAB_ALLOC_ flags (and then translate SLAB_FREE_NOLOCK to
SLAB_ALLOC_NOLOCK when allocating sheaves) rather than reusing
SLAB_ALLOC_ flags...
>> @@ -6065,7 +6074,7 @@ static void rcu_free_sheaf(struct rcu_head *head)
>> */
>> static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
>>
>> -bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
>> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)
>> {
>> struct slub_percpu_sheaves *pcs;
>> struct slab_sheaf *rcu_sheaf;
>> @@ -6081,9 +6090,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void
>> *obj)
>> pcs = this_cpu_ptr(s->cpu_sheaves);
>>
>> if (unlikely(!pcs->rcu_free)) {
>> -
>> struct slab_sheaf *empty;
>> struct node_barn *barn;
>> + unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
>
> which would make this logic more natural.
Right.
>> + gfp_t gfp = GFP_NOWAIT;
>>
>> /* Bootstrap or debug cache, fall back */
>> if (unlikely(!cache_has_sheaves(s))) {
>> @@ -6103,7 +6113,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
>> goto fail;
>> }
>>
>> - empty = barn_get_empty_sheaf(barn, true);
>> + empty = barn_get_empty_sheaf(barn, allow_spin);
>>
>> if (empty) {
>> pcs->rcu_free = empty;
>> @@ -6112,20 +6122,25 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void
>> *obj)
>>
>> local_unlock(&s->cpu_sheaves->lock);
>>
>> - empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
>> + if (unlikely(!allow_spin)) {
>> + alloc_flags = SLAB_ALLOC_TRYLOCK;
>> + gfp = 0;
>
> and this as well (alloc_empty_sheaf() could derive gfp from whatever you
> passed it, by simply knowing alloc_flags = TRYLOCK -> gfp = 0
> (or gfp &= ~__GFP_RECLAIM)).
Right.
--
Cheers,
Harry / Hyeonggon
OpenPGP_signature.asc
Description: OpenPGP digital signature
