Hi Paul,
On Sun, Nov 02, 2025 at 01:44:34PM -0800, Paul E. McKenney wrote:
> Some arm64 platforms have slow per-CPU atomic operations, for example,
> the Neoverse V2. This commit therefore moves SRCU-fast from per-CPU
> atomic operations to interrupt-disabled non-read-modify-write-atomic
> atomic_read()/atomic_set() operations. This works because
> SRCU-fast-updown is not invoked from read-side primitives, which
> means that if srcu_read_unlock_fast() NMI handlers. This means that
> srcu_read_lock_fast_updown() and srcu_read_unlock_fast_updown() can
> exclude themselves and each other
>
> This reduces the overhead of calls to srcu_read_lock_fast_updown() and
> srcu_read_unlock_fast_updown() from about 100ns to about 12ns on an ARM
> Neoverse V2. Although this is not excellent compared to about 2ns on x86,
> it sure beats 100ns.
>
> This command was used to measure the overhead:
>
> tools/testing/selftests/rcutorture/bin/kvm.sh --torture refscale --allcpus
> --duration 5 --configs NOPREEMPT --kconfig "CONFIG_NR_CPUS=64
> CONFIG_TASKS_TRACE_RCU=y" --bootargs "refscale.loops=100000
> refscale.guest_os_delay=5 refscale.nreaders=64 refscale.holdoff=30
> torture.disable_onoff_at_boot refscale.scale_type=srcu-fast-updown
> refscale.verbose_batched=8 torture.verbose_sleep_frequency=8
> torture.verbose_sleep_duration=8 refscale.nruns=100" --trust-make
>
> Signed-off-by: Paul E. McKenney <[email protected]>
> Cc: Catalin Marinas <[email protected]>
> Cc: Will Deacon <[email protected]>
> Cc: Mark Rutland <[email protected]>
> Cc: Mathieu Desnoyers <[email protected]>
> Cc: Steven Rostedt <[email protected]>
> Cc: Sebastian Andrzej Siewior <[email protected]>
> Cc: <[email protected]>
> Cc: <[email protected]>
> ---
> include/linux/srcutree.h | 56 ++++++++++++++++++++++++++++++++++++----
> 1 file changed, 51 insertions(+), 5 deletions(-)
[...]
> @@ -327,12 +355,23 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struct
> srcu_ctr __percpu *scp)
> static inline
> struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct
> srcu_struct *ssp)
> {
> - struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp);
> + struct srcu_ctr __percpu *scp;
>
> - if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
> + if (IS_ENABLED(CONFIG_ARM64) &&
> IS_ENABLED(CONFIG_ARM64_USE_LSE_PERCPU_ATOMICS)) {
> + unsigned long flags;
> +
> + local_irq_save(flags);
> + scp = __srcu_read_lock_fast_na(ssp);
> + local_irq_restore(flags); /* Avoids leaking the critical
> section. */
> + return scp;
> + }
Do we still need to pursue this after Catalin's prefetch suggestion for the
per-cpu atomics?
https://lore.kernel.org/r/[email protected]
Although disabling/enabling interrupts on your system seems to be
significantly faster than an atomic instruction, I'm worried that it's
all very SoC-specific and on a mobile part (especially with pseudo-NMI),
the relative costs could easily be the other way around.
Will
