From: Soumya AR <[email protected]> Hi,
This series implements support for integer atomic fetch min/max in GCC, backing the C++26 std::atomic<T>::fetch_min and std::atomic<T>::fetch_max operations. This is v3 of previous patch series posted here, but has had significant rework that pretty much renders the previous patches obsolete. v1: https://gcc.gnu.org/pipermail/gcc-patches/2025-September/693926.html v2: https://gcc.gnu.org/pipermail/gcc-patches/2026-January/706239.html For some context, v1 initially took the approach of implementing everything through builtins, resulting in a significant builtin explosion. v2 improved upon this by only emitting generic builtins and lowering those to an IFN instead. v3 improves further upon this design, makes the infrastructure more generic for future atomic operations, and has had wider testing. ---- This series introduces the following patches: [1] middle-end: Add support for atomic fetch min/max builtins via CAS lowering [2] expand: Expand IFN_ATOMIC_FETCH_MINMAX to target-specific optabs [3] middle-end: Pattern match CAS loops to IFN_ATOMIC_FETCH_MINMAX [4] aarch64: Add backend support for atomic fetch min/max operations [5] experimental: Move atomic-op-lowering pass to gimple-isel The infrastrucure these patches aim to provide looks something like: __atomic_fetch_{min,max} [1] user-written CAS loop | | | pass_cas_to_atomic_op [3] | | +-----------------+-----------------+ v IFN_ATOMIC_FETCH_MINMAX [1] | --- IPA passes --- | +-----------------+-----------------+ | | can expand cannot expand | | backend-specific optab [2] CAS-loop lowering | pass_lower_atomic_ifn [1] (for aarch64) appropriate expansion [4] [5] is an experimental attempt at moving the lowering pass into gimple-isel, in an effort to not create a separate standalone pass (more on that below). ---- [1] middle-end: Add support for atomic fetch min/max builtins via CAS lowering This pass provides support to lower atomic ops to CAS loops in gimple. Earlier versions of this patch were min/max specific. Since we expect future work (FP fetch_add/sub, atomic reductions + -fno-inline-atomics) to require the same plumbing, the pass has been rewritten to be generic. Adding support for a new atomic fetch IFN now involves creating two op-specific callbacks here that modify the CAS loop accordingly for that op. This pass runs after IPA so that the optab query sees the correct backend. That matters under offloading, where we can compile for either host or target, and querying optabs pre-IPA would not give us the right backend. See https://gcc.gnu.org/pipermail/gcc-patches/2025-November/699630.html for the relevant discussion. Lowering this in gimple rather than expand gives us the benefit of subsequent optimization passes now having access to the lowered CAS loop. We also lower when -fno-inline-atomics is true, since fetch min/max do not have corresponding libcalls in libatomic, the best we can do is emit a CAS libcall. ---- [2] expand: Expand IFN_ATOMIC_FETCH_MINMAX to target-specific optabs The IFN reaches expand only when the backend has a target-specific optab. Still, we choose to emit expand_atomic_fetch_op over the _no_fallback variant because having the optab doesn't guarantee that the insn's operand predicates will accept the rtx we hand it. Therefore, since the actual optab emission could fail, expand_atomic_fetch_op should fall through to a CAS loop or libcall. ---- [3] middle-end: Pattern match CAS loops to IFN_ATOMIC_FETCH_MINMAX The motivation for this patch comes from the discussion here: https://gcc.gnu.org/pipermail/gcc-patches/2025-November/699602.html Pattern-matching to an IFN allows us to emit fetch_min/max specific backend optabs (if they exist), in place of the user written CAS loop. That said, I have only found one CAS loop in the wild so far that implements fetch min/max, but from my reasoning, this is an instance we cannot match: static inline void update_earliest (p64_tick_t exp) { p64_tick_t old; do { old = atomic_load_n (&g_timer.earliest, __ATOMIC_RELAXED); if (exp >= old) return; /* <-- note the early exit */ } while (UNLIKELY (!atomic_compare_exchange_n (&g_timer.earliest, &old, exp, __ATOMIC_RELEASE, __ATOMIC_RELAXED))); } (from progress64, src/p64_timer.c, courtesy of Al Grant who guided me to this) This is technically a fetch-min on g_timer.earliest, but the fast-path return turns the loop into load -> maybe-write -> otherwise-noop. AIUI, fetch min/max is an RMW that always writes (even when the stored value would be unchanged), so collapsing this loop into IFN_ATOMIC_FETCH_MINMAX would change the observable behaviour. https://eel.is/c++draft/atomics#types.pointer-6 on fetch_add/sub/min/max: > Effects: Atomically replaces the value pointed to by this with the result of > the computation applied to the value pointed to by this and the given operand. > Memory is affected according to the value of order. These operations are > atomic read-modify-write operations. If anyone is aware of other projects that pattern-match CAS loops into atomic fetch min/max, I would appreciate pointers. Especially because the pass for it is relatively complex. For most matched loops we do run the risk of rewriting CAS -> IFN only to have the gimple-isel lowering pass turn it right back into a CAS loop when the backend doesn't have an optab, simply because of the few backends that do support it. But as mentioned earlier, we accept that round-trip since we cannot query optabs before IPA. ---- [4] aarch64: Add backend support for atomic fetch min/max operations This patch plugs into the infrastructure from [1] and [2] to emit the relevant aarch64 instructions. Three execution paths depending on LSE availability: - LSE inline (-march=armv8.1-a or later): native LSE ldsmin/ldsmax/ ldumin/ldumax. - Outline atomics (default): libgcc dispatches to LSE or LL/SC at runtime. - Inline LL/SC (-mno-outline-atomics on non-LSE targets): LDXR/STXR with conditional select. ---- [5] experimental: Move atomic-op-lowering pass to gimple-isel This is a follow-up to feedback on the v2 thread: https://gcc.gnu.org/pipermail/gcc-patches/2026-March/710694.html Having a dedicated post-IPA pass just to walk the IR and lower a (usually-absent) IFN unconditionally is costly. This patch folds the lowering into gimple-isel, which already walks the IR post-IPA (at -O0 and above) and already lowers other atomic IFNs. gimple-isel runs very late, after most optimizations, which means the CAS loop we synthesise misses pretty much every gimple-level optimisation that could have cleaned it up. Functionally, this does prove that the lowering can fit inside a pre-existing pass, but in practice lowering this late is close to lowering at RTL expand AIUI. ---- We currently don't extend tsan / asan / analyzer to handle IFN_ATOMIC_FETCH_MINMAX. They all already understand the CAS the IFN lowers to, but they don't know the IFN itself. To extend these, we could do the following: - Teach each analyser to understand the IFN directly. Should be doable for asan but not entirely sure how it would work for the analyzer... For tsan, we would have to emit an "unsupported" warning. - Lower to CAS if any of the sanitizer flags are true so we can use the pre-existing CAS support. ---- Note on integrating atomic reductions in the future: Reductions are atomic store-ops (add/sub/and/or/xor/min/max) that do not produce a visible read. For the most trivial fallback, they can be lowered to a fetch_op with the result discarded. Because we already handle min/max separately from add/sub/and/or/xor, adding store-op reductions on top will give us two paths per family: -fno-inline-atomics path: - add/sub/and/or/xor: substitute the IFN with the corresponding __atomic_fetch_OP builtin and let expand_builtin_atomic_fetch_op emit the libcall. - min/max: lower to a CAS loop in gimple; under -fno-inline-atomics its __atomic_compare_exchange_N becomes a libcall. (libatomic doesn't have min/max support.) -finline-atomics path: - add/sub/and/or/xor: directly use expand_atomic_fetch_op, which already handles the appropriate fallbacks. - min/max: if the backend exposes atomic_fetch_{smin,smax,umin,umax}_optab, we can use it in the expander, otherwise lower to a CAS loop in gimple. Flagging this to highlight the complexity that follows once min/max diverges from the rest of the atomic op handling. ---- Bootstrapped and regression tested on aarch64-linux-gnu and x86_64-linux-gnu. Cross-compiled and regression tested for arm-linux-gnueabihf-armv7-a and aarch64-linux-gnu without LSE. I also tested OpenMP offloading to verify that the lowering pass picks the right backend for both host and target. Soumya AR (5): middle-end: Add support for atomic fetch min/max builtins via CAS lowering expand: Expand IFN_ATOMIC_FETCH_MINMAX to target-specific optabs middle-end: Pattern match CAS loops to IFN_ATOMIC_FETCH_MINMAX aarch64: Add backend support for atomic fetch min/max operations experimental: Move atomic-op-lowering pass to gimple-isel gcc/Makefile.in | 1 + gcc/builtins.cc | 82 + gcc/builtins.h | 1 + gcc/c-family/c-common.cc | 63 +- gcc/config/aarch64/aarch64-protos.h | 4 + gcc/config/aarch64/aarch64.cc | 51 + gcc/config/aarch64/atomics.md | 54 +- gcc/config/aarch64/iterators.md | 30 +- gcc/gimple-isel.cc | 365 +++++ gcc/internal-fn.cc | 8 + gcc/internal-fn.def | 1 + gcc/optabs.cc | 38 +- gcc/optabs.def | 24 + gcc/optabs.h | 2 + gcc/passes.def | 1 + gcc/sync-builtins.def | 7 + .../template/builtin-atomic-overloads6.C | 33 +- .../template/builtin-atomic-overloads7.C | 23 +- .../g++.dg/tree-ssa/cas-to-minmax-1.C | 36 + .../g++.dg/tree-ssa/cas-to-minmax-2.C | 35 + .../g++.dg/tree-ssa/cas-to-minmax-3.C | 36 + .../g++.dg/tree-ssa/cas-to-minmax-4.C | 36 + .../gcc.dg/atomic-fetch-minmax-bad-types.c | 17 + .../gcc.dg/atomic-fetch-minmax-ifn.c | 16 + .../gcc.dg/atomic-fetch-minmax-lower.c | 17 + .../gcc.dg/atomic-fetch-minmax-sign-cast.c | 38 + .../atomic-fetch-minmax-type-mismatch.c | 29 + gcc/testsuite/gcc.dg/atomic-op-1.c | 243 ++- gcc/testsuite/gcc.dg/atomic-op-2.c | 243 ++- gcc/testsuite/gcc.dg/atomic-op-3.c | 243 ++- gcc/testsuite/gcc.dg/atomic-op-4.c | 243 ++- gcc/testsuite/gcc.dg/atomic-op-5.c | 245 ++- .../gcc.dg/tree-ssa/cas-to-minmax-1.c | 29 + .../gcc.dg/tree-ssa/cas-to-minmax-2.c | 28 + .../gcc.dg/tree-ssa/cas-to-minmax-3.c | 25 + .../gcc.dg/tree-ssa/cas-to-minmax-4.c | 33 + .../gcc.dg/tree-ssa/cas-to-minmax-5.c | 29 + .../gcc.dg/tree-ssa/cas-to-minmax-6.c | 28 + .../gcc.dg/tree-ssa/cas-to-minmax-7.c | 34 + .../gcc.dg/tree-ssa/cas-to-minmax-run-1.c | 75 + .../gcc.dg/tree-ssa/cas-to-minmax-run-2.c | 55 + .../gcc.dg/tree-ssa/cas-to-minmax-run-3.c | 59 + .../gcc.target/aarch64/atomic-minmax-lse.c | 122 ++ .../gcc.target/aarch64/atomic-minmax-nolse.c | 196 +++ .../gcc.target/aarch64/atomic-minmax.c | 128 ++ .../gcc.target/aarch64/atomic-minmax.x | 172 ++ gcc/tree-cas-to-atomic-op.cc | 1430 +++++++++++++++++ gcc/tree-pass.h | 1 + libgcc/config/aarch64/lse.S | 62 +- libgcc/config/aarch64/t-lse | 3 +- 50 files changed, 4752 insertions(+), 22 deletions(-) create mode 100644 gcc/testsuite/g++.dg/tree-ssa/cas-to-minmax-1.C create mode 100644 gcc/testsuite/g++.dg/tree-ssa/cas-to-minmax-2.C create mode 100644 gcc/testsuite/g++.dg/tree-ssa/cas-to-minmax-3.C create mode 100644 gcc/testsuite/g++.dg/tree-ssa/cas-to-minmax-4.C create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-bad-types.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-ifn.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-sign-cast.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-type-mismatch.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-1.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-2.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-3.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-4.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-5.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-6.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-7.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-run-1.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-run-2.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-run-3.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax-lse.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax-nolse.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax.x create mode 100644 gcc/tree-cas-to-atomic-op.cc -- 2.43.0
