From: Soumya AR <[email protected]> This is an experimental effort at shifting atomic-op-lowering to gimple-isel, instead of having a standalone pass, as mentioned here:
https://gcc.gnu.org/pipermail/gcc-patches/2026-March/710694.html The lowering needs a pass that meets three constraints: - Must run at -O0: this pass needs to run unconditionally every time to lower the IFN created by the frontend, so it cannot be part of an optimisation-level-gated pass. - Must run after IPA. The lowering decision is made based on optab availability of the atomic_fetch_min/max optabs, which means that in the case of offloading, querying optabs earlier could pick the wrong backend. - Needs to have some level of semantic fit, it wouldn't make senes to force fit this in an unrelated pass. >From what I can see, gimple-isel.cc is a good match: it walks the IR post-IPA at -O0 and above, and it already handles other atomic IFNs. We can reuse this walk and queue any atomic-fetch IFN that the backend can't expand directly. After the walk, we run each queued IFN through lower_via_cas_loop. For this, we can drop the cfun->calls_atomic_ifn bit that the standalone pass used to gate itself, along with its tree-inline / move-sese / LTO streamer plumbing. ---- The IR introduced by lower_via_cas_loop adds a new back-edge to the CFG. For this, we set LOOPS_NEED_FIXUP. When cleanup_tree_cfg triggers after this pass it should repair the loop structures (and not break --enable-checking). ---- Functionally this patch is just a proof of concept that the lowering can fit inside an existing IFN walk. But, in practice, lowering this late means that the gimple optimisers do not get a chance to see the loop either way. Moreover, since future atomic ops will also use this lowering infra, maybe having a dedicated pass for it makes more sense. Do let me know if there is a better suited place where we could do this instead. ---- Bootstrapped and regression tested on aarch64-linux-gnu and x86_64-linux-gnu, no regression. Cross-compiled and regression tested under qemu for arm-linux-gnueabihf-armv7-a and aarch64-linux-gnu without LSE. Signed-off-by: Soumya AR <[email protected]> gcc/ChangeLog: * Makefile.in: Drop atomic-ifn-lowering.o. * atomic-ifn-lowering.cc: Delete. * function.h (struct function): Remove the calls_atomic_ifn bit. * gimple-isel.cc (struct atomic_op_lowering): Move from atomic-ifn-lowering.cc. (insert_type_conversion): Likewise. (lower_via_cas_loop): Likewise. (emit_minmax_desired_value): Likewise. (emit_minmax_lhs_value): Likewise. (can_expand_atomic_minmax): Likewise. (lower_minmax_call): Likewise. (pass_gimple_isel::execute): Lower atomic-fetch IFNs. * lto-streamer-in.cc (input_struct_function_base): Stop streaming calls_atomic_ifn. * lto-streamer-out.cc (output_struct_function_base): Likewise. * passes.def: Drop pass_lower_atomic_ifn. * tree-cas-to-atomic-op.cc (cas_to_minmax::emit_replacement): Stop setting cfun->calls_atomic_ifn. * tree-cfg.cc (move_sese_region_to_fn): Stop propagating calls_atomic_ifn. * tree-inline.cc (expand_call_inline): Likewise. * tree-pass.h (make_pass_lower_atomic_ifn): Remove. gcc/c-family/ChangeLog: * c-common.cc (resolve_overloaded_builtin): Don't set cfun->calls_atomic_ifn. gcc/testsuite/ChangeLog: * gcc.dg/atomic-fetch-minmax-lower.c: Re-anchor scans against the isel dump. --- gcc/Makefile.in | 1 - gcc/atomic-ifn-lowering.cc | 480 ------------------ gcc/c-family/c-common.cc | 4 - gcc/function.h | 4 - gcc/gimple-isel.cc | 365 +++++++++++++ gcc/lto-streamer-in.cc | 1 - gcc/lto-streamer-out.cc | 1 - gcc/passes.def | 1 - .../gcc.dg/atomic-fetch-minmax-lower.c | 6 +- gcc/tree-cas-to-atomic-op.cc | 2 - gcc/tree-cfg.cc | 3 - gcc/tree-inline.cc | 1 - gcc/tree-pass.h | 1 - 13 files changed, 368 insertions(+), 502 deletions(-) delete mode 100644 gcc/atomic-ifn-lowering.cc diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 301a8ffdcf1..fba650a8cc6 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -1421,7 +1421,6 @@ OBJS = \ alias.o \ alloc-pool.o \ asm-toplevel.o \ - atomic-ifn-lowering.o \ auto-inc-dec.o \ auto-profile.o \ bb-reorder.o \ diff --git a/gcc/atomic-ifn-lowering.cc b/gcc/atomic-ifn-lowering.cc deleted file mode 100644 index ae689dfdae7..00000000000 --- a/gcc/atomic-ifn-lowering.cc +++ /dev/null @@ -1,480 +0,0 @@ -/* Lower atomic-fetch internal functions to a Compare-and-Swap (CAS) loop. - Copyright The GNU Toolchain Authors. - - This file is part of GCC. - - GCC is free software; you can redistribute it and/or modify it - under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 3, or (at your option) - any later version. - - GCC is distributed in the hope that it will be useful, but - WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - General Public License for more details. - - You should have received a copy of the GNU General Public License - along with GCC; see the file COPYING3. If not see - <http://www.gnu.org/licenses/>. - -This pass lowers atomic-fetch IFNs to a load + CAS loop: - - current_value = atomic_load (ptr); // initial observation - do { - desired_value = op (current_value, value); - // CAS writes desired_value if memory still holds current_value; - // otherwise it refreshes current_value with the actual contents - // and the loop retries. - } while (!atomic_compare_exchange (ptr, ¤t_value, desired_value)); - // LHS gets current_value (if fetch_op) or desired_value (if op_fetch). - -After lowering the CFG looks somewhat like: - - before_bb: - loaded_value = atomic_load (ptr) - goto loop_bb - - loop_bb: - current_value = PHI<loaded_value, observed_value> - desired_value = op (current_value, value) - cas_succeeded = CAS (ptr, &expected_value, desired_value) - observed_value = expected_value // CAS overwrote it on failure - if (cas_succeeded) goto exit_bb else goto loop_bb - - exit_bb: - // LHS = current_value or desired_value, depending on the IFN -*/ - -#include "config.h" -#include "system.h" -#include "coretypes.h" -#include "backend.h" -#include "target.h" -#include "rtl.h" -#include "tree.h" -#include "gimple.h" -#include "gimple-iterator.h" -#include "cfghooks.h" -#include "cfgloop.h" -#include "tree-pass.h" -#include "gimple-pretty-print.h" -#include "memmodel.h" -#include "ssa.h" -#include "optabs.h" -#include "optabs-query.h" -#include "fold-const.h" -#include "builtins.h" -#include "internal-fn.h" - -/* Descriptor for a single atomic-fetch IFN this pass knows how to lower. - CAN_EXPAND_DIRECTLY returns true when the backend can expand the IFN via - a direct optab, in which case the pass leaves the call alone for the RTL - expander. LOWER is called to replace the IFN call with a CAS loop. */ - -struct atomic_op_lowering -{ - internal_fn ifn; - bool (*can_expand_directly) (gcall *); - void (*lower) (gcall *); -}; - -/* Helper to convert between types and insert into the gimple sequence. - Returns the SSA name holding the converted value. */ - -static tree -insert_type_conversion (gimple_stmt_iterator *gsi, tree src_value, - tree dest_type, enum tree_code convert_code = NOP_EXPR, - bool insert_after = true) -{ - if (TREE_TYPE (src_value) == dest_type) - return src_value; - - tree result = make_ssa_name (dest_type); - gassign *convert_stmt = gimple_build_assign (result, convert_code, src_value); - - if (insert_after) - gsi_insert_after (gsi, convert_stmt, GSI_NEW_STMT); - else - gsi_insert_before (gsi, convert_stmt, GSI_SAME_STMT); - - return result; -} - -/* Lower the atomic-fetch IFN call STMT to a load + CAS loop. The op-specific - work is supplied via two callbacks: EMIT_DESIRED_VALUE produces the value - CAS will try to store, and EMIT_LHS_VALUE produces the value to assign to - STMT's LHS (ie. the IFN's LHS). */ - -static void -lower_via_cas_loop (gcall *stmt, - tree (*emit_desired_value) (gimple_stmt_iterator *, tree, - gcall *), - tree (*emit_lhs_value) (tree, tree, gimple_stmt_iterator *)) -{ - gimple_stmt_iterator iter = gsi_for_stmt (stmt); - - tree ptr = gimple_call_arg (stmt, 0); - tree memorder = gimple_call_arg (stmt, 2); - - tree datatype = TREE_TYPE (gimple_call_arg (stmt, 4)); - - /* Build the atomic builtins we need. */ - int size_log2 = exact_log2 (tree_to_uhwi (TYPE_SIZE_UNIT (datatype))); - built_in_function load_builtin_fn - = (built_in_function) ((int) BUILT_IN_ATOMIC_LOAD_N + size_log2 + 1); - tree load_fn = builtin_decl_explicit (load_builtin_fn); - - /* Atomic operations expect unsigned values as arguments. Calculate the - corresponding unsigned type. */ - tree unsigned_type = unsigned_type_for (datatype); - - /* Insert atomic load before the IFN. We will be using the IFN as a split - point later to insert the loop body. Since the load is not part of the - loop body, we insert it before the IFN. */ - tree loaded_value = make_ssa_name (unsigned_type); - gcall *load_call - = gimple_build_call (load_fn, 2, ptr, - build_int_cst (integer_type_node, MEMMODEL_RELAXED)); - gimple_call_set_lhs (load_call, loaded_value); - gsi_insert_before (&iter, load_call, GSI_SAME_STMT); - - /* Save LHS before removing the IFN. Need it later to assign the result. */ - tree lhs = gimple_call_lhs (stmt); - - /* Build the loop body. We will be inserting this after the split point. */ - gimple_seq loop_body = NULL; - gimple_stmt_iterator gsi_loop = gsi_last (loop_body); - - /* Create a SSA name for the loop variable. This will be the PHI node in the - loop body. */ - tree current_value = make_ssa_name (unsigned_type); - - /* Emit the op-specific gimple producing the value CAS will store. */ - tree desired_value = emit_desired_value (&gsi_loop, current_value, stmt); - - /* Emit the CAS as IFN_ATOMIC_COMPARE_EXCHANGE. The IFN takes 'expected' - by value (no addressable local needed) and returns a complex - (REALPART = observed expected, IMAGPART = success flag). On targets - without a direct CAS optab the IFN expander automatically falls back to - the __atomic_compare_exchange_N builtin call, so emitting unconditionally - is safe. Arg 3 is a flag encoding (weak ? 256 : 0) + size_in_bytes; we - never emit weak here, so the flag is just the size. */ - int flag = int_size_in_bytes (datatype); - tree complex_type = build_complex_type (unsigned_type); - tree cas_result = make_ssa_name (complex_type); - gcall *cas_call - = gimple_build_call_internal (IFN_ATOMIC_COMPARE_EXCHANGE, 6, - ptr, /* address to update */ - current_value, /* expected, by value */ - desired_value, /* desired value */ - build_int_cst (integer_type_node, flag), - memorder, /* success order */ - build_int_cst (integer_type_node, - MEMMODEL_RELAXED)); - gimple_call_set_lhs (cas_call, cas_result); - gsi_insert_after (&gsi_loop, cas_call, GSI_NEW_STMT); - - /* Extract the observed expected (REALPART) and the success flag (IMAGPART) - from the complex result. The IFN returns _Complex unsigned_type, so both - REALPART_EXPR and IMAGPART_EXPR produce values of unsigned_type, not bool. - Cast it back to bool for the exit gcond. */ - tree observed_value = make_ssa_name (unsigned_type); - gsi_insert_after (&gsi_loop, - gimple_build_assign (observed_value, REALPART_EXPR, - build1 (REALPART_EXPR, unsigned_type, - cas_result)), - GSI_NEW_STMT); - - tree cas_succeeded_raw = make_ssa_name (unsigned_type); - gsi_insert_after (&gsi_loop, - gimple_build_assign (cas_succeeded_raw, IMAGPART_EXPR, - build1 (IMAGPART_EXPR, unsigned_type, - cas_result)), - GSI_NEW_STMT); - - tree cas_succeeded = make_ssa_name (boolean_type_node); - gsi_insert_after (&gsi_loop, - gimple_build_assign (cas_succeeded, NOP_EXPR, - cas_succeeded_raw), - GSI_NEW_STMT); - - /* Add conditional to end of loop body. The conditional statement currently - has no successors; we add loop_bb and exit_bb as successors below. */ - gcond *loop_cond = gimple_build_cond (NE_EXPR, cas_succeeded, - build_int_cst (boolean_type_node, 0), - NULL_TREE, NULL_TREE); - gsi_insert_after (&gsi_loop, loop_cond, GSI_NEW_STMT); - - /* Get the previous statement in the basic block. We use it to split the - block after the atomic load. Currently the iter points to the IFN; the - bb looks like: - - <previous_code> - | - atomic_load (ptr) - | - IFN_ATOMIC_FETCH_MINMAX <- gsi - | - <following_code> - - We remove the IFN and replace it with the loop body. Split at the IFN, - then split the resultant edge to insert the loop body, resulting in: - - before_bb (contains <previous_code> and atomic_load (ptr)) - | - loop_bb (contains the MIN/MAX operation and the CAS operation) - | - after_bb (contains <following_code>) - */ - - gimple_stmt_iterator prev_gsi = iter; - gsi_prev (&prev_gsi); - - /* Remove the IFN. */ - gsi_remove (&iter, true); - - edge e = split_block (gsi_bb (prev_gsi), gsi_stmt (prev_gsi)); - basic_block exit_bb = e->dest; - basic_block loop_bb = split_edge (e); - - /* Create PHI node for oldval. One edge will come from the predecessor - (loaded_value), and the other will come from the loop back - (observed_value). */ - gphi *loop_phi = create_phi_node (current_value, loop_bb); - add_phi_arg (loop_phi, loaded_value, single_pred_edge (loop_bb), - UNKNOWN_LOCATION); - - /* Insert the loop body into loop_bb. */ - set_bb_seq (loop_bb, loop_body); - - /* Set the basic block for all statements in the loop body to ensure they - are correctly assigned to loop_bb. */ - for (gimple_stmt_iterator si = gsi_start_bb (loop_bb); !gsi_end_p (si); - gsi_next (&si)) - gimple_set_bb (gsi_stmt (si), loop_bb); - - /* loop_bb currently has a fallthrough edge to exit_bb, due to split_edge. - We need to remove it and create two conditional edges. */ - remove_edge (single_succ_edge (loop_bb)); - edge true_edge = make_edge (loop_bb, exit_bb, EDGE_TRUE_VALUE); - edge false_edge = make_edge (loop_bb, loop_bb, EDGE_FALSE_VALUE); - - false_edge->probability = profile_probability::unlikely (); - true_edge->probability = false_edge->probability.invert (); - - /* We've introduced a new back-edge, flag the loop tree for fixup. */ - if (current_loops) - loops_state_set (LOOPS_NEED_FIXUP); - - /* Add the loop-back phi argument. */ - add_phi_arg (loop_phi, observed_value, false_edge, UNKNOWN_LOCATION); - - if (lhs) - { - gimple_stmt_iterator exit_gsi = gsi_after_labels (exit_bb); - tree lhs_value = emit_lhs_value (datatype, current_value, &exit_gsi); - gsi_insert_before (&exit_gsi, gimple_build_assign (lhs, lhs_value), - GSI_SAME_STMT); - } -} - -/* Emit gimple computing MIN/MAX (CURRENT_VALUE, value) at GSI_LOOP for the - IFN_ATOMIC_FETCH_MINMAX call STMT. CURRENT_VALUE is this iteration's - loaded value in the unsigned type. Returns an SSA name (in the same - unsigned type) holding the value CAS will try to store. */ - -static tree -emit_minmax_desired_value (gimple_stmt_iterator *gsi_loop, tree current_value, - gcall *stmt) -{ - tree value = gimple_call_arg (stmt, 1); - tree datatype = TREE_TYPE (gimple_call_arg (stmt, 4)); - /* Arg 3 of the IFN is the tree_code for the operation (MIN_EXPR or - MAX_EXPR), encoded as an integer constant. */ - tree_code cmp_code - = (tree_code) tree_to_uhwi (gimple_call_arg (stmt, 3)); - gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR); - tree unsigned_type = unsigned_type_for (datatype); - - /* __atomic_load_N and __atomic_compare_exchange_N always operate in an - unsigned mode, so CURRENT_VALUE arrives from the load unsigned. But - MIN_EXPR / MAX_EXPR have semantics that depend on their operands' signs, - so for a signed DATATYPE we have to cast CURRENT_VALUE back to DATATYPE - (otherwise MIN/MAX would do an unsigned compare on the raw bits and - give the wrong answer for negative values). - - Avoid using TYPE_UNSIGNED for this: it returns true for pointers even - though unsigned_type_for gives a different TREE_TYPE, so the check - would skip a real cast and leave the MIN/MAX operands mismatched. */ - tree oldval_typed - = useless_type_conversion_p (datatype, TREE_TYPE (current_value)) - ? current_value - : insert_type_conversion (gsi_loop, current_value, datatype); - - /* Compute newval = MIN/MAX (oldval_typed, value) in DATATYPE. */ - tree newval = make_ssa_name (datatype); - gassign *minmax_assign - = gimple_build_assign (newval, cmp_code, oldval_typed, value); - gsi_insert_after (gsi_loop, minmax_assign, GSI_NEW_STMT); - - /* Cast newval back to UNSIGNED_TYPE so CAS can store it. */ - return useless_type_conversion_p (unsigned_type, datatype) - ? newval - : insert_type_conversion (gsi_loop, newval, unsigned_type); -} - -/* Return the SSA name to assign to the IFN's LHS for IFN_ATOMIC_FETCH_MINMAX. - CURRENT_VALUE (the loaded unsigned value on CAS success) should be cast - back to DATATYPE if it differs, with any cast inserted at EXIT_GSI. */ - -static tree -emit_minmax_lhs_value (tree datatype, tree current_value, - gimple_stmt_iterator *exit_gsi) -{ - return useless_type_conversion_p (datatype, TREE_TYPE (current_value)) - ? current_value - : insert_type_conversion (exit_gsi, current_value, datatype, - NOP_EXPR, false); -} - -/* Return true when the backend can expand an IFN_ATOMIC_FETCH_MINMAX call - directly via a target optab. */ - -static bool -can_expand_atomic_minmax (gcall *call) -{ - if (!flag_inline_atomics) - return false; - - tree datatype = TREE_TYPE (gimple_call_arg (call, 4)); - /* Arg 3 is the operation tree_code (MIN_EXPR or MAX_EXPR). */ - tree_code cmp_code - = (tree_code) tree_to_uhwi (gimple_call_arg (call, 3)); - gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR); - bool is_min = (cmp_code == MIN_EXPR); - bool is_signed = !TYPE_UNSIGNED (datatype); - bool unused_result = (gimple_call_lhs (call) == NULL_TREE); - machine_mode mode = TYPE_MODE (datatype); - - optab fetch_op_optab - = is_signed ? (is_min ? atomic_fetch_smin_optab : atomic_fetch_smax_optab) - : (is_min ? atomic_fetch_umin_optab : atomic_fetch_umax_optab); - - if (optab_handler (fetch_op_optab, mode) != CODE_FOR_nothing) - return true; - - /* If the result is unused, the no-result optab is also sufficient. - MIN/MAX does not support op_fetch in the frontend, so no need to check - for those optabs. Even if they were supported by the backend, they - would not be helpful here. */ - if (unused_result) - { - optab no_result_optab - = is_signed ? (is_min ? atomic_smin_optab : atomic_smax_optab) - : (is_min ? atomic_umin_optab : atomic_umax_optab); - if (direct_optab_handler (no_result_optab, mode) != CODE_FOR_nothing) - return true; - } - - return false; -} - -/* Lower an IFN_ATOMIC_FETCH_MINMAX call to a CAS loop. */ - -static void -lower_minmax_call (gcall *call) -{ - lower_via_cas_loop (call, emit_minmax_desired_value, emit_minmax_lhs_value); -} - -/* Registry of atomic-fetch IFNs this pass lowers when the target lacks a - direct optab. */ - -static const atomic_op_lowering atomic_op_table[] = { - {IFN_ATOMIC_FETCH_MINMAX, can_expand_atomic_minmax, lower_minmax_call}, -}; - -static unsigned int -lower_atomic_ifn (void) -{ - basic_block bb; - bool changed = false; - - /* Avoid lowering atomic IFN calls during the BB walk because lowering - can split basic blocks and corrupt the iterator. Collect the calls - and their matching descriptors first, then lower them one by one. */ - auto_vec<std::pair<gcall *, const atomic_op_lowering *> > to_lower; - FOR_EACH_BB_FN (bb, cfun) - for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi); - gsi_next (&gsi)) - { - gcall *call = dyn_cast<gcall *> (gsi_stmt (gsi)); - if (!call) - continue; - for (const atomic_op_lowering &op : atomic_op_table) - if (gimple_call_internal_p (call, op.ifn)) - { - if (op.can_expand_directly && op.can_expand_directly (call)) - break; - to_lower.safe_push ({call, &op}); - break; - } - } - - for (auto &[call, op] : to_lower) - { - if (dump_file) - { - fprintf (dump_file, "Lowering atomic-fetch IFN to CAS loop:\n "); - print_gimple_stmt (dump_file, call, 0, TDF_SLIM); - } - op->lower (call); - changed = true; - } - - return changed ? TODO_update_ssa | TODO_cleanup_cfg : 0; -} - -static bool -gate_lower_atomic_ifn (function *fn) -{ - /* Skip the pass for functions that never constructed an atomic IFN. */ - return fn->calls_atomic_ifn; -} - -namespace { - -const pass_data pass_data_lower_atomic_ifn = { - GIMPLE_PASS, /* type */ - "lower_atomic_ifn", /* name */ - OPTGROUP_NONE, /* optinfo_flags */ - TV_NONE, /* tv_id */ - (PROP_cfg | PROP_ssa), /* properties_required */ - 0, /* properties_provided */ - 0, /* properties_destroyed */ - 0, /* todo_flags_start */ - 0 /* todo_flags_finish */ -}; - -class pass_lower_atomic_ifn : public gimple_opt_pass -{ -public: - pass_lower_atomic_ifn (gcc::context *ctxt) - : gimple_opt_pass (pass_data_lower_atomic_ifn, ctxt) - {} - - /* opt_pass methods: */ - bool gate (function *fn) final override { return gate_lower_atomic_ifn (fn); } - unsigned int execute (function *) final override - { - return lower_atomic_ifn (); - } -}; // class pass_lower_atomic_ifn - -} // namespace - -gimple_opt_pass * -make_pass_lower_atomic_ifn (gcc::context *ctxt) -{ - return new pass_lower_atomic_ifn (ctxt); -} diff --git a/gcc/c-family/c-common.cc b/gcc/c-family/c-common.cc index d2baa8251ef..04e453b819c 100644 --- a/gcc/c-family/c-common.cc +++ b/gcc/c-family/c-common.cc @@ -8901,10 +8901,6 @@ resolve_overloaded_builtin (location_t loc, tree function, if (!flag_non_call_exceptions) TREE_NOTHROW (ret) = 1; - /* Mark the function so pass_lower_atomic_ifn knows to run. */ - if (cfun) - cfun->calls_atomic_ifn = 1; - return ret; } fncode = (enum built_in_function)((int)orig_code + exact_log2 (n) + 1); diff --git a/gcc/function.h b/gcc/function.h index dc030dbd04c..765c71309fa 100644 --- a/gcc/function.h +++ b/gcc/function.h @@ -375,10 +375,6 @@ struct GTY(()) function { /* Nonzero if function being compiled can call __builtin_eh_return. */ unsigned int calls_eh_return : 1; - /* Nonzero if function being compiled contains an atomic-fetch IFN that - pass_lower_atomic_ifn needs to consider. */ - unsigned int calls_atomic_ifn : 1; - /* Nonzero if function being compiled receives nonlocal gotos from nested functions. */ unsigned int has_nonlocal_label : 1; diff --git a/gcc/gimple-isel.cc b/gcc/gimple-isel.cc index b193e27b183..3fe5640ffbf 100644 --- a/gcc/gimple-isel.cc +++ b/gcc/gimple-isel.cc @@ -37,9 +37,14 @@ along with GCC; see the file COPYING3. If not see #include "tree-ssa-dce.h" #include "memmodel.h" #include "optabs.h" +#include "optabs-query.h" #include "gimple-fold.h" #include "internal-fn.h" #include "fold-const.h" +#include "cfghooks.h" +#include "cfgloop.h" +#include "builtins.h" +#include "gimple-pretty-print.h" /* Expand all ARRAY_REF(VIEW_CONVERT_EXPR) gimple assignments into calls to internal function based on vector type of selected expansion. @@ -402,6 +407,334 @@ public: }; // class pass_gimple_isel +/* Descriptor for a single atomic-fetch IFN this pass knows how to lower. + CAN_EXPAND_DIRECTLY returns true when the backend can expand the IFN via + a direct optab, in which case the pass leaves the call alone for the RTL + expander. LOWER is called to replace the IFN call with a CAS loop. */ + +struct atomic_op_lowering +{ + internal_fn ifn; + bool (*can_expand_directly) (gcall *); + void (*lower) (gcall *); +}; + +/* Helper to convert between types and insert into the gimple sequence. + Returns the SSA name holding the converted value. */ + +static tree +insert_type_conversion (gimple_stmt_iterator *gsi, tree src_value, + tree dest_type, enum tree_code convert_code = NOP_EXPR, + bool insert_after = true) +{ + if (TREE_TYPE (src_value) == dest_type) + return src_value; + + tree result = make_ssa_name (dest_type); + gassign *convert_stmt = gimple_build_assign (result, convert_code, src_value); + + if (insert_after) + gsi_insert_after (gsi, convert_stmt, GSI_NEW_STMT); + else + gsi_insert_before (gsi, convert_stmt, GSI_SAME_STMT); + + return result; +} + +/* Lower the atomic-fetch IFN call STMT to a load + CAS loop. The op-specific + work is supplied via two callbacks: EMIT_DESIRED_VALUE produces the value + CAS will try to store, and EMIT_LHS_VALUE produces the value to assign to + STMT's LHS (ie. the IFN's LHS). */ + +static void +lower_via_cas_loop (gcall *stmt, + tree (*emit_desired_value) (gimple_stmt_iterator *, tree, + gcall *), + tree (*emit_lhs_value) (tree, tree, gimple_stmt_iterator *)) +{ + gimple_stmt_iterator iter = gsi_for_stmt (stmt); + + tree ptr = gimple_call_arg (stmt, 0); + tree memorder = gimple_call_arg (stmt, 2); + + tree datatype = TREE_TYPE (gimple_call_arg (stmt, 4)); + + /* Build the atomic builtins we need. */ + int size_log2 = exact_log2 (tree_to_uhwi (TYPE_SIZE_UNIT (datatype))); + built_in_function load_builtin_fn + = (built_in_function) ((int) BUILT_IN_ATOMIC_LOAD_N + size_log2 + 1); + tree load_fn = builtin_decl_explicit (load_builtin_fn); + + /* Atomic operations expect unsigned values as arguments. Calculate the + corresponding unsigned type. */ + tree unsigned_type = unsigned_type_for (datatype); + + /* Insert atomic load before the IFN. We will be using the IFN as a split + point later to insert the loop body. Since the load is not part of the + loop body, we insert it before the IFN. */ + tree loaded_value = make_ssa_name (unsigned_type); + gcall *load_call + = gimple_build_call (load_fn, 2, ptr, + build_int_cst (integer_type_node, MEMMODEL_RELAXED)); + gimple_call_set_lhs (load_call, loaded_value); + gsi_insert_before (&iter, load_call, GSI_SAME_STMT); + + /* Save LHS before removing the IFN. Need it later to assign the result. */ + tree lhs = gimple_call_lhs (stmt); + + /* Build the loop body. We will be inserting this after the split point. */ + gimple_seq loop_body = NULL; + gimple_stmt_iterator gsi_loop = gsi_last (loop_body); + + /* Create a SSA name for the loop variable. This will be the PHI node in the + loop body. */ + tree current_value = make_ssa_name (unsigned_type); + + /* Emit the op-specific gimple producing the value CAS will store. */ + tree desired_value = emit_desired_value (&gsi_loop, current_value, stmt); + + /* Emit the CAS as IFN_ATOMIC_COMPARE_EXCHANGE. The IFN takes 'expected' + by value (no addressable local needed) and returns a complex + (REALPART = observed expected, IMAGPART = success flag). On targets + without a direct CAS optab the IFN expander automatically falls back to + the __atomic_compare_exchange_N builtin call, so emitting unconditionally + is safe. Arg 3 is a flag encoding (weak ? 256 : 0) + size_in_bytes; we + never emit weak here, so the flag is just the size. */ + int flag = int_size_in_bytes (datatype); + tree complex_type = build_complex_type (unsigned_type); + tree cas_result = make_ssa_name (complex_type); + gcall *cas_call + = gimple_build_call_internal (IFN_ATOMIC_COMPARE_EXCHANGE, 6, + ptr, /* address to update */ + current_value, /* expected, by value */ + desired_value, /* desired value */ + build_int_cst (integer_type_node, flag), + memorder, /* success order */ + build_int_cst (integer_type_node, + MEMMODEL_RELAXED)); + gimple_call_set_lhs (cas_call, cas_result); + gsi_insert_after (&gsi_loop, cas_call, GSI_NEW_STMT); + + /* Extract the observed expected (REALPART) and the success flag (IMAGPART) + from the complex result. The IFN returns _Complex unsigned_type, so both + REALPART_EXPR and IMAGPART_EXPR produce values of unsigned_type, not bool. + Cast it back to bool for the exit gcond. */ + tree observed_value = make_ssa_name (unsigned_type); + gsi_insert_after (&gsi_loop, + gimple_build_assign (observed_value, REALPART_EXPR, + build1 (REALPART_EXPR, unsigned_type, + cas_result)), + GSI_NEW_STMT); + + tree cas_succeeded_raw = make_ssa_name (unsigned_type); + gsi_insert_after (&gsi_loop, + gimple_build_assign (cas_succeeded_raw, IMAGPART_EXPR, + build1 (IMAGPART_EXPR, unsigned_type, + cas_result)), + GSI_NEW_STMT); + + tree cas_succeeded = make_ssa_name (boolean_type_node); + gsi_insert_after (&gsi_loop, + gimple_build_assign (cas_succeeded, NOP_EXPR, + cas_succeeded_raw), + GSI_NEW_STMT); + + /* Add conditional to end of loop body. The conditional statement currently + has no successors; we add loop_bb and exit_bb as successors below. */ + gcond *loop_cond = gimple_build_cond (NE_EXPR, cas_succeeded, + build_int_cst (boolean_type_node, 0), + NULL_TREE, NULL_TREE); + gsi_insert_after (&gsi_loop, loop_cond, GSI_NEW_STMT); + + /* Get the previous statement in the basic block. We use it to split the + block after the atomic load. Currently the iter points to the IFN; the + bb looks like: + + <previous_code> + | + atomic_load (ptr) + | + IFN_ATOMIC_FETCH_MINMAX <- gsi + | + <following_code> + + We remove the IFN and replace it with the loop body. Split at the IFN, + then split the resultant edge to insert the loop body, resulting in: + + before_bb (contains <previous_code> and atomic_load (ptr)) + | + loop_bb (contains the MIN/MAX operation and the CAS operation) + | + after_bb (contains <following_code>) + */ + + gimple_stmt_iterator prev_gsi = iter; + gsi_prev (&prev_gsi); + + /* Remove the IFN. */ + gsi_remove (&iter, true); + + edge e = split_block (gsi_bb (prev_gsi), gsi_stmt (prev_gsi)); + basic_block exit_bb = e->dest; + basic_block loop_bb = split_edge (e); + + /* Create PHI node for oldval. One edge will come from the predecessor + (loaded_value), and the other will come from the loop back + (observed_value). */ + gphi *loop_phi = create_phi_node (current_value, loop_bb); + add_phi_arg (loop_phi, loaded_value, single_pred_edge (loop_bb), + UNKNOWN_LOCATION); + + /* Insert the loop body into loop_bb. */ + set_bb_seq (loop_bb, loop_body); + + /* Set the basic block for all statements in the loop body to ensure they + are correctly assigned to loop_bb. */ + for (gimple_stmt_iterator si = gsi_start_bb (loop_bb); !gsi_end_p (si); + gsi_next (&si)) + gimple_set_bb (gsi_stmt (si), loop_bb); + + /* loop_bb currently has a fallthrough edge to exit_bb, due to split_edge. + We need to remove it and create two conditional edges. */ + remove_edge (single_succ_edge (loop_bb)); + edge true_edge = make_edge (loop_bb, exit_bb, EDGE_TRUE_VALUE); + edge false_edge = make_edge (loop_bb, loop_bb, EDGE_FALSE_VALUE); + + false_edge->probability = profile_probability::unlikely (); + true_edge->probability = false_edge->probability.invert (); + + /* We've introduced a new back-edge, flag the loop tree for fixup. */ + if (current_loops) + loops_state_set (LOOPS_NEED_FIXUP); + + /* Add the loop-back phi argument. */ + add_phi_arg (loop_phi, observed_value, false_edge, UNKNOWN_LOCATION); + + if (lhs) + { + gimple_stmt_iterator exit_gsi = gsi_after_labels (exit_bb); + tree lhs_value = emit_lhs_value (datatype, current_value, &exit_gsi); + gsi_insert_before (&exit_gsi, gimple_build_assign (lhs, lhs_value), + GSI_SAME_STMT); + } +} + +/* Emit gimple computing MIN/MAX (CURRENT_VALUE, value) at GSI_LOOP for the + IFN_ATOMIC_FETCH_MINMAX call STMT. CURRENT_VALUE is this iteration's + loaded value in the unsigned type. Returns an SSA name (in the same + unsigned type) holding the value CAS will try to store. */ + +static tree +emit_minmax_desired_value (gimple_stmt_iterator *gsi_loop, tree current_value, + gcall *stmt) +{ + tree value = gimple_call_arg (stmt, 1); + tree datatype = TREE_TYPE (gimple_call_arg (stmt, 4)); + /* Arg 3 of the IFN is the tree_code for the operation (MIN_EXPR or + MAX_EXPR), encoded as an integer constant. */ + tree_code cmp_code + = (tree_code) tree_to_uhwi (gimple_call_arg (stmt, 3)); + gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR); + tree unsigned_type = unsigned_type_for (datatype); + + /* __atomic_load_N and __atomic_compare_exchange_N always operate in an + unsigned mode, so CURRENT_VALUE arrives from the load unsigned. But + MIN_EXPR / MAX_EXPR have semantics that depend on their operands' signs, + so for a signed DATATYPE we have to cast CURRENT_VALUE back to DATATYPE + (otherwise MIN/MAX would do an unsigned compare on the raw bits and + give the wrong answer for negative values). + + Avoid using TYPE_UNSIGNED for this: it returns true for pointers even + though unsigned_type_for gives a different TREE_TYPE, so the check + would skip a real cast and leave the MIN/MAX operands mismatched. */ + tree oldval_typed + = useless_type_conversion_p (datatype, TREE_TYPE (current_value)) + ? current_value + : insert_type_conversion (gsi_loop, current_value, datatype); + + /* Compute newval = MIN/MAX (oldval_typed, value) in DATATYPE. */ + tree newval = make_ssa_name (datatype); + gassign *minmax_assign + = gimple_build_assign (newval, cmp_code, oldval_typed, value); + gsi_insert_after (gsi_loop, minmax_assign, GSI_NEW_STMT); + + /* Cast newval back to UNSIGNED_TYPE so CAS can store it. */ + return useless_type_conversion_p (unsigned_type, datatype) + ? newval + : insert_type_conversion (gsi_loop, newval, unsigned_type); +} + +/* Return the SSA name to assign to the IFN's LHS for IFN_ATOMIC_FETCH_MINMAX. + CURRENT_VALUE (the loaded unsigned value on CAS success) should be cast + back to DATATYPE if it differs, with any cast inserted at EXIT_GSI. */ + +static tree +emit_minmax_lhs_value (tree datatype, tree current_value, + gimple_stmt_iterator *exit_gsi) +{ + return useless_type_conversion_p (datatype, TREE_TYPE (current_value)) + ? current_value + : insert_type_conversion (exit_gsi, current_value, datatype, + NOP_EXPR, false); +} + +/* Return true when the backend can expand an IFN_ATOMIC_FETCH_MINMAX call + directly via a target optab. */ + +static bool +can_expand_atomic_minmax (gcall *call) +{ + if (!flag_inline_atomics) + return false; + + tree datatype = TREE_TYPE (gimple_call_arg (call, 4)); + /* Arg 3 is the operation tree_code (MIN_EXPR or MAX_EXPR). */ + tree_code cmp_code + = (tree_code) tree_to_uhwi (gimple_call_arg (call, 3)); + gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR); + bool is_min = (cmp_code == MIN_EXPR); + bool is_signed = !TYPE_UNSIGNED (datatype); + bool unused_result = (gimple_call_lhs (call) == NULL_TREE); + machine_mode mode = TYPE_MODE (datatype); + + optab fetch_op_optab + = is_signed ? (is_min ? atomic_fetch_smin_optab : atomic_fetch_smax_optab) + : (is_min ? atomic_fetch_umin_optab : atomic_fetch_umax_optab); + + if (optab_handler (fetch_op_optab, mode) != CODE_FOR_nothing) + return true; + + /* If the result is unused, the no-result optab is also sufficient. + MIN/MAX does not support op_fetch in the frontend, so no need to check + for those optabs. Even if they were supported by the backend, they + would not be helpful here. */ + if (unused_result) + { + optab no_result_optab + = is_signed ? (is_min ? atomic_smin_optab : atomic_smax_optab) + : (is_min ? atomic_umin_optab : atomic_umax_optab); + if (direct_optab_handler (no_result_optab, mode) != CODE_FOR_nothing) + return true; + } + + return false; +} + +/* Lower an IFN_ATOMIC_FETCH_MINMAX call to a CAS loop. */ + +static void +lower_minmax_call (gcall *call) +{ + lower_via_cas_loop (call, emit_minmax_desired_value, emit_minmax_lhs_value); +} + +/* Registry of atomic-fetch IFNs this pass lowers when the target lacks a + direct optab. */ + +static const atomic_op_lowering atomic_op_table[] = { + {IFN_ATOMIC_FETCH_MINMAX, can_expand_atomic_minmax, lower_minmax_call}, +}; + /* Convert _1 = __atomic_fetch_or_* (ptr_6, 1, _3); @@ -1353,6 +1686,10 @@ pass_gimple_isel::execute (struct function *fun) basic_block bb; bool cfg_changed = false; + /* Atomic-fetch IFNs whose target lacks a direct optab. Lowering them + splits BBs, so collect during the walk and lower after. */ + auto_vec<std::pair<gcall *, const atomic_op_lowering *> > atomic_to_lower; + FOR_EACH_BB_FN (bb, fun) { for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) @@ -1377,6 +1714,21 @@ pass_gimple_isel::execute (struct function *fun) if (gcall *call = dyn_cast <gcall*>(*gsi)) { + /* Atomic-fetch IFN: if the target can't expand directly, queue + it for CAS-loop lowering after the walk. Otherwise the IFN + survives to RTL expand. */ + bool matched = false; + for (const atomic_op_lowering &op : atomic_op_table) + if (gimple_call_internal_p (call, op.ifn)) + { + if (!op.can_expand_directly (call)) + atomic_to_lower.safe_push ({call, &op}); + matched = true; + break; + } + if (matched) + continue; + gimple_isel_builtin_call (call, &gsi); continue; } @@ -1390,6 +1742,19 @@ pass_gimple_isel::execute (struct function *fun) } } + /* Lower collected atomic IFNs. */ + for (auto &[call, op] : atomic_to_lower) + { + if (dump_file) + { + fprintf (dump_file, + "Lowering atomic-fetch IFN to CAS loop:\n "); + print_gimple_stmt (dump_file, call, 0, TDF_SLIM); + } + op->lower (call); + cfg_changed = true; + } + return cfg_changed ? TODO_cleanup_cfg : 0; } diff --git a/gcc/lto-streamer-in.cc b/gcc/lto-streamer-in.cc index 46e5feb852f..ee2fa2affbc 100644 --- a/gcc/lto-streamer-in.cc +++ b/gcc/lto-streamer-in.cc @@ -1328,7 +1328,6 @@ input_struct_function_base (struct function *fn, class data_in *data_in, fn->has_musttail = bp_unpack_value (&bp, 1); fn->has_unroll = bp_unpack_value (&bp, 1); fn->assume_function = bp_unpack_value (&bp, 1); - fn->calls_atomic_ifn = bp_unpack_value (&bp, 1); fn->va_list_fpr_size = bp_unpack_value (&bp, 8); fn->va_list_gpr_size = bp_unpack_value (&bp, 8); fn->last_clique = bp_unpack_value (&bp, sizeof (short) * 8); diff --git a/gcc/lto-streamer-out.cc b/gcc/lto-streamer-out.cc index 379861f02d6..f2feefc1532 100644 --- a/gcc/lto-streamer-out.cc +++ b/gcc/lto-streamer-out.cc @@ -2318,7 +2318,6 @@ output_struct_function_base (struct output_block *ob, struct function *fn) bp_pack_value (&bp, fn->has_musttail, 1); bp_pack_value (&bp, fn->has_unroll, 1); bp_pack_value (&bp, fn->assume_function, 1); - bp_pack_value (&bp, fn->calls_atomic_ifn, 1); bp_pack_value (&bp, fn->va_list_fpr_size, 8); bp_pack_value (&bp, fn->va_list_gpr_size, 8); bp_pack_value (&bp, fn->last_clique, sizeof (short) * 8); diff --git a/gcc/passes.def b/gcc/passes.def index 38b44495805..612e98e2113 100644 --- a/gcc/passes.def +++ b/gcc/passes.def @@ -207,7 +207,6 @@ along with GCC; see the file COPYING3. If not see NEXT_PASS (pass_omp_target_link); NEXT_PASS (pass_adjust_alignment); NEXT_PASS (pass_harden_control_flow_redundancy); - NEXT_PASS (pass_lower_atomic_ifn); NEXT_PASS (pass_all_optimizations); PUSH_INSERT_PASSES_WITHIN (pass_all_optimizations) NEXT_PASS (pass_remove_cgraph_callee_edges); diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c index 764bf70762d..eb9621ce785 100644 --- a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c +++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -fno-inline-atomics -fdump-tree-lower_atomic_ifn" } */ +/* { dg-options "-O2 -fno-inline-atomics -fdump-tree-isel" } */ int global_val; @@ -13,5 +13,5 @@ int test_max (int x) return __atomic_fetch_max (&global_val, x, __ATOMIC_SEQ_CST); } -/* { dg-final { scan-tree-dump-times "Lowering atomic-fetch IFN to CAS loop" 2 "lower_atomic_ifn" } } */ -/* { dg-final { scan-tree-dump-times "\\.ATOMIC_COMPARE_EXCHANGE" 2 "lower_atomic_ifn" } } */ +/* { dg-final { scan-tree-dump-times "Lowering atomic-fetch IFN to CAS loop" 2 "isel" } } */ +/* { dg-final { scan-tree-dump-times "\\.ATOMIC_COMPARE_EXCHANGE" 2 "isel" } } */ diff --git a/gcc/tree-cas-to-atomic-op.cc b/gcc/tree-cas-to-atomic-op.cc index 653049be033..6bfebf622ba 100644 --- a/gcc/tree-cas-to-atomic-op.cc +++ b/gcc/tree-cas-to-atomic-op.cc @@ -1292,8 +1292,6 @@ cas_to_minmax::emit_replacement () m_success_model, op_code_tree, type_arg); gimple_call_set_nothrow (ifn_call, gimple_call_nothrow_p (m_cas_call)); - cfun->calls_atomic_ifn = 1; - gimple_stmt_iterator cas_gsi = gsi_for_stmt (m_cas_call); bool is_ifn = gimple_call_internal_p (m_cas_call, IFN_ATOMIC_COMPARE_EXCHANGE); diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc index e1c848c8920..8fb80b8121e 100644 --- a/gcc/tree-cfg.cc +++ b/gcc/tree-cfg.cc @@ -8036,9 +8036,6 @@ move_sese_region_to_fn (struct function *dest_cfun, basic_block entry_bb, /* Adjust the maximum clique used. */ dest_cfun->last_clique = saved_cfun->last_clique; - /* Mark if this function contains an atomic IFN. */ - dest_cfun->calls_atomic_ifn |= saved_cfun->calls_atomic_ifn; - loop->aux = NULL; loop0->aux = NULL; /* Loop sizes are no longer correct, fix them up. */ diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc index 27da06d405d..302ab8d6b7c 100644 --- a/gcc/tree-inline.cc +++ b/gcc/tree-inline.cc @@ -5095,7 +5095,6 @@ expand_call_inline (basic_block bb, gimple *stmt, copy_body_data *id, if (src_properties != prop_mask) dst_cfun->curr_properties &= src_properties | ~prop_mask; dst_cfun->calls_eh_return |= id->src_cfun->calls_eh_return; - dst_cfun->calls_atomic_ifn |= id->src_cfun->calls_atomic_ifn; id->dst_node->has_omp_variant_constructs |= id->src_node->has_omp_variant_constructs; diff --git a/gcc/tree-pass.h b/gcc/tree-pass.h index 0fcf4fefc25..972d13d3c4e 100644 --- a/gcc/tree-pass.h +++ b/gcc/tree-pass.h @@ -422,7 +422,6 @@ extern gimple_opt_pass *make_pass_pre (gcc::context *ctxt); extern unsigned int tail_merge_optimize (bool); extern gimple_opt_pass *make_pass_profile (gcc::context *ctxt); extern gimple_opt_pass *make_pass_strip_predict_hints (gcc::context *ctxt); -extern gimple_opt_pass *make_pass_lower_atomic_ifn (gcc::context *ctxt); extern gimple_opt_pass *make_pass_transform_cas_loops_to_atomic_op (gcc::context *ctxt); extern gimple_opt_pass *make_pass_rebuild_frequencies (gcc::context *ctxt); extern gimple_opt_pass *make_pass_lower_complex_O0 (gcc::context *ctxt); -- 2.43.0
