> Am 29.08.2025 um 15:21 schrieb Tamar Christina <[email protected]>:
>
>
>>
>> -----Original Message-----
>> From: Richard Biener <[email protected]>
>> Sent: Friday, August 29, 2025 12:40 PM
>> To: [email protected]
>> Cc: RISC-V CI <[email protected]>; Tamar Christina
>> <[email protected]>
>> Subject: [PATCH 2/4] Separate reduction info and associate it with SLP nodes
>>
>> The following splits out reduction related information from
>> stmt_vec_info, retaining (and duplicating) parts used by scalar
>> cycle analysis. The data is then associated with SLP nodes
>> forming reduction cycles and accessible via info_for_reduction.
>> The data is created at SLP discovery time as we look at it even
>> pre-vectorizable_reduction analysis, but most of the data is
>> only populated by the latter. There is no reduction info with
>> nested cycles that are not part of an outer reduction.
>> In the process this adds cycle info to each SLP tree, notably
>> the reduc-idx and a way to identify the reduction info.
>>
>> Cleanup possibilities will be realized in a later patch of the
>> series. This patch is going to be squashed with the first.
>
> This one breaks the aarch64 build
>
> /opt/buildAgent/work/505bfdd4dad8af3d/gcc/config/aarch64/aarch64.cc: In
> function 'bool aarch64_force_single_cycle(vec_info*, stmt_vec_info)':
> /opt/buildAgent/work/505bfdd4dad8af3d/gcc/config/aarch64/aarch64.cc:17778:41:
> error: invalid conversion from 'vec_info*' to 'loop_vec_info' {aka
> '_loop_vec_info*'} [-fpermissive]
> 17778 | auto reduc_info = info_for_reduction (vinfo, stmt_info);
> | ^~~~~
> | |
> | vec_info*
> /opt/buildAgent/work/505bfdd4dad8af3d/gcc/config/aarch64/aarch64.cc:17778:48:
> error: cannot convert 'stmt_vec_info' {aka '_stmt_vec_info*'} to 'slp_tree'
> {aka '_slp_tree*'}
> 17778 | auto reduc_info = info_for_reduction (vinfo, stmt_info);
> | ^~~~~~~~~
> | |
> | stmt_vec_info {aka
> _stmt_vec_info*}
>
> Since info_for_reduction no longer takes the stmt_info. If I'm reading this
> right for AArch64 we'd need:
>
> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> index 1cdd5a26a83..2831ef38948 100644
> --- a/gcc/config/aarch64/aarch64.cc
> +++ b/gcc/config/aarch64/aarch64.cc
> @@ -17770,12 +17770,13 @@ aarch64_adjust_stmt_cost (vec_info *vinfo,
> vect_cost_for_stmt kind,
>
> with the single accumulator being read and written multiple times. */
> static bool
> -aarch64_force_single_cycle (vec_info *vinfo, stmt_vec_info stmt_info)
> +aarch64_force_single_cycle (vec_info *vinfo, slp_tree node)
> {
> + stmt_vec_info stmt_info = SLP_TREE_REPRESENTATIVE (node);
> if (!STMT_VINFO_REDUC_DEF (stmt_info))
> return false;
>
> - auto reduc_info = info_for_reduction (vinfo, stmt_info);
> + auto reduc_info = info_for_reduction (vinfo, node);
> return VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info);
> }
>
> @@ -17803,7 +17804,7 @@ aarch64_vector_costs::count_ops (unsigned int count,
> vect_cost_for_stmt kind,
> = aarch64_in_loop_reduction_latency (m_vinfo, node,
> stmt_info, m_vec_flags);
> if (m_costing_for_scalar
> - || aarch64_force_single_cycle (m_vinfo, stmt_info))
> + || aarch64_force_single_cycle (m_vinfo, node))
> /* ??? Ideally we'd use a tree to reduce the copies down to 1 vector,
> and then accumulate that, but at the moment the loop-carried
> dependency includes all copies. */
>
> Should I try again with this change?
Yes please.
Richard
> Thanks,
> Tamar
>
>>
>> Bootstrapped and tested on x86_64-unknown-linux-gnu.
>>
>> * tree-vectorizer.h (_slp_tree::cycle_info): New member.
>> (SLP_TREE_REDUC_IDX): Likewise.
>> (vect_reduc_info_s): Move/copy data from ...
>> (_stmt_vec_info): ... here.
>> (_loop_vec_info::redcu_infos): New member.
>> (info_for_reduction): Adjust to take SLP node.
>> (vect_reduc_type): Adjust.
>> (vect_is_reduction): Add overload for SLP node.
>> * tree-vectorizer.cc (vec_info::new_stmt_vec_info):
>> Do not initialize removed members.
>> (vec_info::free_stmt_vec_info): Do not release them.
>> * tree-vect-stmts.cc (vectorizable_condition): Adjust.
>> * tree-vect-slp.cc (_slp_tree::_slp_tree): Initialize
>> cycle info.
>> (vect_build_slp_tree_2): Compute SLP reduc_idx and store
>> it. Create, populate and propagate reduction info.
>> (vect_print_slp_tree): Print cycle info.
>> (vect_analyze_slp_reduc_chain): Set cycle info on the
>> manual added conversion node.
>> (vect_optimize_slp_pass::start_choosing_layouts): Adjust.
>> * tree-vect-loop.cc (_loop_vec_info::~_loop_vec_info):
>> Release reduction infos.
>> (info_for_reduction): Get the reduction info from
>> the vector in the loop_vinfo.
>> (vect_create_epilog_for_reduction): Adjust.
>> (vectorizable_reduction): Likewise.
>> (vect_transform_reduction): Likewise.
>> (vect_transform_cycle_phi): Likewise, deal with nested
>> cycles not part of a double reduction have no reduction info.
>> ---
>> gcc/tree-vect-loop.cc | 82 +++++++---------------
>> gcc/tree-vect-slp.cc | 79 +++++++++++++++++++--
>> gcc/tree-vect-stmts.cc | 3 +-
>> gcc/tree-vectorizer.cc | 12 ----
>> gcc/tree-vectorizer.h | 154 +++++++++++++++++++++++------------------
>> 5 files changed, 190 insertions(+), 140 deletions(-)
>>
>> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
>> index a0e77bdced6..db63891955b 100644
>> --- a/gcc/tree-vect-loop.cc
>> +++ b/gcc/tree-vect-loop.cc
>> @@ -955,6 +955,8 @@ _loop_vec_info::~_loop_vec_info ()
>> delete scan_map;
>> delete scalar_costs;
>> delete vector_costs;
>> + for (auto reduc_info : reduc_infos)
>> + delete reduc_info;
>>
>> /* When we release an epiloge vinfo that we do not intend to use
>> avoid clearing AUX of the main loop which should continue to
>> @@ -5135,46 +5137,12 @@ get_initial_defs_for_reduction (loop_vec_info
>> loop_vinfo,
>> vect_emit_reduction_init_stmts (loop_vinfo, reduc_info, ctor_seq);
>> }
>>
>> -/* For a statement STMT_INFO taking part in a reduction operation return
>> - the stmt_vec_info the meta information is stored on. */
>> -
>> -static vect_reduc_info
>> -info_for_reduction (vec_info *vinfo, stmt_vec_info stmt_info, bool create)
>> -{
>> - stmt_info = vect_orig_stmt (stmt_info);
>> - gcc_assert (STMT_VINFO_REDUC_DEF (stmt_info));
>> - if (!is_a <gphi *> (stmt_info->stmt)
>> - || !VECTORIZABLE_CYCLE_DEF (STMT_VINFO_DEF_TYPE (stmt_info)))
>> - stmt_info = STMT_VINFO_REDUC_DEF (stmt_info);
>> - gphi *phi = as_a <gphi *> (stmt_info->stmt);
>> - if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_double_reduction_def)
>> - {
>> - if (gimple_phi_num_args (phi) == 1)
>> - stmt_info = STMT_VINFO_REDUC_DEF (stmt_info);
>> - }
>> - else if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_nested_cycle)
>> - {
>> - stmt_vec_info info = vinfo->lookup_def (vect_phi_initial_value (phi));
>> - if (info && STMT_VINFO_DEF_TYPE (info) == vect_double_reduction_def)
>> - stmt_info = info;
>> - }
>> - if (create)
>> - stmt_info->is_reduc_info = true;
>> - else
>> - gcc_assert (stmt_info->is_reduc_info);
>> - return vect_reduc_info (stmt_info);
>> -}
>> -
>> -vect_reduc_info
>> -info_for_reduction (vec_info *vinfo, stmt_vec_info stmt_info)
>> -{
>> - return info_for_reduction (vinfo, stmt_info, false);
>> -}
>> -
>> vect_reduc_info
>> -create_info_for_reduction (vec_info *vinfo, stmt_vec_info stmt_info)
>> +info_for_reduction (loop_vec_info loop_vinfo, slp_tree node)
>> {
>> - return info_for_reduction (vinfo, stmt_info, true);
>> + if (node->cycle_info.id == -1)
>> + return NULL;
>> + return loop_vinfo->reduc_infos[node->cycle_info.id];
>> }
>>
>> /* See if LOOP_VINFO is an epilogue loop whose main loop had a reduction that
>> @@ -5434,7 +5402,7 @@ vect_create_epilog_for_reduction (loop_vec_info
>> loop_vinfo,
>> slp_instance slp_node_instance,
>> edge loop_exit)
>> {
>> - vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, stmt_info);
>> + vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, slp_node);
>> /* For double reductions we need to get at the inner loop reduction
>> stmt which has the meta info attached. Our stmt_info is that of the
>> loop-closed PHI of the inner loop which we remember as
>> @@ -6920,7 +6888,7 @@ vectorizable_lane_reducing (loop_vec_info loop_vinfo,
>> stmt_vec_info stmt_info,
>> || STMT_VINFO_REDUC_IDX (stmt_info) < 0)
>> return false;
>>
>> - vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, stmt_info);
>> + vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, slp_node);
>>
>> /* Lane-reducing pattern inside any inner loop of LOOP_VINFO is not
>> recoginized. */
>> @@ -7083,8 +7051,8 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> && STMT_VINFO_DEF_TYPE (stmt_info) != vect_nested_cycle)
>> return false;
>>
>> - /* The stmt we store reduction analysis meta on. */
>> - vect_reduc_info reduc_info = create_info_for_reduction (loop_vinfo,
>> stmt_info);
>> + /* The reduction meta. */
>> + vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, slp_node);
>>
>> if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_nested_cycle)
>> {
>> @@ -7160,7 +7128,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> slp_tree vdef_slp = slp_node_instance->root;
>> /* For double-reductions we start SLP analysis at the inner loop LC PHI
>> which is the def of the outer loop live stmt. */
>> - if (STMT_VINFO_DEF_TYPE (reduc_info.fixme ()) ==
>> vect_double_reduction_def)
>> + if (VECT_REDUC_INFO_DEF_TYPE (reduc_info) == vect_double_reduction_def)
>> vdef_slp = SLP_TREE_CHILDREN (vdef_slp)[0];
>> while (reduc_def != PHI_RESULT (reduc_def_phi))
>> {
>> @@ -7399,8 +7367,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> }
>> }
>>
>> - enum vect_reduction_type reduction_type = STMT_VINFO_REDUC_TYPE
>> (phi_info);
>> - VECT_REDUC_INFO_TYPE (reduc_info) = reduction_type;
>> + enum vect_reduction_type reduction_type = VECT_REDUC_INFO_TYPE
>> (reduc_info);
>> /* If we have a condition reduction, see if we can simplify it further. */
>> if (reduction_type == COND_REDUCTION)
>> {
>> @@ -7514,7 +7481,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>>
>> if (nested_cycle)
>> {
>> - gcc_assert (STMT_VINFO_DEF_TYPE (reduc_info.fixme ())
>> + gcc_assert (VECT_REDUC_INFO_DEF_TYPE (reduc_info)
>> == vect_double_reduction_def);
>> double_reduc = true;
>> }
>> @@ -7554,7 +7521,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo,
>> (and also the same tree-code) when generating the epilog code and
>> when generating the code inside the loop. */
>>
>> - code_helper orig_code = STMT_VINFO_REDUC_CODE (phi_info);
>> + code_helper orig_code = VECT_REDUC_INFO_CODE (reduc_info);
>>
>> /* If conversion might have created a conditional operation like
>> IFN_COND_ADD already. Use the internal code for the following checks.
>> */
>> @@ -8031,12 +7998,13 @@ vect_transform_reduction (loop_vec_info
>> loop_vinfo,
>> class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
>> unsigned vec_num;
>>
>> - vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, stmt_info);
>> + vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, slp_node);
>>
>> if (nested_in_vect_loop_p (loop, stmt_info))
>> {
>> loop = loop->inner;
>> - gcc_assert (STMT_VINFO_DEF_TYPE (reduc_info.fixme ()) ==
>> vect_double_reduction_def);
>> + gcc_assert (VECT_REDUC_INFO_DEF_TYPE (reduc_info)
>> + == vect_double_reduction_def);
>> }
>>
>> gimple_match_op op;
>> @@ -8382,10 +8350,10 @@ vect_transform_cycle_phi (loop_vec_info
>> loop_vinfo,
>> nested_cycle = true;
>> }
>>
>> - vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, stmt_info);
>> -
>> - if (VECT_REDUC_INFO_TYPE (reduc_info) == EXTRACT_LAST_REDUCTION
>> - || VECT_REDUC_INFO_TYPE (reduc_info) == FOLD_LEFT_REDUCTION)
>> + vect_reduc_info reduc_info = info_for_reduction (loop_vinfo, slp_node);
>> + if (reduc_info
>> + && (VECT_REDUC_INFO_TYPE (reduc_info) == EXTRACT_LAST_REDUCTION
>> + || VECT_REDUC_INFO_TYPE (reduc_info) == FOLD_LEFT_REDUCTION))
>> /* Leave the scalar phi in place. */
>> return true;
>>
>> @@ -8393,7 +8361,7 @@ vect_transform_cycle_phi (loop_vec_info loop_vinfo,
>>
>> /* Check whether we should use a single PHI node and accumulate
>> vectors to one before the backedge. */
>> - if (VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info))
>> + if (reduc_info && VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info))
>> vec_num = 1;
>>
>> /* Create the destination vector */
>> @@ -8408,7 +8376,8 @@ vect_transform_cycle_phi (loop_vec_info loop_vinfo,
>> /* Optimize: if initial_def is for REDUC_MAX smaller than the base
>> and we can't use zero for induc_val, use initial_def. Similarly
>> for REDUC_MIN and initial_def larger than the base. */
>> - if (VECT_REDUC_INFO_TYPE (reduc_info) ==
>> INTEGER_INDUC_COND_REDUCTION)
>> + if (reduc_info
>> + && VECT_REDUC_INFO_TYPE (reduc_info) ==
>> INTEGER_INDUC_COND_REDUCTION)
>> {
>> gcc_assert (SLP_TREE_LANES (slp_node) == 1);
>> tree initial_def = vect_phi_initial_value (phi);
>> @@ -8486,6 +8455,7 @@ vect_transform_cycle_phi (loop_vec_info loop_vinfo,
>> vec_initial_defs.quick_push (vec_initial_def);
>> }
>>
>> + if (reduc_info)
>> if (auto *accumulator = VECT_REDUC_INFO_REUSED_ACCUMULATOR
>> (reduc_info))
>> {
>> tree def = accumulator->reduc_input;
>> @@ -10280,7 +10250,7 @@ vectorizable_live_operation (vec_info *vinfo,
>> stmt_vec_info stmt_info,
>> if (SLP_INSTANCE_KIND (slp_node_instance) == slp_inst_kind_reduc_group
>> && slp_index != 0)
>> return true;
>> - vect_reduc_info reduc_info = info_for_reduction (loop_vinfo,
>> stmt_info);
>> + vect_reduc_info reduc_info = info_for_reduction (loop_vinfo,
>> slp_node);
>> if (VECT_REDUC_INFO_TYPE (reduc_info) == FOLD_LEFT_REDUCTION
>> || VECT_REDUC_INFO_TYPE (reduc_info) ==
>> EXTRACT_LAST_REDUCTION)
>> return true;
>> diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
>> index 733f4ece724..5236eac5a42 100644
>> --- a/gcc/tree-vect-slp.cc
>> +++ b/gcc/tree-vect-slp.cc
>> @@ -126,6 +126,8 @@ _slp_tree::_slp_tree ()
>> this->avoid_stlf_fail = false;
>> SLP_TREE_VECTYPE (this) = NULL_TREE;
>> SLP_TREE_REPRESENTATIVE (this) = NULL;
>> + this->cycle_info.id = -1;
>> + this->cycle_info.reduc_idx = -1;
>> SLP_TREE_REF_COUNT (this) = 1;
>> this->failed = NULL;
>> this->max_nunits = 1;
>> @@ -2735,6 +2737,7 @@ out:
>>
>> stmt_info = stmts[0];
>>
>> + int reduc_idx = -1;
>> int gs_scale = 0;
>> tree gs_base = NULL_TREE;
>>
>> @@ -2826,6 +2829,33 @@ out:
>> continue;
>> }
>>
>> + /* See which SLP operand a reduction chain continues on. We want
>> + to chain even PHIs but not backedges. */
>> + if (VECTORIZABLE_CYCLE_DEF (oprnd_info->first_dt)
>> + || STMT_VINFO_REDUC_IDX (oprnd_info->def_stmts[0]) != -1)
>> + {
>> + if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_nested_cycle)
>> + {
>> + if (oprnd_info->first_dt == vect_double_reduction_def)
>> + reduc_idx = i;
>> + }
>> + else if (is_a <gphi *> (stmt_info->stmt)
>> + && gimple_phi_num_args
>> + (as_a <gphi *> (stmt_info->stmt)) != 1)
>> + ;
>> + else if (STMT_VINFO_REDUC_IDX (stmt_info) == -1
>> + && STMT_VINFO_DEF_TYPE (stmt_info) !=
>> vect_double_reduction_def)
>> + ;
>> + else if (reduc_idx == -1)
>> + reduc_idx = i;
>> + else
>> + /* For .COND_* reduction operations the else value can be the
>> + same as one of the operation operands. The other def
>> + stmts have been moved, so we can't check easily. Check
>> + it's a call at least. */
>> + gcc_assert (is_a <gcall *> (stmt_info->stmt));
>> + }
>> +
>> /* When we have a masked load with uniform mask discover this
>> as a single-lane mask with a splat permute. This way we can
>> recognize this as a masked load-lane by stripping the splat. */
>> @@ -3157,6 +3187,41 @@ fail:
>> SLP_TREE_CHILDREN (node).splice (children);
>> SLP_TREE_GS_SCALE (node) = gs_scale;
>> SLP_TREE_GS_BASE (node) = gs_base;
>> + if (reduc_idx != -1)
>> + {
>> + gcc_assert (STMT_VINFO_REDUC_IDX (stmt_info) != -1
>> + || STMT_VINFO_DEF_TYPE (stmt_info) == vect_nested_cycle
>> + || STMT_VINFO_DEF_TYPE (stmt_info) ==
>> vect_double_reduction_def);
>> + SLP_TREE_REDUC_IDX (node) = reduc_idx;
>> + node->cycle_info.id = SLP_TREE_CHILDREN
>> (node)[reduc_idx]->cycle_info.id;
>> + }
>> + /* When reaching the reduction PHI, create a vect_reduc_info. */
>> + else if ((STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def
>> + || STMT_VINFO_DEF_TYPE (stmt_info) == vect_double_reduction_def)
>> + && is_a <gphi *> (STMT_VINFO_STMT (stmt_info)))
>> + {
>> + loop_vec_info loop_vinfo = as_a <loop_vec_info> (vinfo);
>> + gcc_assert (STMT_VINFO_REDUC_IDX (stmt_info) == -1);
>> + node->cycle_info.id = loop_vinfo->reduc_infos.length ();
>> + vect_reduc_info reduc_info = new vect_reduc_info_s ();
>> + loop_vinfo->reduc_infos.safe_push (reduc_info);
>> + stmt_vec_info reduc_phi = stmt_info;
>> + /* ??? For double reductions vect_is_simple_reduction stores the
>> + reduction type and code on the inner loop header PHI. */
>> + if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_double_reduction_def)
>> + {
>> + use_operand_p use_p;
>> + gimple *use_stmt;
>> + bool res = single_imm_use (gimple_phi_result (stmt_info->stmt),
>> + &use_p, &use_stmt);
>> + gcc_assert (res);
>> + reduc_phi = loop_vinfo->lookup_stmt (use_stmt);
>> + }
>> + VECT_REDUC_INFO_DEF_TYPE (reduc_info) = STMT_VINFO_DEF_TYPE
>> (stmt_info);
>> + VECT_REDUC_INFO_TYPE (reduc_info) = STMT_VINFO_REDUC_TYPE
>> (reduc_phi);
>> + VECT_REDUC_INFO_CODE (reduc_info) = STMT_VINFO_REDUC_CODE
>> (reduc_phi);
>> + VECT_REDUC_INFO_FN (reduc_info) = IFN_LAST;
>> + }
>> return node;
>> }
>>
>> @@ -3185,8 +3250,12 @@ vect_print_slp_tree (dump_flags_t dump_kind,
>> dump_location_t loc,
>> SLP_TREE_REF_COUNT (node));
>> if (SLP_TREE_VECTYPE (node))
>> dump_printf (metadata, " %T", SLP_TREE_VECTYPE (node));
>> - dump_printf (metadata, "%s\n",
>> + dump_printf (metadata, "%s",
>> node->avoid_stlf_fail ? " (avoid-stlf-fail)" : "");
>> + if (node->cycle_info.id != -1 || node->cycle_info.reduc_idx != -1)
>> + dump_printf (metadata, " cycle %d, link %d", node->cycle_info.id,
>> + node->cycle_info.reduc_idx);
>> + dump_printf (metadata, "\n");
>> if (SLP_TREE_DEF_TYPE (node) == vect_internal_def)
>> {
>> if (SLP_TREE_PERMUTE_P (node))
>> @@ -4241,6 +4310,8 @@ vect_analyze_slp_reduc_chain (vec_info *vinfo,
>> TREE_TYPE
>> (gimple_assign_lhs (scalar_def)),
>> group_size);
>> + SLP_TREE_REDUC_IDX (conv) = 0;
>> + conv->cycle_info.id = node->cycle_info.id;
>> SLP_TREE_CHILDREN (conv).quick_push (node);
>> SLP_INSTANCE_TREE (new_instance) = conv;
>> /* We also have to fake this conversion stmt as SLP reduction
>> @@ -6719,11 +6790,9 @@ vect_optimize_slp_pass::start_choosing_layouts ()
>> {
>> stmt_vec_info stmt_info
>> = SLP_TREE_REPRESENTATIVE (SLP_INSTANCE_TREE (instance));
>> - /* ??? vectorizable_reduction did not run yet, scalar cycle
>> - detection sets reduc_code. Either that or SLP discovery
>> - should create a reduction info. */
>> vect_reduc_info reduc_info
>> - = create_info_for_reduction (m_vinfo, stmt_info);
>> + = info_for_reduction (as_a <loop_vec_info> (m_vinfo),
>> + SLP_INSTANCE_TREE (instance));
>> if (needs_fold_left_reduction_p (TREE_TYPE
>> (gimple_get_lhs (stmt_info->stmt)),
>> VECT_REDUC_INFO_CODE (reduc_info)))
>> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
>> index 2248b361e1c..77a03ed4a7b 100644
>> --- a/gcc/tree-vect-stmts.cc
>> +++ b/gcc/tree-vect-stmts.cc
>> @@ -11572,7 +11572,8 @@ vectorizable_condition (vec_info *vinfo,
>> every stmt, use the conservative default setting then. */
>> if (STMT_VINFO_REDUC_DEF (vect_orig_stmt (stmt_info)))
>> {
>> - vect_reduc_info reduc_info = info_for_reduction (vinfo, stmt_info);
>> + vect_reduc_info reduc_info
>> + = info_for_reduction (loop_vinfo, slp_node);
>> reduction_type = VECT_REDUC_INFO_TYPE (reduc_info);
>> nested_cycle_p = nested_in_vect_loop_p (LOOP_VINFO_LOOP
>> (loop_vinfo),
>> stmt_info);
>> diff --git a/gcc/tree-vectorizer.cc b/gcc/tree-vectorizer.cc
>> index 726383f487c..d7dc30bbeac 100644
>> --- a/gcc/tree-vectorizer.cc
>> +++ b/gcc/tree-vectorizer.cc
>> @@ -724,16 +724,6 @@ vec_info::new_stmt_vec_info (gimple *stmt)
>> STMT_VINFO_SLP_VECT_ONLY (res) = false;
>> STMT_VINFO_SLP_VECT_ONLY_PATTERN (res) = false;
>>
>> - /* To be moved. */
>> - res->reduc_epilogue_adjustment = NULL_TREE;
>> - res->force_single_cycle = false;
>> - res->reduc_fn = IFN_LAST;
>> - res->reduc_initial_values = vNULL;
>> - res->reduc_scalar_results = vNULL;
>> - res->reduc_vectype = NULL_TREE;
>> - res->induc_cond_initial_val = NULL_TREE;
>> - res->reused_accumulator = NULL;
>> -
>> if (is_a <loop_vec_info> (this)
>> && gimple_code (stmt) == GIMPLE_PHI
>> && is_loop_header_bb_p (gimple_bb (stmt)))
>> @@ -794,8 +784,6 @@ vec_info::free_stmt_vec_info (stmt_vec_info stmt_info)
>> release_ssa_name (lhs);
>> }
>>
>> - stmt_info->reduc_initial_values.release ();
>> - stmt_info->reduc_scalar_results.release ();
>> free (stmt_info);
>> }
>>
>> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
>> index 90d862f2987..260cb2ddd3e 100644
>> --- a/gcc/tree-vectorizer.h
>> +++ b/gcc/tree-vectorizer.h
>> @@ -310,6 +310,13 @@ struct _slp_tree {
>> code generation. */
>> stmt_vec_info representative;
>>
>> + struct {
>> + /* SLP cycle the node resides in, or -1. */
>> + int id;
>> + /* The SLP operand index with the edge on the SLP cycle, or -1. */
>> + int reduc_idx;
>> + } cycle_info;
>> +
>> /* Load permutation relative to the stores, NULL if there is no
>> permutation. */
>> load_permutation_t load_permutation;
>> @@ -446,6 +453,7 @@ public:
>> #define SLP_TREE_TYPE(S) (S)->type
>> #define SLP_TREE_GS_SCALE(S) (S)->gs_scale
>> #define SLP_TREE_GS_BASE(S) (S)->gs_base
>> +#define SLP_TREE_REDUC_IDX(S) (S)->cycle_info.reduc_idx
>> #define SLP_TREE_PERMUTE_P(S) ((S)->code ==
>> VEC_PERM_EXPR)
>>
>> inline vect_memory_access_type
>> @@ -816,26 +824,70 @@ typedef auto_vec<std::pair<data_reference*, tree> >
>> drs_init_vec;
>>
>> /* Abstraction around info on reductions which is still in stmt_vec_info
>> but will be duplicated or moved elsewhere. */
>> -class vect_reduc_info
>> +class vect_reduc_info_s
>> {
>> public:
>> - explicit vect_reduc_info (stmt_vec_info s) : i (s) {}
>> - stmt_vec_info fixme () const { return i; }
>> -private:
>> - stmt_vec_info i;
>> + /* The def type of the main reduction PHI, vect_reduction_def or
>> + vect_double_reduction_def. */
>> + enum vect_def_type def_type;
>> +
>> + /* The reduction type as detected by
>> + vect_is_simple_reduction and vectorizable_reduction. */
>> + enum vect_reduction_type reduc_type;
>> +
>> + /* The original scalar reduction code, to be used in the epilogue. */
>> + code_helper reduc_code;
>> +
>> + /* A vector internal function we should use in the epilogue. */
>> + internal_fn reduc_fn;
>> +
>> + /* For loop reduction with multiple vectorized results (ncopies > 1), a
>> + lane-reducing operation participating in it may not use all of those
>> + results, this field specifies result index starting from which any
>> + following land-reducing operation would be assigned to. */
>> + unsigned int reduc_result_pos;
>> +
>> + /* Whether we force a single cycle PHI during reduction vectorization. */
>> + bool force_single_cycle;
>> +
>> + /* The vector type for performing the actual reduction operation. */
>> + tree reduc_vectype;
>> +
>> + /* For INTEGER_INDUC_COND_REDUCTION, the initial value to be used. */
>> + tree induc_cond_initial_val;
>> +
>> + /* If not NULL the value to be added to compute final reduction value. */
>> + tree reduc_epilogue_adjustment;
>> +
>> + /* If non-null, the reduction is being performed by an epilogue loop
>> + and we have decided to reuse this accumulator from the main loop. */
>> + struct vect_reusable_accumulator *reused_accumulator;
>> +
>> + /* If the vector code is performing N scalar reductions in parallel,
>> + this variable gives the initial scalar values of those N reductions.
>> */
>> + auto_vec<tree> reduc_initial_values;
>> +
>> + /* If the vector code is performing N scalar reductions in parallel, this
>> + variable gives the vectorized code's final (scalar) result for each of
>> + those N reductions. In other words, REDUC_SCALAR_RESULTS[I] replaces
>> + the original scalar code's loop-closed SSA PHI for reduction number I.
>> */
>> + auto_vec<tree> reduc_scalar_results;
>> };
>>
>> -#define VECT_REDUC_INFO_TYPE(I) ((I).fixme ()->reduc_type)
>> -#define VECT_REDUC_INFO_CODE(I) ((I).fixme ()->reduc_code)
>> -#define VECT_REDUC_INFO_FN(I) ((I).fixme ()->reduc_fn)
>> -#define VECT_REDUC_INFO_SCALAR_RESULTS(I) ((I).fixme ()-
>>> reduc_scalar_results)
>> -#define VECT_REDUC_INFO_INITIAL_VALUES(I) ((I).fixme
>> ()->reduc_initial_values)
>> -#define VECT_REDUC_INFO_REUSED_ACCUMULATOR(I) ((I).fixme ()-
>>> reused_accumulator)
>> -#define VECT_REDUC_INFO_INDUC_COND_INITIAL_VAL(I) ((I).fixme ()-
>>> induc_cond_initial_val)
>> -#define VECT_REDUC_INFO_EPILOGUE_ADJUSTMENT(I) ((I).fixme ()-
>>> reduc_epilogue_adjustment)
>> -#define VECT_REDUC_INFO_VECTYPE(I) ((I).fixme ()->reduc_vectype)
>> -#define VECT_REDUC_INFO_FORCE_SINGLE_CYCLE(I) ((I).fixme ()-
>>> force_single_cycle)
>> -#define VECT_REDUC_INFO_RESULT_POS(I) ((I).fixme ()->reduc_result_pos)
>> +typedef class vect_reduc_info_s *vect_reduc_info;
>> +
>> +#define VECT_REDUC_INFO_DEF_TYPE(I) ((I)->def_type)
>> +#define VECT_REDUC_INFO_TYPE(I) ((I)->reduc_type)
>> +#define VECT_REDUC_INFO_CODE(I) ((I)->reduc_code)
>> +#define VECT_REDUC_INFO_FN(I) ((I)->reduc_fn)
>> +#define VECT_REDUC_INFO_SCALAR_RESULTS(I) ((I)->reduc_scalar_results)
>> +#define VECT_REDUC_INFO_INITIAL_VALUES(I) ((I)->reduc_initial_values)
>> +#define VECT_REDUC_INFO_REUSED_ACCUMULATOR(I) ((I)-
>>> reused_accumulator)
>> +#define VECT_REDUC_INFO_INDUC_COND_INITIAL_VAL(I) ((I)-
>>> induc_cond_initial_val)
>> +#define VECT_REDUC_INFO_EPILOGUE_ADJUSTMENT(I) ((I)-
>>> reduc_epilogue_adjustment)
>> +#define VECT_REDUC_INFO_VECTYPE(I) ((I)->reduc_vectype)
>> +#define VECT_REDUC_INFO_FORCE_SINGLE_CYCLE(I) ((I)->force_single_cycle)
>> +#define VECT_REDUC_INFO_RESULT_POS(I) ((I)->reduc_result_pos)
>>
>> /* Information about a reduction accumulator from the main loop that could
>> conceivably be reused as the input to a reduction in an epilogue loop. */
>> @@ -902,6 +954,10 @@ public:
>> the main loop, this edge is the one that skips the epilogue. */
>> edge skip_this_loop_edge;
>>
>> + /* Reduction descriptors of this loop. Referenced to from SLP nodes
>> + by index. */
>> + auto_vec<vect_reduc_info> reduc_infos;
>> +
>> /* The vectorized form of a standard reduction replaces the original
>> scalar code's final result (a loop-closed SSA PHI) with the result
>> of a vector-to-scalar reduction operation. After vectorization,
>> @@ -1517,62 +1573,22 @@ public:
>> /* For both loads and stores. */
>> unsigned simd_lane_access_p : 3;
>>
>> - /* For INTEGER_INDUC_COND_REDUCTION, the initial value to be used. */
>> - tree induc_cond_initial_val;
>> -
>> - /* If not NULL the value to be added to compute final reduction value. */
>> - tree reduc_epilogue_adjustment;
>> -
>> /* On a reduction PHI the reduction type as detected by
>> - vect_is_simple_reduction and vectorizable_reduction. */
>> + vect_is_simple_reduction. */
>> enum vect_reduction_type reduc_type;
>>
>> - /* The original reduction code, to be used in the epilogue. */
>> + /* On a reduction PHI, the original reduction code as detected by
>> + vect_is_simple_reduction. */
>> code_helper reduc_code;
>> - /* An internal function we should use in the epilogue. */
>> - internal_fn reduc_fn;
>>
>> - /* On a stmt participating in the reduction the index of the operand
>> + /* On a stmt participating in a reduction the index of the operand
>> on the reduction SSA cycle. */
>> int reduc_idx;
>>
>> - /* On a reduction PHI the def returned by vect_force_simple_reduction.
>> - On the def returned by vect_force_simple_reduction the
>> - corresponding PHI. */
>> + /* On a reduction PHI the def returned by vect_is_simple_reduction.
>> + On the def returned by vect_is_simple_reduction the corresponding PHI.
>> */
>> stmt_vec_info reduc_def;
>>
>> - /* The vector type for performing the actual reduction. */
>> - tree reduc_vectype;
>> -
>> - /* For loop reduction with multiple vectorized results (ncopies > 1), a
>> - lane-reducing operation participating in it may not use all of those
>> - results, this field specifies result index starting from which any
>> - following land-reducing operation would be assigned to. */
>> - unsigned int reduc_result_pos;
>> -
>> - /* If IS_REDUC_INFO is true and if the vector code is performing
>> - N scalar reductions in parallel, this variable gives the initial
>> - scalar values of those N reductions. */
>> - vec<tree> reduc_initial_values;
>> -
>> - /* If IS_REDUC_INFO is true and if the vector code is performing
>> - N scalar reductions in parallel, this variable gives the vectorized
>> code's
>> - final (scalar) result for each of those N reductions. In other words,
>> - REDUC_SCALAR_RESULTS[I] replaces the original scalar code's loop-closed
>> - SSA PHI for reduction number I. */
>> - vec<tree> reduc_scalar_results;
>> -
>> - /* Only meaningful if IS_REDUC_INFO. If non-null, the reduction is
>> - being performed by an epilogue loop and we have decided to reuse
>> - this accumulator from the main loop. */
>> - vect_reusable_accumulator *reused_accumulator;
>> -
>> - /* Whether we force a single cycle PHI during reduction vectorization. */
>> - bool force_single_cycle;
>> -
>> - /* Whether on this stmt reduction meta is recorded. */
>> - bool is_reduc_info;
>> -
>> /* If nonzero, the lhs of the statement could be truncated to this
>> many bits without affecting any users of the result. */
>> unsigned int min_output_precision;
>> @@ -2674,8 +2690,7 @@ extern tree vect_gen_loop_len_mask (loop_vec_info,
>> gimple_stmt_iterator *,
>> unsigned int, tree, tree, unsigned int,
>> unsigned int);
>> extern gimple_seq vect_gen_len (tree, tree, tree, tree);
>> -extern vect_reduc_info info_for_reduction (vec_info *, stmt_vec_info);
>> -extern vect_reduc_info create_info_for_reduction (vec_info *,
>> stmt_vec_info);
>> +extern vect_reduc_info info_for_reduction (loop_vec_info, slp_tree);
>> extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *);
>>
>> /* Drive for loop transformation stage. */
>> @@ -2891,7 +2906,14 @@ vect_is_store_elt_extraction (vect_cost_for_stmt
>> kind, stmt_vec_info stmt_info)
>> inline bool
>> vect_is_reduction (stmt_vec_info stmt_info)
>> {
>> - return STMT_VINFO_REDUC_IDX (stmt_info) >= 0;
>> + return STMT_VINFO_REDUC_IDX (stmt_info) != -1;
>> +}
>> +
>> +/* Return true if SLP_NODE represents part of a reduction. */
>> +inline bool
>> +vect_is_reduction (slp_tree slp_node)
>> +{
>> + return SLP_TREE_REDUC_IDX (slp_node) != -1;
>> }
>>
>> /* If STMT_INFO describes a reduction, return the vect_reduction_type
>> @@ -2905,7 +2927,7 @@ vect_reduc_type (vec_info *vinfo, slp_tree node)
>> if (STMT_VINFO_REDUC_DEF (stmt_info))
>> {
>> vect_reduc_info reduc_info
>> - = info_for_reduction (loop_vinfo, stmt_info);
>> + = info_for_reduction (loop_vinfo, node);
>> return int (VECT_REDUC_INFO_TYPE (reduc_info));
>> }
>> }
>> --
>> 2.43.0
>
