In general, the existing gimple_build_vector function cannot
safely be used for variable-length vector types because it
relies on the type associated with the given tree_vector_builder
having a constant number of subparts, or all of the encoded
elements of that builder being constant. This imposes a lot of
constraints on callers that they might not be aware of.
Even if the first limitation were lifted (for example by using
the lower bound of the number of subparts), it would be impossible
for gimple_build_vector to implement correct semantics for
builders that have a VLA type and non-constant elements because a
builder encodes an infinite sequence of elements that a CONSTRUCTOR
node might not be able to represent.
It is possible to specify a zero-padded vector of infinite length
as a tree_vector_builder in which each potentially non-zero element
is specified as a pattern of two encoded elements in which the
second (repeating) element is constant zero. However, it seems
undesirable to impose that complexity on callers of
gimple_build_vector, with corresponding complexity inside the
callee to determine whether or not the passed-in tree_vector_builder
satisfies all of the contraints to allow it to be represented as a
CONSTRUCTOR node.
There are only two call sites at which representing the input to
gimple_build_vector as a tree_vector_builder is necessary (in
get_initial_defs_for_reduction and vect_create_constant_vectors); in
all other cases, an infinite-length sequence is not needed.
The existing function is renamed as gimple_try_build_repeating_vector
to make its semantics clear and because it can now return a failure
indication to relieve its callers of trying to predict its failure
modes. It was only ever passed a tree_vector_builder that had one
element per pattern (and it did not work for stepped patterns),
therefore it has been modified to accept a vector type and list of
elements like the duplicate_and_interleave function it is often used
as an alternative to. This removes any possible impression that it
might support other shapes and helps to eliminate code duplication
between the two call sites where repetition is actually needed.
Created a new function, gimple_build_simple_vector, for use when
creating vectorized definitions for basic block vectorization in
vect_create_constant_vectors. It also takes a vector type and list
of elements to be put into the built vector, instead of an instance
of tree_vector_builder. If the number of values is zero then a zero
constant is built. If all values are constant then a vector constant
is built. Otherwise, a new constructor node is created. Unlike
gimple_try_build_repeating_vector, this function cannot fail.
Whenever the tree_vector_builder supplied by an existing
caller of gimple_build_vector specified exactly the number of
elements in a non-VLA type, a call to gimple_build_simple_vector
has been substituted to make the code more unambiguous and
robust. This simplifies a lot of existing code by removing the
requirement for callers to understand tree_vector_builder and set
its parameters correctly.
gcc/ChangeLog:
* gimple-fold.cc (gimple_try_build_repeating_vector):
New function to build a VECTOR_CST or CONSTRUCTOR node
from a sequence generated by repeating a given list of
element nodes to fill a (potentially variable-length)
vector, if possible. Can fail to create a CONSTRUCTOR
node and return NULL_TREE, if non-constant.
(gimple_build_vector): Deleted.
(gimple_build_simple_vector): New function to build
a VECTOR_CST or CONSTRUCTOR node from a sequence
generated by zero-padding a given list of element nodes
to fill a (potentially variable-length) vector. Cannot
fail.
* gimple-fold.h (gimple_build_vector): Deleted.
(gimple_try_build_repeating_vector): Declare the new
function and simpler overloaded versions with fewer
parameters.
(gimple_build_simple_vector): Declare the new
function and a simpler overloaded version with fewer
parameters.
* tree-vect-loop.cc (get_initial_defs_for_reduction):
Substitute an auto_vec of element nodes for the
tree_vector_builder that had only one shape and was
only required by gimple_build_vector.
Remove tracking of whether at least one element is
non-constant and instead check the result of
gimple_try_build_repeating_vector.
Call gimple_build_simple_vector or
gimple_try_build_repeating_vector as appropriate
(instead of gimple_build_vector).
(vect_create_nonlinear_iv_init): Substitute
auto_vec of elements nodes for tree_vector_builder
that had only one shape.
Call gimple_build_simple_vector instead of
gimple_build_vector where repetition is not needed.
(vectorizable_induction): As above.
* tree-vect-slp.cc (duplicate_and_interleave):
As above.
(vect_create_constant_vectors): Substitute an
auto_vec of element nodes for a tree_vector_builder
that had only one shape.
Use gimple_build_simple_vector instead of
duplicate_and_interleave to create vectors for BB SLP
vectorization.
Call gimple_build_simple_vector or
gimple_try_build_repeating_vector as appropriate for
loop vectorization (instead of gimple_build_vector).
Remove tracking of whether at least one element is
non-constant and instead check the result of
gimple_try_build_repeating_vector.
Add an assertion to guard against an outer loop never
terminating because an inner loop is never entered.
---
gcc/gimple-fold.cc | 146 +++++++++++++++++++++++++++++++-----------
gcc/gimple-fold.h | 44 ++++++++++---
gcc/tree-vect-loop.cc | 60 ++++++++++-------
gcc/tree-vect-slp.cc | 93 ++++++++++++++++++---------
4 files changed, 244 insertions(+), 99 deletions(-)
diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc
index 4422a48383d..36df2b744bc 100644
--- a/gcc/gimple-fold.cc
+++ b/gcc/gimple-fold.cc
@@ -11421,45 +11421,119 @@ gimple_build_vector_from_val (gimple_stmt_iterator
*gsi,
return res;
}
-/* Build a vector from BUILDER, handling the case in which some elements
- are non-constant. Return a gimple value for the result, inserting
- any new instructions to GSI honoring BEFORE and UPDATE.
+/* Build a vector of type VECTYPE from a repeating list of ELTS, potentially
+ handling the case in which some elements are non-constant. Return NULL_TREE
+ if VECTYPE has a variable number of subparts and at least one of the ELTS
+ is non-constant. Otherwise, return a gimple value for the result, inserting
+ any new instructions to GSI honoring BEFORE and UPDATE. */
- BUILDER must not have a stepped encoding on entry. This is because
- the function is not geared up to handle the arithmetic that would
- be needed in the variable case, and any code building a vector that
- is known to be constant should use BUILDER->build () directly. */
+tree
+gimple_try_build_repeating_vector (gimple_stmt_iterator *gsi, bool before,
+ gsi_iterator_update update, location_t loc,
+ tree vectype, const vec<tree> &elts)
+{
+ unsigned int nelts = elts.length ();
+
+ tree_vector_builder builder (vectype, nelts, 1);
+ builder.quick_grow (nelts);
+
+ bool is_constant = true;
+ for (unsigned int i = 0; i < nelts; ++i)
+ {
+ if (!CONSTANT_CLASS_P (elts[i]))
+ is_constant = false;
+
+ builder[i] = elts[i];
+ }
+
+ if (is_constant)
+ return builder.build ();
+
+ /* If the number of subparts is not constant then we cannot create
+ a CONSTRUCTOR node that has that number of elements. The pattern
+ needs to repeat indefinitely, which requires a VECTOR_CST. */
+ unsigned HOST_WIDE_INT const_nelts;
+ if (!TYPE_VECTOR_SUBPARTS (vectype).is_constant (&const_nelts))
+ return NULL_TREE;
+
+ vec<constructor_elt, va_gc> *v;
+ vec_alloc (v, const_nelts);
+ for (unsigned HOST_WIDE_INT i = 0; i < const_nelts; ++i)
+ CONSTRUCTOR_APPEND_ELT (v, NULL_TREE, builder.elt (i));
+
+ tree res;
+ if (gimple_in_ssa_p (cfun))
+ res = make_ssa_name (vectype);
+ else
+ res = create_tmp_reg (vectype);
+
+ gimple *stmt = gimple_build_assign (res, build_constructor (vectype, v));
+ gimple_set_location (stmt, loc);
+
+ gimple_seq seq = NULL;
+ gimple_seq_add_stmt_without_update (&seq, stmt);
+ gimple_build_insert_seq (gsi, before, update, seq);
+ return res;
+}
+
+/* Build a vector of type VECTYPE from a partial list of ELTS, handling the
case
+ in which some elements are non-constant. Return a gimple value for the
+ result, inserting any new instructions to GSI honoring BEFORE and UPDATE.
+
+ Omitted elements are implicitly zero. Unlike
+ gimple_try_build_repeating_vector, this function can be used for a
+ variable-length VECTYPE even if some elements might be non-constant because
+ the resultant CONSTRUCTOR node need only explicitly encode the number of
+ elements supplied in ELTS.
+ */
tree
-gimple_build_vector (gimple_stmt_iterator *gsi,
- bool before, gsi_iterator_update update,
- location_t loc, tree_vector_builder *builder)
-{
- gcc_assert (builder->nelts_per_pattern () <= 2);
- unsigned int encoded_nelts = builder->encoded_nelts ();
- for (unsigned int i = 0; i < encoded_nelts; ++i)
- if (!CONSTANT_CLASS_P ((*builder)[i]))
- {
- gimple_seq seq = NULL;
- tree type = builder->type ();
- unsigned int nelts = TYPE_VECTOR_SUBPARTS (type).to_constant ();
- vec<constructor_elt, va_gc> *v;
- vec_alloc (v, nelts);
- for (i = 0; i < nelts; ++i)
- CONSTRUCTOR_APPEND_ELT (v, NULL_TREE, builder->elt (i));
-
- tree res;
- if (gimple_in_ssa_p (cfun))
- res = make_ssa_name (type);
- else
- res = create_tmp_reg (type);
- gimple *stmt = gimple_build_assign (res, build_constructor (type, v));
- gimple_set_location (stmt, loc);
- gimple_seq_add_stmt_without_update (&seq, stmt);
- gimple_build_insert_seq (gsi, before, update, seq);
- return res;
- }
- return builder->build ();
+gimple_build_simple_vector (gimple_stmt_iterator *gsi, bool before,
+ gsi_iterator_update update, location_t loc,
+ tree vectype, const vec<tree> &elts)
+{
+ unsigned int nelts = elts.length ();
+ gcc_assert (TREE_CODE (vectype) == VECTOR_TYPE);
+ unsigned int lower_bound
+ = constant_lower_bound (TYPE_VECTOR_SUBPARTS (vectype));
+ gcc_assert (nelts <= lower_bound);
+
+ if (nelts == 0)
+ return build_zero_cst (vectype);
+
+ /* Prepare a vector of constructor elements and find out whether all
+ of the element values are constant. */
+ vec<constructor_elt, va_gc> *v;
+ vec_alloc (v, nelts);
+ bool is_constant = true;
+
+ for (unsigned int i = 0; i < nelts; ++i)
+ {
+ if (!CONSTANT_CLASS_P (elts[i]))
+ is_constant = false;
+
+ CONSTRUCTOR_APPEND_ELT (v, NULL_TREE, elts[i]);
+ }
+
+ /* If all element values are constant then we can return a new VECTOR_CST
+ node. Any elements for which no value is supplied will be zero. */
+ if (is_constant)
+ return build_vector_from_ctor (vectype, v);
+
+ tree res;
+ if (gimple_in_ssa_p (cfun))
+ res = make_ssa_name (vectype);
+ else
+ res = create_tmp_reg (vectype);
+
+ gimple *stmt = gimple_build_assign (res, build_constructor (vectype, v));
+ gimple_set_location (stmt, loc);
+
+ gimple_seq seq = NULL;
+ gimple_seq_add_stmt_without_update (&seq, stmt);
+ gimple_build_insert_seq (gsi, before, update, seq);
+
+ return res;
}
/* Emit gimple statements into &stmts that take a value given in OLD_SIZE
diff --git a/gcc/gimple-fold.h b/gcc/gimple-fold.h
index f1853560779..42a0e78e4a1 100644
--- a/gcc/gimple-fold.h
+++ b/gcc/gimple-fold.h
@@ -224,23 +224,47 @@ gimple_build_vector_from_val (gimple_seq *seq, tree type,
tree op)
}
class tree_vector_builder;
-extern tree gimple_build_vector (gimple_stmt_iterator *, bool,
- enum gsi_iterator_update,
- location_t, tree_vector_builder *);
+extern tree gimple_try_build_repeating_vector (gimple_stmt_iterator *, bool,
+ enum gsi_iterator_update,
+ location_t, tree vectype,
+ const vec<tree> &elts);
inline tree
-gimple_build_vector (gimple_seq *seq, location_t loc,
- tree_vector_builder *builder)
+gimple_try_build_repeating_vector (gimple_seq *seq, location_t loc,
+ tree vectype, const vec<tree> &elts)
{
gimple_stmt_iterator gsi = gsi_last (*seq);
- return gimple_build_vector (&gsi, false, GSI_CONTINUE_LINKING,
- loc, builder);
+ return gimple_try_build_repeating_vector (&gsi, false, GSI_CONTINUE_LINKING,
+ loc, vectype, elts);
}
inline tree
-gimple_build_vector (gimple_seq *seq, tree_vector_builder *builder)
+gimple_try_build_repeating_vector (gimple_seq *seq, tree vectype,
+ const vec<tree> &elts)
{
gimple_stmt_iterator gsi = gsi_last (*seq);
- return gimple_build_vector (&gsi, false, GSI_CONTINUE_LINKING,
- UNKNOWN_LOCATION, builder);
+ return gimple_try_build_repeating_vector (&gsi, false, GSI_CONTINUE_LINKING,
+ UNKNOWN_LOCATION, vectype, elts);
+}
+
+extern tree gimple_build_simple_vector (gimple_stmt_iterator *, bool,
+ enum gsi_iterator_update, location_t,
+ tree vectype, const vec<tree> &elts);
+
+inline tree
+gimple_build_simple_vector (gimple_seq *seq, location_t loc, tree vectype,
+ const vec<tree> &elts)
+{
+ gimple_stmt_iterator gsi = gsi_last (*seq);
+ return gimple_build_simple_vector (&gsi, false, GSI_CONTINUE_LINKING, loc,
+ vectype, elts);
+}
+
+inline tree
+gimple_build_simple_vector (gimple_seq *seq, tree vectype,
+ const vec<tree> &elts)
+{
+ gimple_stmt_iterator gsi = gsi_last (*seq);
+ return gimple_build_simple_vector (&gsi, false, GSI_CONTINUE_LINKING,
+ UNKNOWN_LOCATION, vectype, elts);
}
extern tree gimple_build_round_up (gimple_stmt_iterator *, bool,
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 0167d52b28b..aa74fc097d1 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -4941,8 +4941,8 @@ get_initial_defs_for_reduction (loop_vec_info loop_vinfo,
tree vector_elt_type = TREE_TYPE (vector_type);
number_of_places_left_in_vector = nunits;
- bool constant_p = true;
- tree_vector_builder elts (vector_type, nunits, 1);
+ auto_vec<tree> elts;
+ elts.create (nunits);
elts.quick_grow (nunits);
gimple_seq ctor_seq = NULL;
if (neutral_op
@@ -4991,17 +4991,16 @@ get_initial_defs_for_reduction (loop_vec_info
loop_vinfo,
/* Create 'vect_ = {op0,op1,...,opn}'. */
number_of_places_left_in_vector--;
elts[nunits - number_of_places_left_in_vector - 1] = op;
- if (!CONSTANT_CLASS_P (op))
- constant_p = false;
if (number_of_places_left_in_vector == 0)
{
- tree init;
- if (constant_p && !neutral_op
- ? multiple_p (TYPE_VECTOR_SUBPARTS (vector_type), nunits)
- : known_eq (TYPE_VECTOR_SUBPARTS (vector_type), nunits))
- /* Build the vector directly from ELTS. */
- init = gimple_build_vector (&ctor_seq, &elts);
+ tree init = NULL_TREE;
+ if (known_eq (TYPE_VECTOR_SUBPARTS (vector_type), nunits))
+ {
+ /* No repetition or neutral value is needed. */
+ init = gimple_build_simple_vector (&ctor_seq, vector_type, elts);
+ gcc_checking_assert (init);
+ }
else if (neutral_op)
{
/* Build a vector of the neutral value and shift the
@@ -5017,8 +5016,18 @@ get_initial_defs_for_reduction (loop_vec_info loop_vinfo,
init = gimple_build (&ctor_seq, CFN_VEC_SHL_INSERT,
vector_type, init, elts[k]);
}
+ gcc_checking_assert (init);
}
- else
+ else if (multiple_p (TYPE_VECTOR_SUBPARTS (vector_type), nunits))
+ {
+ /* Repeat ELTS to fill the vector. This can fail to build a
+ CONSTRUCTOR node if there are non-constant elements and
+ VECTOR_TYPE is variable-length. */
+ init = gimple_try_build_repeating_vector (&ctor_seq, vector_type,
+ elts);
+ }
+
+ if (!init)
{
/* First time round, duplicate ELTS to fill the
required number of vectors. */
@@ -5029,9 +5038,6 @@ get_initial_defs_for_reduction (loop_vec_info loop_vinfo,
vec_oprnds->quick_push (init);
number_of_places_left_in_vector = nunits;
- elts.new_vector (vector_type, nunits, 1);
- elts.quick_grow (nunits);
- constant_p = true;
}
}
if (ctor_seq != NULL)
@@ -8988,7 +8994,8 @@ vect_create_nonlinear_iv_init (gimple_seq* stmts, tree
init_expr,
vec_init = gimple_build_vector_from_val (stmts,
uvectype,
new_name);
- tree_vector_builder elts (uvectype, const_nunits, 1);
+ auto_vec<tree> elts;
+ elts.reserve (const_nunits);
tree elt_step = build_one_cst (utype);
elts.quick_push (elt_step);
@@ -9001,9 +9008,8 @@ vect_create_nonlinear_iv_init (gimple_seq* stmts, tree
init_expr,
}
/* Create a vector from [new_name_0, new_name_1, ...,
new_name_nunits-1]. */
- tree vec_mul = gimple_build_vector (stmts, &elts);
- vec_init = gimple_build (stmts, MULT_EXPR, uvectype,
- vec_init, vec_mul);
+ tree vec_mul = gimple_build_simple_vector (stmts, uvectype, elts);
+ vec_init = gimple_build (stmts, MULT_EXPR, uvectype, vec_init, vec_mul);
vec_init = gimple_convert (stmts, vectype, vec_init);
}
break;
@@ -9798,9 +9804,12 @@ vectorizable_induction (loop_vec_info loop_vinfo,
bool invariant = true;
if (nunits.is_constant (&const_nunits))
{
- tree_vector_builder step_elts (step_vectype, const_nunits, 1);
- tree_vector_builder init_elts (vectype, const_nunits, 1);
- tree_vector_builder mul_elts (step_vectype, const_nunits, 1);
+ auto_vec<tree> step_elts, init_elts, mul_elts;
+ step_elts.reserve (const_nunits);
+ mul_elts.reserve (const_nunits);
+ if (!init_node)
+ init_elts.reserve (const_nunits);
+
for (unsigned eltn = 0; eltn < const_nunits; ++eltn)
{
/* The scalar steps of the IVs. */
@@ -9824,10 +9833,13 @@ vectorizable_induction (loop_vec_info loop_vinfo,
UNSIGNED)
: build_int_cstu (stept, mul_elt));
}
- vec_step = gimple_build_vector (&init_stmts, &step_elts);
- step_mul = gimple_build_vector (&init_stmts, &mul_elts);
+ vec_step
+ = gimple_build_simple_vector (&init_stmts, step_vectype, step_elts);
+ step_mul
+ = gimple_build_simple_vector (&init_stmts, step_vectype, mul_elts);
if (!init_node)
- vec_init = gimple_build_vector (&init_stmts, &init_elts);
+ vec_init
+ = gimple_build_simple_vector (&init_stmts, vectype, init_elts);
}
else
{
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 5d7e96b4e7e..6a7a3af1dad 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -10676,19 +10676,22 @@ duplicate_and_interleave (vec_info *vinfo, gimple_seq
*seq, tree vector_type,
unsigned int partial_nelts = nelts / nvectors;
tree partial_vector_type = build_vector_type (element_type, partial_nelts);
- tree_vector_builder partial_elts;
+ auto_vec<tree> partial_elts;
auto_vec<tree, 32> pieces (nvectors * 2);
pieces.quick_grow_cleared (nvectors * 2);
for (unsigned int i = 0; i < nvectors; ++i)
{
/* (2) Replace ELTS[0:NELTS] with ELTS'[0:NELTS'], where each element of
ELTS' has mode IM. */
- partial_elts.new_vector (partial_vector_type, partial_nelts, 1);
+ partial_elts.truncate (0);
+ partial_elts.reserve (partial_nelts);
+
for (unsigned int j = 0; j < partial_nelts; ++j)
partial_elts.quick_push (elts[i * partial_nelts + j]);
- tree t = gimple_build_vector (seq, &partial_elts);
- t = gimple_build (seq, VIEW_CONVERT_EXPR,
- TREE_TYPE (new_vector_type), t);
+
+ tree t
+ = gimple_build_simple_vector (seq, partial_vector_type, partial_elts);
+ t = gimple_build (seq, VIEW_CONVERT_EXPR, TREE_TYPE (new_vector_type),
t);
/* (3) Duplicate each ELTS'[I] into a vector of mode VM. */
pieces[i] = gimple_build_vector_from_val (seq, new_vector_type, t);
@@ -10762,14 +10765,12 @@ static void
vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node)
{
unsigned HOST_WIDE_INT nunits;
- tree vec_cst;
unsigned j, number_of_places_left_in_vector;
tree vector_type;
tree vop;
- int group_size = op_node->ops.length ();
+ unsigned int group_size = op_node->ops.length ();
unsigned int vec_num, i;
unsigned number_of_copies = 1;
- bool constant_p;
gimple_seq ctor_seq = NULL;
auto_vec<tree, 16> permute_results;
@@ -10796,17 +10797,30 @@ vect_create_constant_vectors (vec_info *vinfo,
slp_tree op_node)
(s1, s2, ..., s8). We will create two vectors {s1, s2, s3, s4} and
{s5, s6, s7, s8}. */
- /* When using duplicate_and_interleave, we just need one element for
- each scalar statement. */
- if (!TYPE_VECTOR_SUBPARTS (vector_type).is_constant (&nunits))
- nunits = group_size;
+ if (is_a<bb_vec_info> (vinfo))
+ {
+ /* We don't use duplicate_and_interleave for basic block vectorization.
+ We know that either the group size is exactly divisible by the vector
+ length or it fits within a single vector. */
+ nunits = constant_lower_bound (TYPE_VECTOR_SUBPARTS (vector_type));
+ gcc_checking_assert (multiple_p (group_size, nunits)
+ || known_le (group_size, nunits));
+ nunits = MIN (nunits, group_size);
+ }
+ else
+ {
+ /* When using duplicate_and_interleave, we just need one element for
+ each scalar statement. */
+ if (!TYPE_VECTOR_SUBPARTS (vector_type).is_constant (&nunits))
+ nunits = group_size;
+ }
number_of_copies = nunits * number_of_vectors / group_size;
number_of_places_left_in_vector = nunits;
- constant_p = true;
tree uniform_elt = NULL_TREE;
- tree_vector_builder elts (vector_type, nunits, 1);
+ auto_vec<tree> elts;
+ elts.create (nunits);
elts.quick_grow (nunits);
stmt_vec_info insert_after = NULL;
for (j = 0; j < number_of_copies; j++)
@@ -10872,8 +10886,7 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree
op_node)
}
}
elts[number_of_places_left_in_vector] = op;
- if (!CONSTANT_CLASS_P (op))
- constant_p = false;
+
/* For BB vectorization we have to compute an insert location
when a def is inside the analyzed region since we cannot
simply insert at the BB start in this case. */
@@ -10891,15 +10904,34 @@ vect_create_constant_vectors (vec_info *vinfo,
slp_tree op_node)
if (number_of_places_left_in_vector == 0)
{
+ tree vec_cst = NULL_TREE;
auto type_nunits = TYPE_VECTOR_SUBPARTS (vector_type);
if (uniform_elt)
- vec_cst = gimple_build_vector_from_val (&ctor_seq, vector_type,
- elts[0]);
- else if (constant_p
- ? multiple_p (type_nunits, nunits)
- : known_eq (type_nunits, nunits))
- vec_cst = gimple_build_vector (&ctor_seq, &elts);
- else
+ {
+ vec_cst = gimple_build_vector_from_val (&ctor_seq,
+ vector_type, elts[0]);
+ gcc_checking_assert (vec_cst);
+ }
+ else if (is_a<bb_vec_info> (vinfo)
+ || known_eq (type_nunits, nunits))
+ {
+ /* No repetition is needed because we are doing basic block
+ vectorization or a value was supplied for every element. */
+ vec_cst
+ = gimple_build_simple_vector (&ctor_seq, vector_type, elts);
+ gcc_checking_assert (vec_cst);
+ }
+ else if (multiple_p (type_nunits, nunits))
+ {
+ /* The encoded elements must be repeated to fill every subpart
+ of SLP_TREE_VECTYPE for loop vectorization. This can fail
+ to build a CONSTRUCTOR node if there are non-constant
+ elements and VECTOR_TYPE is variable-length. */
+ vec_cst
+ = gimple_try_build_repeating_vector (&ctor_seq, vector_type,
+ elts);
+ }
+ if (!vec_cst)
{
if (permute_results.is_empty ())
duplicate_and_interleave (vinfo, &ctor_seq, vector_type,
@@ -10943,9 +10975,6 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree
op_node)
voprnds.quick_push (vec_cst);
insert_after = NULL;
number_of_places_left_in_vector = nunits;
- constant_p = true;
- elts.new_vector (vector_type, nunits, 1);
- elts.quick_grow (nunits);
}
}
}
@@ -10964,9 +10993,15 @@ vect_create_constant_vectors (vec_info *vinfo,
slp_tree op_node)
NUMBER_OF_SCALARS/NUNITS or NUNITS/NUMBER_OF_SCALARS, and hence we have
to replicate the vectors. */
while (number_of_vectors > SLP_TREE_VEC_DEFS (op_node).length ())
- for (i = 0; SLP_TREE_VEC_DEFS (op_node).iterate (i, &vop) && i < vec_num;
- i++)
- SLP_TREE_VEC_DEFS (op_node).quick_push (vop);
+ {
+ /* Guard against the outer loop never terminating because the
+ inner loop is never entered. */
+ gcc_checking_assert (vec_num > 0);
+
+ for (i = 0; SLP_TREE_VEC_DEFS (op_node).iterate (i, &vop) && i < vec_num;
+ i++)
+ SLP_TREE_VEC_DEFS (op_node).quick_push (vop);
+ }
}
/* Get the scalar definition of the Nth lane from SLP_NODE or NULL_TREE
--
2.43.0
