> -----Original Message-----
> From: Richard Biener <[email protected]>
> Sent: 08 June 2026 14:08
> To: Tamar Christina <[email protected]>
> Cc: [email protected]; nd <[email protected]>; [email protected]
> Subject: Re: [PATCH][vect][WIP]: teach the vectorizer to be able to vectorize
> already vectorized code [PR119187]
>
> On Fri, 5 Jun 2026, Tamar Christina wrote:
>
> > NOTE: This branch is still very much WIP. It has not been cleaned up enough
> > to be upstreamable yet, but I am sending it now to give Richi an idea of the
> > design direction.
> >
> > This should apply to any reasonably recent trunk. I am currently based on
> > g:c8ca8e647caf945dc4694107884cf0f4eba5da2f.
> >
> > The branch builds and runs, but it does not bootstrap. I have only tested
> > incremental builds.
> >
> > Some parts are more worked out than others, since I have not reached all of
> > them yet. I have tried to write down the intended direction as well, so
> > that
> > you can give me "the look", Richi :)
> >
> > I *also* want to point out that the AArch64 code here is just illustrative
> > and
> > not how we will implement these. Work is ongoing to replace our intrinsics
> with
> > ones that generate new overloaded builtins, to allow legality checks to be
> > cleaner and also not need the type shims between the internal vectorizer
> > represenations and intrinsics. i.e. atm we need a type shim between
> >
> > int32x4_t and a signed type made form V4SI. This will be handled by adding
> > to the build-ins internal overloads with the correct types.
> >
> > ### Introduction
> >
> > The main intended purpose of this work is threefold:
> >
> > 1. Support re-optimizing intrinsics and vector code in SLP that was created
> by
> > earlier vectorization passes. For example, two back-to-back loops could
> > be
> > completely unrolled and vectorized during loop vectorization, but would
> not
> > then be optimized together. The idea is to let SLP discovery re-analyze
> > the
> > vector code as a single entity. This makes the long-standing question of
> > whether BB SLP or loop vectorization should run first less important,
> > since
> > we can re-vectorize the generated vector code.
> >
> > 2. Support Arm SVE2's Hybrid Vector Length Agnostic (HVLA) programming
> model.
> > The intention is to transform code written with intrinsics for older
> > AdvSIMD
> > code into VLA code, so that we can use longer vector lengths.
> >
> > 3. Support workloads, particularly scientific-computing workloads, in which
> > users manually create vector types that are longer than the target
> > supports
> > using GCC vector extensions. The expectation is that the compiler can
> > then
> > decompose those operations into supported operations. Clang can do this
> for
> > some examples, although its approach seems quite limited and appears to
> work
> > only when the operations are constructors of PHI nodes.
>
> I hope to replace "optimized" vector lowering with re-vectorization,
> possibly only keep lowering to scalar operations.
>
Yeah, I only started on this use case recently hence why it's a bit less worked
out.
I plan on continuing work on this next week to flesh it out more.
> > To solve these problems, the approach I have taken in GCC is to model vector
> > types as possible base "scalar" types. The main advantage is that the
> > current
> > compiler workflows can stay the same. Fundamentally, this does not change
> most
> > of the vectorizer, apart from analysis and the usual extensions to the code
> > generators.
> >
> > This also means that the code remains subject to cost modelling and to other
> > features we have added over the years, such as epilogue masks.
> >
> > As an example:
> >
> > ```c
> > #include <arm_neon.h>
> >
> > void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> > {
> > for (int i = 0; i < n; i += 16, a += 16, b += 16, c += 16)
> > {
> > uint8x8_t av1 = vld1_u8 (a);
> > uint8x8_t av2 = vld1_u8 (a + 8);
> > uint8x8_t bv1 = vld1_u8 (b);
> > uint8x8_t bv2 = vld1_u8 (b + 8);
> > vst1_u8 (c, vadd_u8 (av1, bv1));
> > vst1_u8 (c + 8, vadd_u8 (av2, bv2));
> > }
> > }
> > ```
> >
> > At the moment, `-O3` generates:
> >
> > ```asm
> > .L3:
> > ldr d29, [x0, x4]
> > ldr d28, [x1, x4]
> > ldr d31, [x7, x4]
> > ldr d30, [x6, x4]
> > add v28.8b, v29.8b, v28.8b
> > add v30.8b, v31.8b, v30.8b
> > str d28, [x2, x4]
> > str d30, [x5, x4]
> > add x4, x4, 16
> > cmp w3, w4
> > bgt .L3
> > ```
> >
> > This underuses the available load bandwidth. Ideally these would be Q-sized
> > loads and one ADD, i.e. SLP would combine them.
> >
> > With the branch, `-O3 -march=armv8-a` generates:
> >
> > ```asm
> > .L5:
> > ldr q31, [x0, x4]
> > ldr q30, [x1, x4]
> > add v30.16b, v31.16b, v30.16b
> > str q30, [x2, x4]
> > add x4, x4, 16
> > cmp x4, x7
> > bne .L5
> > ```
> >
> > `-O3 -march=armv8-a+sve` generates:
> >
> > ```asm
> > .L5:
> > ldr q31, [x0, x3]
> > ldr q30, [x1, x3]
> > add v30.16b, v31.16b, v30.16b
> > str q30, [x2, x3]
> > add x3, x3, 16
> > cmp x5, x3
> > bne .L5
> > lsl w3, w4, 4
> > cmp w6, w3
> > beq .L1
> > .L4:
> > sub w6, w6, w3
> > ubfiz x3, x3, 3, 32
> > whilelo p7.b, xzr, x6
> > ld1b z29.b, p7/z, [x0, x3]
> > ld1b z28.b, p7/z, [x1, x3]
> > add z28.b, z28.b, z29.b
> > st1b z28.b, p7, [x2, x3]
> > .L1:
> > ```
> >
> > `-O3 -march=armv8-a+sve -mautovec-preference=sve-only` generates:
> >
> > ```asm
> > .L4:
> > ld1b z31.b, p7/z, [x0, x4]
> > ld1b z30.b, p7/z, [x1, x4]
> > add z30.b, z30.b, z31.b
> > st1b z30.b, p7, [x2, x4]
> > add x4, x4, x5
> > whilelo p7.b, x4, x3
> > b.any .L4
> > ```
> >
> > ### Representation
> >
> > There is no change to the SLP trees themselves. The vector operation is
> added
> > to the node with N copies. `print_slp` will be updated to show that these
> > are
> > vector lanes rather than individual scalar operations. This is a
> > representational thing since SLP internal nodes are vectorized from the
> > representation and not the scalar elements, but it makes everything line
> > up.
> >
> > The group size for the operation is then set to the number of units in the
> > incoming vector. For memory accesses, `dr_group_size` is increased too. At
> > the moment, and probably for the first version, I only plan to support
> > linear
> > loads and load-lane operations. Recognizing the other load types requires a
> > target hook and time to fill in the DRs.
>
> I think we need to nail this at least. So for a v4si add you end up
> with 4 times the vector add in SLP_TREE_SCALAR_STMTS, one stmt
> per resulting vector lane. I see how this is the least problematic
> representation, but is it a good one? In most cases we do not
> absolutely need an entry in SLP_TREE_SCALAR_STMTS. The other
> "least" problematic representation would be to have no
> SLP_TREE_SCALAR_STMTS. I think I originally suggested to forfeit
> the one lane, one SLP_TREE_SCALAR_STMTS entry invariant so you'd
> have a single vector stmt entry in SLP_TREE_SCALAR_STMTS.
Hmm yeah I agree it's good to nail this down as it's the most critical piece.
I went with the one scalar statement per lane approach because there is
at least during early analysis a lot of code that look at the scalar statements
when analyzing loads and stores.
So we could go down to having the single vector statement variant by reworking
things a bit. My first patch tried this what became complicated is that
group_size
needed tweaking, since the single statement means that before trying to find a
lane
we'd have to figure out the order of statements.
That is with
Vector stmt
Scalar stmt
Vector stmt
Accessing lane X becomes harder as we have to do a stmt walk. This seemed like
an
Issue when determining e.g. what a permute is pointing to. So I chose to add
the
Statements repeatedly, and everything but the first one is just a dummy node,
and
Infact for printing I'll hide the other lanes and show something like "--
vector lane x--"
That said I now have a better understanding of this code so can try again if
you wish
to see what the knock on effects are.
>
> We do currently have (redudnant) SLP_TREE_LANES () that is most
> of the time equal to 'group size' which is currently most of the
> time equal to SLP_TREE_SCALAR_STMTS.length (). This is somewhat
> of a partial transition worth completing with this work.
Hmm that's one I hadn't tried. I was wondering why we have both SLP_TREE_LANES
and group_size. So If I finish the transition to SLP_TREE_LANES at least part
of
the issue would be solved since I wouldn't need the .length () to match.
I think a big driver to this representation instead is if we want to support
re-vectorization of VLA code. There are definitely use cases where this is
beneficial
but at t he least since I can't decompose into scalar stmts it can't be done at
the present.
So should I give the SLP_TREE_LANES () representation a try?
>
> BB external "vectors" currently populate SLP_TREE_VEC_DEFS, not
> SLP_TREE_SCALAR_STMTS, they are vect_external and facing SLP
> we always have a VEC_PERM node. That would be a very minimal
> case of "vector SLP" to handle in the new representation.
>
Yeah for BB vectorization I started the discovery as a constructor so the vector
statements stores can't be added as DEFS. It works for such things as
#include <arm_neon.h>
void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
uint8x8_t av1 = vld1_u8 (a);
uint8x8_t av2 = vld1_u8 (a+8);
uint8x8_t bv1 = vld1_u8 (b);
uint8x8_t bv2 = vld1_u8 (b+8);
vst1_u8 (c, vadd_u8 (av1, bv1));
vst1_u8 (c+8, vadd_u8 (av2, bv2));
}
> DRs group-size adjusting should be done based on scalar elements,
> we do eventually need to track gaps between two vector loads and
> code that advances the number by 1 + GAP needs to be adjusted to
> advance by <size> + GAP. For group element 0 we overload
> DR_GROUP_SIZE, for elts > 0 it's basically unused, so we'd want
> a DR_GROUP_ELT_SIZE I guess.
Yeah, I also wonder how to deal with e.g. gathers. My plan for now is
to ignore them. But we have a lot of non-liear loads in the ISA or
various different load + splat.
Like LD1RQB loads and splats 128-bit vectors over a VLA vector.
>
> + /* We should probably adjust group size, but for now lets hack. */
> + unsigned HOST_WIDE_INT vect_nunits = 0;
> + auto vscalar_stmt = STMT_VINFO_STMT (stmt_info);
> + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> + if (VECTOR_TYPE_P (vscalar_type)
> + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant (&vect_nunits))
> + dr_group_size *= vect_nunits;
>
> DR_GROUP_SIZE should be the size in elements. I'll note the above
> suggests that you currently do not (want to?) handle mixed type
> scalar stmts in an SLP node? Like re-vectorizing V2SI + SI
> to predicated tail V4SI?
>
Hmm isn't group size the number of units being loaded? The above code is
misleading in that it's expected that dr_group_size be 1 here before the
multiply. So really the above is the same as dr_group_size = vect_units.
The comment was trying to say that really I should do it when constructing
the DR. But similar to the group_size adjustment, doing it early makes it
an all or nothing thing so I delay it while I patch up code in smaller chunks.
I also do want to handle the V2SI + SI combinations like
#include <arm_neon.h>
void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
for (int i = 0; i < n; i+=16, a+=16, b+=16, c+=16)
{
uint8x8_t av1 = vld1_u8 (a);
uint8x8_t bv1 = vld1_u8 (b);
vst1_u8 (c, vadd_u8 (av1, bv1));
c[i+8] = a[i+8] + b[i+8];
c[i+9] = a[i+9] + b[i+9];
c[i+10] = a[i+10] + b[i+10];
c[i+11] = a[i+11] + b[i+11];
c[i+12] = a[i+12] + b[i+12];
c[i+13] = a[i+13] + b[i+13];
c[i+14] = a[i+14] + b[i+14];
c[i+15] = a[i+15] + b[i+15];
}
}
Atm though SLP build fails because of the different "type" of loads
The vector loads are
_68 = MEM <__Uint8x8_tD.2488> [(unsigned charD.21 * {ref-all})a_63(D) +
ivtmp.12_111 * 1];
Vs
_4 = MEM[(uint8_tD.4679 *)_91];
Not, similarly for the stores. This isn't unfixable, just probably needs
folded into how we want
to handle discovery.
> > For BB SLP, the current code starts a seed whenever it finds a store with a
> > vector type that the target does not support. This then fails analysis
> > because
> > no vector type can be found. The plan is to make vector type selection
> > return
> > the largest supported type for the given inner mode. During
> > `vectorizable_*`
> > analysis, we would then avoid doing any other check beyond whether the
> target
> > supports the new mode for the given operation. To match the incoming VF,
> we
> > would increase the required unroll factor so that we get the same effective
> VF.
> >
> > Decreasing the VF is not possible at the moment. For example:
> >
> > ```c
> > #include <arm_neon.h>
> >
> > void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> > {
> > for (int i = 0; i < n; i += 32, a += 32, b += 32, c += 32)
> > {
> > uint8x16_t av1 = vld1q_u8 (a);
> > uint8x16_t av2 = vld1q_u8 (a + 16);
> > uint8x16_t bv1 = vld1q_u8 (b);
> > uint8x16_t bv2 = vld1q_u8 (b + 16);
> > vst1q_u8 (c, vaddq_u8 (av1, bv1));
> > vst1q_u8 (c + 16, vaddq_u8 (av2, bv2));
> > }
> > }
> > ```
> >
> > cannot collapse down to:
> >
> > ```asm
> > .L5:
> > ldr q31, [x0, x4]
> > ldr q30, [x1, x4]
> > add v30.16b, v31.16b, v30.16b
> > str q30, [x2, x4]
> > add x4, x4, 16
> > cmp x4, x7
> > bne .L5
> > ```
> >
> > This would require fractional VF support; in other words, it would require
> > support for re-rolling the loop. With the current design, that should work
> > naturally once we add fractional VF support.
> >
> > ### Areas that need the most changes
> >
> > These are the areas that I think need the most work. The branch is
> > currently
> > structured so that the number-of-units adjustment is done in each callee
> rather
> > than when building the SLP representation. This is purely temporary, and
> there
> > are `#if 1`s around the temporary code so that I can remove it later.
> >
> > This lets me change the functions one at a time instead of having everything
> > ICE because asserts are hit in areas that I have not reached yet.
> >
> > ### Vector Operations
> >
> > There are two types of vector operations that we want to support here.
> >
> > The first type uses a normal GIMPLE representation. In the examples above,
> > `vadd` is lowered as a normal `PLUS_EXPR`, so testing and transforming it is
> > straightforward.
> >
> > In many other cases, however, we see target intrinsics. To vectorize
> > these, I
> > have added a new `vectorizable_vect_call` that uses a target hook,
> > `targetm.vectorize.translate_builtin_call`, to fill in vectorization
> > information about the call. In particular, the hook must be able to tell
> > the
> > vectorizer whether we are going from an unmasked mode to a masked
> mode, whether
> > there is a builtin with masking support that we can use, and where the mask
> > argument appears in the argument list.
>
> So with all the fancy stuff below I think we should start with a core
> set that IMO should tackle basic vector operations, loads and stores
> and those BB external vector data inputs.
>
Yeah, my basic testcase is load+store+add so e.g.
#include <arm_neon.h>
void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
{
for (int i = 0; i < n; i+=16, a+=16, b+=16, c+=16)
{
uint8x8_t av1 = vld1_u8 (a);
uint8x8_t av2 = vld1_u8 (a+8);
uint8x8_t bv1 = vld1_u8 (b);
uint8x8_t bv2 = vld1_u8 (b+8);
vst1_u8 (c, vadd_u8 (av1, bv1));
vst1_u8 (c+8, vadd_u8 (av2, bv2));
}
}
Which I can also just describe as vector extensions. But the HVLA work
is important for us as well so that's why I wanted to point it out and flesh
out the implementation a bit to avoid surprises :)
> Vector bools are indeed special but also important, it looks like what
> you do is reasonable.
>
> Given there's overlap (I'll cross-review that now) with SLP scalar
> stmts and DR represnetation in the predicated tail vectorization
> series getting this part correct will avoid future churn.
>
> I see "target hook" mentioned too many times ;)
So the target hooks are really only there for things that require actual target
input for HVLA. For code produced by an earlier vectorization pass or vector
extensions they're not needed.
But since HVLA transforms intrinsics code is completely opaque to GIMPLE
I need a way to describe them. I did try to minimize the number of hooks
need. Atm there is three:
1. Describe data-references for memory operations.
2. Discover and transform operations. In particularly check if you have a
version
with the mode required and if it needs a mask, and if so, where does the
mask
go. There is an alternative to this hook, which is using MASK_CALL and
folding.
This would work, but makes the tests quite a bit more involved than what
the
hook would do.
3. Describe essentially vector patterns, but again since the operations are
opague
there's no real alternative here.
But we get into details below, just wanted to clearly state them.
>
> > ### Vector Booleans
> >
> > Normally, the vectorizer has to detect booleans in order to differentiate
> > data
> > operations from control operations. When vectorizing existing vector code,
> > that has already been done for us. However, we still need to be able to
> expand
> > the lanes and, if necessary, convert between boolean representation models.
> >
> > For example, when changing AdvSIMD code into SVE code, the boolean
> > representation has to change because the underlying truth type is different.
> >
> > To fix this and support boolean operations, there is a new pattern,
> > `vect_recog_vect_bool_pattern`, that runs only on boolean vector
> comparisons.
> > It transforms them using a pattern as a scalar boolean. That is, it
> > creates:
> >
> > ```c
> > cmp ? -1 : 0
> > ```
> >
> > from a vector comparison, with the type of the operation set to the truth
> type
> > of the original code. This allows us to vectorize these as if they were
> > normal
> > scalar comparisons.
> >
> > For example:
> >
> > ```c
> > #include <arm_neon.h>
> >
> > void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> > {
> > for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
> > {
> > uint8x8_t av1 = vld1_u8 (a);
> > uint8x8_t bv1 = vld1_u8 (b);
> > vst1_u8 (c, vcgt_u8 (av1, bv1));
> > }
> > }
> > ```
> >
> > compiled with `-march=armv9-a+sve -O3` gives:
> >
> > ```asm
> > .L5:
> > ldr q31, [x1, x3]
> > ldr q30, [x0, x3]
> > cmhi v30.16b, v30.16b, v31.16b
> > str q30, [x2, x3]
> > add x3, x3, 16
> > cmp x3, x5
> > bne .L5
> > lsl w3, w4, 4
> > cmp w6, w3
> > beq .L1
> > .L4:
> > sub w6, w6, w3
> > ptrue p6.b, all
> > ubfiz x3, x3, 3, 32
> > whilelo p7.b, xzr, x6
> > ld1b z28.b, p7/z, [x0, x3]
> > ld1b z27.b, p7/z, [x1, x3]
> > cmplo p15.b, p6/z, z27.b, z28.b
> > mov z29.b, p15/z, #-1
> > st1b z29.b, p7, [x2, x3]
> > .L1:
> > ret
> > .p2align 2,,3
> > .L6:
> > mov x4, 0
> > .p2align 5,,15
> > .L3:
> > ldr d26, [x0, x4]
> > ldr d25, [x1, x4]
> > cmhi v25.8b, v26.8b, v25.8b
> > str d25, [x2, x4]
> > add x4, x4, 8
> > cmp w3, w4
> > bgt .L3
> > ret
> > .L7:
> > mov w3, 0
> > b .L4
> > ```
> >
> > The vectorizer is now able to handle the conversion between different
> boolean
> > modes while preserving the lane values of the original code.
> >
> > ### Vector Patterns
> >
> > The incoming code may use vector operations to perform masking through
> manually
> > if-converted code. For example:
> >
> > ```c
> > #include <arm_neon.h>
> > #include <arm_sve.h>
> >
> > void foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
> > {
> > for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
> > {
> > uint8x8_t av1 = vld1_u8 (a);
> > uint8x8_t bv1 = vld1_u8 (b);
> > uint8x8_t cmp = vcgt_u8 (bv1, vdup_n_u8 (0));
> > uint8x8_t add = vadd_u8 (bv1, vdup_n_u8 (1));
> > uint8x8_t sel = vbsl_u8 (cmp, add, bv1);
> > vst1_u8 (a, sel);
> > }
> > }
> > ```
> >
> > This is a manually if-converted conditional add. We can re-vectorize it
> > as-is,
> > but what we really want is to optimize it as a masked operation.
> >
> > Since the operations are opaque to the vectorizer, there is a new target
> > hook,
> > `targetm.vectorize.recog_patterns_builtin_call`, and a pattern,
> > `vect_recog_vect_scalar_pattern`, that allow the backend to define which
> > sequence of operations should become a vectorizer pattern.
>
> How much of this can we handle with SLP patterns? I think it would be
> useful that we can recognize target vector builtins as the generic
> per-lane operations they are, so we do not have to dispatch to target
> code all the time?
SLP Patterns would work too, I went with scalar ones to increase the changes
of SLP build itself succeeding. But admittedly this is academic atm. I have no
evidence one way or the other so SLP patterns is a good as any.
But yes, pure blending operations which have been lowered into VEC_PERMs
can be handled entirely as generic SLP patterns.
The hook really exist for the more architecture specific but very common
sequences. For instance we've recommended people for years to use such
sequences
https://github.com/ARM-software/optimized-routines/blob/master/string/aarch64/strcpy.S#L127
when performing index reductions. But these operations don't have a gimple
representation. Like pairwise operations, etc.
So I think if we add new gimple representations for the select operations that
need it for this (atm we already have ADDHN so only PAIRWISE_MAX, PAIRWISE_MIN)
would be needed, we can probably do without the hook.
I can do that for now and see how far we get, and only add the hook if really
needed?
I wasn't sure which you'd prefer because previously both you and Richard S were
against lowering more intrinsics to gimple.
>
> > I have tried to avoid leaking internal vectorizer abstractions here, such as
> > pattern definitions. The hook simply asks for the expected output type and
> the
> > operations that make up the pattern, returning the new operation to put in
> > place.
> >
> > I have not fully worked out this hook yet, or the previous one, so the
> > design
> > is still in flux.
> >
> > With the hook, the example above is vectorized as:
> >
> > ```asm
> > mov z30.b, #1
> > .L4:
> > ld1b z31.b, p7/z, [x1, x2]
> > cmpne p6.b, p7/z, z31.b, #0
> > add z31.b, p6/m, z31.b, z30.b
> > st1b z31.b, p7, [x0, x2]
> > incb x2
> > whilelo p7.b, x2, x3
> > b.any .L4
> > ```
> >
> > There will also be default implementations for patterns that we can detect
> from
> > generic code, such as `VEC_PERM_EXPR`.
> >
> > ### If-Conversion
> >
> > Some vector code manually creates vector control-flow statements using
> > reduction operations:
> >
> > ```c
> > <bool> == vector reduction chain
> > if (<bool>)
> > add ...
> > ```
> >
> > Ideally, we would if-convert these back down using the right values as
> masks.
> > For instance, if the boolean above was created by:
> >
> > ```c
> > a = b `cmp` d
> > e = smaxv (a)
> > if (e != 0)
> > add ...
> > ```
> >
> > we should be able to transform that into:
> >
> > ```c
> > a = b `cmp` d
> > cond_add (a, ...)
> > ```
> >
> > The idea is to use `targetm.vectorize.recog_patterns_builtin_call`, if
> > possible, to do this as well.
>
> Likewise, SLP patterns?
This one is not currently possible, as the branch needs to be eliminated by
ifcvt.
the CFG elimination would need to be done by ifcvt. That said, when writing
this example it seemed very academic to me. The above sequence would really
be more beneficial as a select, it avoids the reduction and branch. So I think
this particular usecase is not important for now. I'm in the process of
analysing a large number of intrinsics workloads and so far this doesn't occur
often.
So I'll push it back down the list for now.
>
> > This is not currently implemented.
> >
> > ### Live Values
> >
> > Scalar live-out values are easy to support, for example for reductions.
> > However, if the live value is a vector, we cannot currently do anything.
>
> True. With loop vect we'd see a scalar lane extract eventually, but only
> outside of the (re-)vectorization scope. With BB vectorization the
> situation might be better, but we'd have to include the lane extract
> as SLP graph entry then.
>
> I'd like to see testcases where it matters.
This happens when we split accumulators, or basic reductions like
#include <arm_neon.h>
uint8_t foo (uint8_t *a, int n)
{
uint8x8_t accum = {0};
for (int i = 0; i < n; i+=8, a+=8)
{
uint8x8_t av1 = vld1_u8 (a);
accum = vadd_u8 (av1, accum);
}
return vaddv_u8 (accum);
}
Where accum is live, we can still transform it but as you say the reduction is
outside of the vectorization scope, but our accumulator splitting code would
generate the same pattern. So my plan was to start a discovery on the live
value and see if all it's uses end in a reduction.
>
> > For example, if the function returns `a` and `a` is an `int32x4_t`, then we
> > are
> > stuck because we are required to return a 128-bit vector without changing
> the
> > order or values of the lanes.
> >
> > In some cases, we can still handle this depending on the operation. For
> > example, if the value is a simple load and accumulate then we can, since
> HVLA
> > adds new instructions, including loads, permutes, stores, and reductions,
> that
> > work on 128-bit element sizes. Effectively, every "element" is an AdvSIMD
> > vector, and there are operations that work on those containers, such as:
> >
> > https://dougallj.github.io/asil/doc/addqv_z_p_z_16.html
> >
> > I am not entirely sure how to model this yet. Richard S chose not to
> introduce
> > new modes here, but instead to represent the "element" and result modes
> using
> > existing modes. For `ADDQV`, this is a reduction from element mode, say
> > `VNx16QI`, into a normal vector mode, also say `VNx16QI`.
> >
> > This representation may introduce ambiguity in GIMPLE, but given that these
> > operations are essentially opaque to GIMPLE, perhaps that is OK. I could
> > use
> a
> > property on the SLP node to distinguish them in the vectorizer.
> >
> > ### Reductions, Including Accumulator Splits
> >
> > Reductions have to make the same distinction as live values between
> reductions
> > to scalars and reductions to vectors, then use the appropriate instructions.
> > When we do accumulator splits, for example, the live reduction on the live
> > value needs to be updated as well.
> >
> > The initial values for reductions also need either to be splats or sequences
> > with a recognizable pattern, so that when we extend the VF we can extend
> the
> > pattern.
> >
> > Alternatively, we must be able to find a neutral value to insert in the new
> > lanes that will not affect the reduction, such as inserting zeros for
> > addition.
>
> A vector reduction inside the loop basically looks like N scalar
> reductions, thus a SLP reduction, only the epilog ties them together.
>
> I'll note that BB vectorization reduction does not chose to represent
> the reduction step but keeps it as "root" stmts in the graph entry.
> A proper SLP representation for a lane reduction would be needed to
> do better I think. Apart from adding "hacks".
Yeah, I know it's been a long standing TODO. Last time I remember someone
trying it wasn't trivial to do since we still had non-SLP.
I guess it could be easier now, which would help with dotprod recognition.
I don't know the scope of this though. I'm also definitely less concerned about
reductions inside the loop, as I doubt anyone writing performant code would do
this, but I'll make a note of this and go look at the previous attempts to add
lane reductions to SLP.
>
> > ### Inductions
> >
> > Inductions have the same issue as reductions and permutes, but I expect
> them to
> > be much less common. Most people still use scalar inductions with their
> vector
> > code. I therefore do not think I will spend much time supporting vector
> > inductions in the first version, unless most of the support falls out
> > naturally
> > from the reduction and permute work.
> >
> > ### Unpacking, Matching `_hi` and `_lo`
> >
> > Transforming from an ISA that has `_hi`/`_lo` unpacking support to one that
> has
> > `_odd`/`_even` would require us to recognize the `_hi`/`_lo` operations and
> > treat them as the same operation. The loop was effectively unrolled for
> > this
> > anyway. This would be similar to `TWO_OPERANDS` nodes, and probably
> just an
> > extension of them.
> >
> > This would allow us to re-synthesize new `_hi`/`_lo` operations when
> increasing
> > the VF. If changing ISA, we would also need to be able to pack the
> > operation
> > again.
> >
> > This is still a TODO. I think I can make it work, but I am not sure
> > whether it
> > belongs in v1.
>
> Definitely not in [1/n] ;) IMO this is sth similar as the SLP permute
> optimization phase, but we'd eliminate branches in the SLP graph by
> merging nodes and permuting only to undo that then in a different way.
>
Yeah, I guess the proper way of doing it is to expose the permutes during
lowering.
> We currently do not have any case where we branch/merge the SLP graph
> and the number of lanes difffer for incoming/outgoing edges btw.,
> there's code that gets confused by this (also see - multiple vector
> sizes in a loop, also see "max_nunits" we track and how we compute
> the unroll factor based on that).
>
Ack.
> >
> > ### Permutes
> >
> > Permutes will be a challenge. Extending an existing short permute to a
> > longer
> > permute, or to VLA, can only happen if the permute represents some kind of
> > regular pattern, such as `REVERSE` or `ZIP`. We will therefore be limited
> > in
> > what we can transform.
> >
> > ### Alias Analysis
> >
> > I think alias analysis should work naturally once I can describe the DR
> > correctly. I have listed it here because I still have a TODO to check this.
>
> When we change the VF in loop vectorization (aka, VF != 1), yes. I think
> this should mostly just work.
>
> > ---
> >
> > Anyway, this is a long email. What do you think, Richi? I still have
> > enough
> > time to pivot. This is 1 of 3 large vectorizer patches that I have planned
> > and
> > am working on for GCC 17.
>
> I say thanks for tackling this. The SLP/DR representation issues
> are the most interesting bits, most of the easy cases should just
> work.
Yeah, it's been "Work" so far but most things just worked naturally.
So I think before I continue further I'll try the representational changes
first.
Thanks for taking a look!
Tamar
>
> Now I'll pivot to the predicated tail series parts that touch SLP.
>
> Richard.
>
> >
> >
> > gcc/ChangeLog:
> >
> > * config/aarch64/aarch64-builtins.cc (aarch64_eq_type_p):
> > (aarch64_lookup_fndecl_by_name):
> > (aarch64_find_builtin_fndecl_1):
> > (aarch64_find_builtin_fndecl):
> > (aarch64_internal_translate_builtin_call):
> > * config/aarch64/aarch64-protos.h:
> > * config/aarch64/aarch64-sve-builtins.cc
> (lookup_registered_function_by_name):
> > (canonicalize_function_name):
> > (lookup_fndecl_by_name):
> > * config/aarch64/aarch64-sve-builtins.h (lookup_fndecl_by_name):
> > * config/aarch64/aarch64-sve2.md (*cond_add_bsl<mode>):
> > * config/aarch64/aarch64.cc
> (aarch64_internal_translate_builtin_call):
> > (aarch64_autovectorize_translate_builtin_call):
> > (TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL):
> > * doc/tm.texi:
> > * doc/tm.texi.in:
> > * target.def:
> > * tree-phinodes.cc (ideal_phi_node_len):
> > * tree-phinodes.h (make_phi_node):
> > * tree-vect-data-refs.cc (vect_analyze_group_access_1):
> > (vect_analyze_data_ref_access):
> > * tree-vect-loop.cc (vectorizable_reduction):
> > * tree-vect-patterns.cc (vect_recog_vect_bool_pattern):
> > (vect_recog_vect_scalar_pattern):
> > * tree-vect-slp.cc (vect_build_slp_tree):
> > (vect_build_slp_tree_2):
> > (vect_build_slp_instance):
> > (vect_analyze_slp_instance):
> > (vect_slp_check_for_roots):
> > (vect_transform_slp_perm_load_1):
> > (vect_schedule_slp):
> > * tree-vect-stmts.cc (get_load_store_type):
> > (vectorizable_operation):
> > (vectorizable_store):
> > (vectorizable_load):
> > (vect_is_simple_cond):
> > (vect_shim_call_arg):
> > (vectorizable_vect_call):
> > (vect_analyze_stmt):
> > (vect_transform_stmt):
> > (get_related_vectype_for_scalar_type):
> > (vect_get_vector_types_for_stmt):
> > (vect_gen_len):
> > * tree-vectorizer.h (enum stmt_vec_info_type):
> >
> > ---
> > diff --git a/gcc/config/aarch64/aarch64-builtins.cc
> b/gcc/config/aarch64/aarch64-builtins.cc
> > index
> b9cc2bf404f612a9688db4ce2f6a042b148ac938..54d7d2e918f613fa449ab
> 268b84581d7ada483f3 100644
> > --- a/gcc/config/aarch64/aarch64-builtins.cc
> > +++ b/gcc/config/aarch64/aarch64-builtins.cc
> > @@ -29,6 +29,7 @@
> > #include "rtl.h"
> > #include "tree.h"
> > #include "gimple.h"
> > +#include "cfgloop.h"
> > #include "ssa.h"
> > #include "memmodel.h"
> > #include "tm_p.h"
> > @@ -42,6 +43,7 @@
> > #include "expr.h"
> > #include "langhooks.h"
> > #include "gimple-iterator.h"
> > +#include "tree-vectorizer.h"
> > #include "case-cfn-macros.h"
> > #include "regs.h"
> > #include "emit-rtl.h"
> > @@ -5591,6 +5593,201 @@ aarch64_general_gimple_fold_builtin
> (unsigned int fcode, gcall *stmt,
> > return new_stmt;
> > }
> >
> > +/* Intrinsics type have a different type decorator than normal types. So
> > + compare signess and mode instead. */
> > +static bool
> > +aarch64_eq_type_p (tree ty0, tree ty1)
> > +{
> > + if (TYPE_UNSIGNED (ty0) != TYPE_UNSIGNED (ty1))
> > + return false;
> > +
> > + return TYPE_MODE (ty0) == TYPE_MODE (ty1);
> > +}
> > +
> > +static tree aarch64_lookup_fndecl_by_name (const char *);
> > +
> > +static tree
> > +aarch64_find_builtin_fndecl_1 (tree fndecl, vec<tree> &types,
> > + tree req_ret_ty,
> > + tree req_arg0_ty, tree req_arg1_ty,
> > + int *maskopno = NULL)
> > +{
> > + tree t = fndecl;
> > + do
> > + {
> > + if (TREE_CODE (t) != FUNCTION_DECL)
> > + break;
> > +
> > + tree fntype = TREE_TYPE (t);
> > + tree ret_ty = TREE_TYPE (fntype);
> > + tree args = TYPE_ARG_TYPES (fntype);
> > +
> > + if (!TREE_CHAIN (args))
> > + break;
> > +
> > + tree arg0_ty = TREE_VALUE (args);
> > + tree arg1_ty = TREE_VALUE (TREE_CHAIN (args));
> > +
> > + if (aarch64_eq_type_p (ret_ty, req_ret_ty)
> > + && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
> > + && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
> > + {
> > + types.create (3);
> > + types.quick_push (ret_ty);
> > + types.quick_push (arg0_ty);
> > + types.quick_push (arg1_ty);
> > + return t;
> > + }
> > +
> > + /* Check if it could be masked, very wrong and unsafe but good
> > enough.
> */
> > + if (maskopno && VECTOR_BOOLEAN_TYPE_P (arg0_ty))
> > + {
> > + *maskopno = 0;
> > + tree arg0_ty = TREE_VALUE (TREE_CHAIN (args));
> > + tree arg1_ty = TREE_VALUE (TREE_CHAIN (TREE_CHAIN (args)));
> > + if (aarch64_eq_type_p (ret_ty, req_ret_ty)
> > + && aarch64_eq_type_p (arg0_ty, req_arg0_ty)
> > + && aarch64_eq_type_p (arg1_ty, req_arg1_ty))
> > + {
> > + types.create (4);
> > + types.quick_push (ret_ty);
> > + types.quick_push (TREE_VALUE (args));
> > + types.quick_push (arg0_ty);
> > + types.quick_push (arg1_ty);
> > + return t;
> > + }
> > + }
> > +
> > + if (t == fndecl)
> > + t = DECL_CHAIN (fndecl);
> > + else
> > + t = DECL_CHAIN (t);
> > + }
> > + while (t);
> > +
> > + return NULL;
> > +}
> > +
> > +static tree
> > +aarch64_find_builtin_fndecl (tree fndecl, tree req_ret_ty,
> > + tree req_arg0_ty, tree req_arg1_ty,
> > + int *maskopno, vec<tree> &types,
> > + const char *fallback_name = nullptr)
> > +{
> > + tree res = aarch64_find_builtin_fndecl_1 (fndecl, types,
> > + req_ret_ty, req_arg0_ty,
> > + req_arg1_ty);
> > + if (res)
> > + return res;
> > +
> > + if (fallback_name)
> > + {
> > + tree t = aarch64_lookup_fndecl_by_name (fallback_name);
> > + gcc_assert (t);
> > + *maskopno = -1;
> > + res = aarch64_find_builtin_fndecl_1 (t, types, req_ret_ty,
> > req_arg0_ty,
> > + req_arg1_ty, maskopno);
> > + return res;
> > + }
> > +
> > + return NULL_TREE;
> > +}
> > +
> > +static tree
> > +aarch64_lookup_fndecl_by_name (const char *name)
> > +{
> > + tree id = get_identifier (name);
> > + if (!id)
> > + return NULL_TREE;
> > +
> > + extern tree lookup_name (tree);
> > + tree decl = lookup_name (id);
> > + if (!decl || TREE_CODE (decl) != FUNCTION_DECL)
> > + return aarch64_sve::lookup_fndecl_by_name (name);
> > +
> > + return decl;
> > +}
> > +
> > +/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL. */
> > +gcall *
> > +aarch64_internal_translate_builtin_call (vec_info *vinfo,
> > + slp_tree node,
> > + stmt_vec_info stmt_info,
> > + int *maskopno,
> > + vec<tree> &types,
> > + tree *vectype_out)
> > +{
> > + gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
> > + gcc_assert (stmt);
> > + tree fndecl = gimple_call_fndecl (stmt);
> > + unsigned int code = DECL_MD_FUNCTION_CODE (fndecl);
> > + unsigned int subcode = code >> AARCH64_BUILTIN_SHIFT;
> > +
> > + /* There's nothing to do for SVE at this point in time, so bail out. */
> > + if ((code & AARCH64_BUILTIN_CLASS) == AARCH64_BUILTIN_SVE)
> > + return NULL;
> > +
> > + switch (subcode)
> > + {
> > + BUILTIN_VSDQ_I (BINOP_SSU, suqadd, 0, DEFAULT)
> > + BUILTIN_VSDQ_I (BINOP_UUS, usqadd, 0, DEFAULT)
> > + {
> > + tree arg0 = gimple_call_arg (stmt, 0);
> > + tree arg1 = gimple_call_arg (stmt, 1);
> > + tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
> > + tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
> > + if (!ty0 || !ty1)
> > + return NULL;
> > +
> > + /* For usqadd/suqadd the result type is the vectorized type of arg0.
> */
> > + tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty0, ty0, ty1,
> > + maskopno, types,
> > + "svsqadd_u8_x");
> > + if (!new_fndecl)
> > + return NULL;
> > +
> > + /* This is type incorrect, but we don't have the arguments yet. The
> > + vectorizer will take care of it. */
> > + gcall *new_call;
> > + if (*maskopno == -1)
> > + new_call = gimple_build_call (new_fndecl, 2, NULL, NULL);
> > + else
> > + new_call = gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
> > + *vectype_out = ty0;
> > + return new_call;
> > + }
> > + BUILTIN_VDQQH (BSL_P, simd_bsl, 0, DEFAULT)
> > + VAR2 (BSL_P, simd_bsl,0, DEFAULT, di, v2di)
> > + BUILTIN_VSDQ_I_DI (BSL_U, simd_bsl, 0, DEFAULT)
> > + BUILTIN_VALLDIF (BSL_S, simd_bsl, 0, QUIET)
> > + {
> > + tree arg0 = gimple_call_arg (stmt, 0);
> > + tree arg1 = gimple_call_arg (stmt, 1);
> > + tree ty0 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg0));
> > + tree ty1 = get_vectype_for_scalar_type (vinfo, TREE_TYPE (arg1));
> > + if (!ty0 || !ty1)
> > + return NULL;
> > +
> > + /* For usqadd/suqadd the result type is the vectorized type of arg0.
> */
> > + tree new_fndecl = aarch64_find_builtin_fndecl (fndecl, ty1, ty0, ty1,
> > + maskopno, types,
> > + "svbsl_u8");
> > + if (!new_fndecl)
> > + return NULL;
> > +
> > + types.safe_push (types[1]);
> > + /* BSL does not support masks. */
> > + gcc_assert (*maskopno == -1);
> > + *vectype_out = ty0;
> > + return gimple_build_call (new_fndecl, 3, NULL, NULL, NULL);
> > + }
> > + default:
> > + return NULL;
> > + }
> > +
> > + return NULL;
> > +}
> > +
> > void
> > aarch64_atomic_assign_expand_fenv (tree *hold, tree *clear, tree *update)
> > {
> > diff --git a/gcc/config/aarch64/aarch64-protos.h
> b/gcc/config/aarch64/aarch64-protos.h
> > index
> 513b556398fac7c733de47c27587c3aaa07cbcf3..fb7062b3d75902e1298b9
> d64846daf69b418fbb7 100644
> > --- a/gcc/config/aarch64/aarch64-protos.h
> > +++ b/gcc/config/aarch64/aarch64-protos.h
> > @@ -1185,6 +1185,7 @@ namespace aarch64_sve {
> > void handle_arm_sme_h (bool);
> > void handle_arm_neon_sve_bridge_h (bool);
> > tree builtin_decl (unsigned, bool);
> > + tree lookup_fndecl_by_name (const char *);
> > bool builtin_type_p (const_tree);
> > bool builtin_type_p (const_tree, unsigned int *, unsigned int *);
> > const char *mangle_builtin_type (const_tree);
> > diff --git a/gcc/config/aarch64/aarch64-sve-builtins.cc
> b/gcc/config/aarch64/aarch64-sve-builtins.cc
> > index
> cd78713440f61ac18146f0be49a43a204dbb3ae1..fe5e267f087e1b2b62885
> 695785e9065bee4d85a 100644
> > --- a/gcc/config/aarch64/aarch64-sve-builtins.cc
> > +++ b/gcc/config/aarch64/aarch64-sve-builtins.cc
> > @@ -1063,6 +1063,45 @@ static GTY(())
> hash_table<registered_function_hasher> *function_table;
> > are IDENTIFIER_NODEs. */
> > static GTY(()) hash_map<tree, registered_function *> *overload_names[2];
> >
> > +/* Return the decl of the registered SVE function called NAME, or null if
> > + no such function has been registered. */
> > +static tree
> > +lookup_registered_function_by_name (const char *name)
> > +{
> > + unsigned int length = vec_safe_length (registered_functions);
> > + for (unsigned int i = 0; i < length; ++i)
> > + {
> > + registered_function *rfn = (*registered_functions)[i];
> > + if (!rfn || TREE_CODE (rfn->decl) != FUNCTION_DECL)
> > + continue;
> > +
> > + tree decl_name = DECL_NAME (rfn->decl);
> > + if (decl_name && strcmp (IDENTIFIER_POINTER (decl_name), name) ==
> 0)
> > + return rfn->decl;
> > + }
> > +
> > + return NULL_TREE;
> > +}
> > +
> > +/* Return the ACLE spelling that corresponds to NAME. */
> > +static const char *
> > +canonicalize_function_name (const char *name)
> > +{
> > + if (startswith (name, "__builtin_aarch64_"))
> > + return name + strlen ("__builtin_aarch64_");
> > +
> > + if (startswith (name, "__builtin_sve_"))
> > + return name + strlen ("__builtin_sve_");
> > +
> > + if (startswith (name, "__builtin_")
> > + && (startswith (name + strlen ("__builtin_"), "sv")
> > + || startswith (name + strlen ("__builtin_"), "arm_")
> > + || startswith (name + strlen ("__builtin_"), "__arm_")))
> > + return name + strlen ("__builtin_");
> > +
> > + return name;
> > +}
> > +
> > /* Record that TYPE is an ABI-defined SVE type that contains NUM_ZR SVE
> vectors
> > and NUM_PR SVE predicates. MANGLED_NAME, if nonnull, is the ABI-
> defined
> > mangling of the type. mangling of the type. ACLE_NAME is the
> <arm_sve.h>
> > @@ -4939,6 +4978,29 @@ builtin_decl (unsigned int code, bool)
> > return (*registered_functions)[code]->decl;
> > }
> >
> > +/* Return the decl for the SVE builtin called NAME, if one exists. Accept
> > + both the ACLE spelling and the common builtin-prefixed aliases. */
> > +tree
> > +lookup_fndecl_by_name (const char *name)
> > +{
> > + const char *canonical_name = canonicalize_function_name (name);
> > + tree decl = lookup_registered_function_by_name (canonical_name);
> > + if (decl)
> > + return decl;
> > +
> > + /* Internal callers sometimes need SVE decls even when arm_sve.h was
> not
> > + included in the source. Materialize them lazily in that case. */
> > + if (!function_table && (lang_GNU_C () || lang_GNU_CXX ()))
> > + {
> > + handle_arm_sve_h (false);
> > + decl = lookup_registered_function_by_name (canonical_name);
> > + if (decl)
> > + return decl;
> > + }
> > +
> > + return NULL_TREE;
> > +}
> > +
> > /* Implement #pragma GCC aarch64 "arm_sme.h". */
> > void
> > handle_arm_sme_h (bool function_nulls_p)
> > diff --git a/gcc/config/aarch64/aarch64-sve-builtins.h
> b/gcc/config/aarch64/aarch64-sve-builtins.h
> > index
> f9e8c4cd7293a379e3334fecb846121cf88e0ebf..d4123dbe8b3c0a60f1b1a6
> b1a398d05989987c23 100644
> > --- a/gcc/config/aarch64/aarch64-sve-builtins.h
> > +++ b/gcc/config/aarch64/aarch64-sve-builtins.h
> > @@ -856,6 +856,7 @@ bool vector_cst_all_same (tree, unsigned int);
> > bool is_ptrue (tree, unsigned int);
> > bool is_pfalse (tree);
> > const function_instance *lookup_fndecl (tree);
> > +tree lookup_fndecl_by_name (const char *);
> >
> > /* Try to find a mode with the given mode_suffix_info fields. Return the
> > mode on success or MODE_none on failure. */
> > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > index
> 5a859e12b1a55e70284469e3e9c175eee849e18d..a9f3af387ddce6b86370
> 992a4dc9717df5f7e82c 100644
> > --- a/gcc/config/aarch64/aarch64.cc
> > +++ b/gcc/config/aarch64/aarch64.cc
> > @@ -103,6 +103,7 @@
> > #include "aarch64-sched-dispatch.h"
> > #include "aarch64-json-tunings-printer.h"
> > #include "aarch64-json-tunings-parser.h"
> > +#include "aarch64-builtins.h"
> >
> > /* This file should be included last. */
> > #include "target-def.h"
> > @@ -23868,6 +23869,24 @@ aarch64_autovectorize_vector_modes
> (vector_modes *modes, bool)
> > return flags;
> > }
> >
> > +/* This is just temporary as I don't have an entry point into the new
> intrinsics
> > + framework yet. */
> > +extern gcall *aarch64_internal_translate_builtin_call (vec_info *,
> > slp_tree,
> > + stmt_vec_info, int*,
> > + vec<tree> &types, tree *);;
> > +
> > +/* Implement TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL. */
> > +static gcall *
> > +aarch64_autovectorize_translate_builtin_call (vec_info *vinfo,
> > + slp_tree node,
> > + stmt_vec_info stmt_info,
> > + int *maskopno, vec<tree> &types,
> > + tree *vectype_out)
> > +{
> > + return aarch64_internal_translate_builtin_call (vinfo, node, stmt_info,
> > + maskopno, types,
> vectype_out);
> > +}
> > +
> > /* Implement TARGET_CONVERT_TO_TYPE. Convert EXPR to TYPE. */
> >
> > static tree
> > @@ -34098,6 +34117,10 @@ aarch64_libgcc_floating_mode_supported_p
> > #define TARGET_VECTORIZE_AUTOVECTORIZE_VECTOR_MODES \
> > aarch64_autovectorize_vector_modes
> >
> > +#undef TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
> > +#define TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL \
> > + aarch64_autovectorize_translate_builtin_call
> > +
> > #undef TARGET_CONVERT_TO_TYPE
> > #define TARGET_CONVERT_TO_TYPE aarch64_convert_to_type
> >
> > diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
> > index
> 1fbf43c6c57799372752f581528e122689a33f15..4f48737bb916fefd51dd7
> 38a235679af1b4f693c 100644
> > --- a/gcc/doc/tm.texi
> > +++ b/gcc/doc/tm.texi
> > @@ -6537,6 +6537,20 @@ current cost model is for the scalar version of a
> loop or block; otherwise
> > it is for the vector version.
> > @end deftypefn
> >
> > +@deftypefn {Target Hook} {gcall *}
> TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL (vec_info *@var{vinfo}, struct
> _slp_tree *@var{node}, class _stmt_vec_info *@var{stmt_info}, int
> *@var{maskopno}, vec<tree> @var{&types}, tree *@var{vectype_out})
> > +This hook returns true if masked operation @var{ifn} (really of
> > +type @code{internal_fn}) should be considered more expensive to use than
> > +implementing the same operation without masking. GCC can then try to
> use
> > +unconditional operations instead with extra selects.
> > +@end deftypefn
> > +
> > +@deftypefn {Target Hook} {gimple *}
> TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL (vec_info
> *@var{vinfo}, struct _slp_tree *@var{node}, class _stmt_vec_info
> *@var{stmt_info}, vec<gimple> @var{&chain}, tree *@var{vectype_out})
> > +This hook returns true if masked operation @var{ifn} (really of
> > +type @code{internal_fn}) should be considered more expensive to use than
> > +implementing the same operation without masking. GCC can then try to
> use
> > +unconditional operations instead with extra selects.
> > +@end deftypefn
> > +
> > @deftypefn {Target Hook} tree TARGET_VECTORIZE_BUILTIN_GATHER
> (const_tree @var{mem_vectype}, const_tree @var{index_type}, int
> @var{scale})
> > Target builtin that implements vector gather operation.
> @var{mem_vectype}
> > is the vector type of the load and @var{index_type} is scalar type of
> > diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
> > index
> 6ecbeda09ca9b6473f12f78f8805ced44907f146..bc6d60d2ec47e62a31596
> bd96f3d106ae65254fb 100644
> > --- a/gcc/doc/tm.texi.in
> > +++ b/gcc/doc/tm.texi.in
> > @@ -4341,6 +4341,10 @@ address; but often a machine-dependent
> strategy can generate better code.
> >
> > @hook TARGET_VECTORIZE_CREATE_COSTS
> >
> > +@hook TARGET_VECTORIZE_TRANSLATE_BUILTIN_CALL
> > +
> > +@hook TARGET_VECTORIZE_RECOG_PATTERNS_BUILTIN_CALL
> > +
> > @hook TARGET_VECTORIZE_BUILTIN_GATHER
> >
> > @hook TARGET_VECTORIZE_BUILTIN_SCATTER
> > diff --git a/gcc/target.def b/gcc/target.def
> > index
> 25a94559e198aef2c3e842f02aa5414c5193c4a6..3c9a2aba29691b80ab502
> b057e9e7244e56758a4 100644
> > --- a/gcc/target.def
> > +++ b/gcc/target.def
> > @@ -2098,6 +2098,33 @@ stores.",
> > (const_tree vectype, const_tree index_type, int scale),
> > NULL)
> >
> > +/* Function to say how to transform a gimple call. If transformation is
> > + possible return the new gcall, otherwise NULL. */
> > +DEFHOOK
> > +(translate_builtin_call,
> > + "This hook returns true if masked operation @var{ifn} (really of\n\
> > +type @code{internal_fn}) should be considered more expensive to use
> than\n\
> > +implementing the same operation without masking. GCC can then try to
> use\n\
> > +unconditional operations instead with extra selects.",
> > + gcall *,
> > + (vec_info *vinfo, struct _slp_tree *node, class _stmt_vec_info *stmt_info,
> > + int *maskopno, vec<tree> &types, tree *vectype_out),
> > + NULL)
> > +
> > +/* Function to recognize patterns during vectorization of gimple calls.
> > If a
> > + pattern is matched then return the new pattern and other statements it
> > + consumes, otherwise NULL. */
> > +DEFHOOK
> > +(recog_patterns_builtin_call,
> > + "This hook returns true if masked operation @var{ifn} (really of\n\
> > +type @code{internal_fn}) should be considered more expensive to use
> than\n\
> > +implementing the same operation without masking. GCC can then try to
> use\n\
> > +unconditional operations instead with extra selects.",
> > + gimple *,
> > + (vec_info *vinfo, class _stmt_vec_info *stmt_info,
> > + vec<gimple *> &chain, tree *vectype_out),
> > + NULL)
> > +
> > /* Target function to initialize the cost model for a loop or block. */
> > DEFHOOK
> > (create_costs,
> > diff --git a/gcc/tree-phinodes.cc b/gcc/tree-phinodes.cc
> > index
> e0663d96b7d4b96a1a48e5bfb9fef1329e526ae7..d97c6148d2f303bf9280d
> bbf19a3610133d36f8c 100644
> > --- a/gcc/tree-phinodes.cc
> > +++ b/gcc/tree-phinodes.cc
> > @@ -165,7 +165,7 @@ ideal_phi_node_len (int len)
> >
> > /* Return a PHI node with LEN argument slots for variable VAR. */
> >
> > -static gphi *
> > +gphi *
> > make_phi_node (tree var, int len)
> > {
> > gphi *phi;
> > diff --git a/gcc/tree-phinodes.h b/gcc/tree-phinodes.h
> > index
> b6fa492cb248f4d0e1a7838d0b34da6ae3a108fd..08741bb4076d94594541
> 20ead335e52e49953352 100644
> > --- a/gcc/tree-phinodes.h
> > +++ b/gcc/tree-phinodes.h
> > @@ -23,6 +23,7 @@ along with GCC; see the file COPYING3. If not see
> > extern void phinodes_print_statistics (void);
> > extern void reserve_phi_args_for_new_edge (basic_block);
> > extern gphi *create_phi_node (tree, basic_block);
> > +gphi *make_phi_node (tree, int);
> > extern void add_phi_arg (gphi *, tree, edge, location_t);
> > extern void remove_phi_args (edge);
> > extern void remove_phi_node (gimple_stmt_iterator *, bool);
> > diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
> > index
> 0e57e1068d6a1906d1d8e9edb761d5dd4b7da758..eefd5ad2f91e0292cda5
> 2354bf70b406a62adbcf 100644
> > --- a/gcc/tree-vect-data-refs.cc
> > +++ b/gcc/tree-vect-data-refs.cc
> > @@ -3176,6 +3176,17 @@ vect_analyze_group_access_1 (vec_info *vinfo,
> dr_vec_info *dr_info)
> > data_reference *dr = dr_info->dr;
> > tree step = DR_STEP (dr);
> > tree scalar_type = TREE_TYPE (DR_REF (dr));
> > + unsigned long nunits = 0;
> > + /* TODO: Add? */
> > + if (VECTOR_TYPE_P (scalar_type)
> > + && !TYPE_VECTOR_SUBPARTS (scalar_type).is_constant (&nunits))
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_NOTE, vect_location,
> > + "Variable length vectors not supported yet: %T\n",
> > + step);
> > + return false;
> > + }
> > HOST_WIDE_INT type_size = TREE_INT_CST_LOW (TYPE_SIZE_UNIT
> (scalar_type));
> > stmt_vec_info stmt_info = dr_info->stmt;
> > loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
> > @@ -3207,6 +3218,8 @@ vect_analyze_group_access_1 (vec_info *vinfo,
> dr_vec_info *dr_info)
> > }
> > groupsize = absu_hwi (dr_step) / type_size;
> > }
> > + else if (bb_vinfo && nunits > 0)
> > + groupsize = nunits;
> > else
> > groupsize = 0;
> >
> > @@ -3463,6 +3476,29 @@ vect_analyze_data_ref_access (vec_info *vinfo,
> dr_vec_info *dr_info)
> > }
> > }
> >
> > + /* Allow loads and stores with zero steps and vector types in BB
> > + vectorization. TODO: This needs support for checking the type of
> > access
> > + so e.g. LOAD_LANES, Gathers etc. */
> > + if (!loop_vinfo && integer_zerop (step) && VECTOR_TYPE_P (scalar_type))
> > + {
> > + if (!TYPE_VECTOR_SUBPARTS (scalar_type).is_constant ())
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_NOTE, vect_location,
> > + "Variable length vectors not supported yet: %T\n",
> > + step);
> > + return false;
> > + }
> > + DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
> > + DR_GROUP_NEXT_ELEMENT (stmt_info) = NULL;
> > +
> > + if (dump_enabled_p () && 0)
> > + dump_printf_loc (MSG_NOTE, vect_location,
> > + "Allowing vector memory access in BB\n");
> > +
> > + return true;
> > + }
> > +
> > if (loop && nested_in_vect_loop_p (loop, stmt_info))
> > {
> > /* Interleaved accesses are not yet supported within outer-loop
> > @@ -3485,9 +3521,21 @@ vect_analyze_data_ref_access (vec_info *vinfo,
> dr_vec_info *dr_info)
> > if (TREE_CODE (step) == INTEGER_CST)
> > {
> > HOST_WIDE_INT dr_step = TREE_INT_CST_LOW (step);
> > - if (!tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
> > - || (dr_step < 0
> > - && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step)))
> > + bool sequantial = false;
> > + poly_int64 step_p, type_size_p;
> > + if (poly_int_tree_p (step, &step_p)
> > + && poly_int_tree_p (TYPE_SIZE_UNIT (scalar_type), &type_size_p))
> > + sequantial
> > + = known_eq (step_p, type_size_p)
> > + || (known_lt (step_p, 0)
> > + && !known_eq (-step_p, type_size_p));
> > + else
> > + sequantial
> > + = !tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
> > + || (dr_step < 0
> > + && !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -
> dr_step));
> > +
> > + if (sequantial)
> > {
> > /* Mark that it is not interleaving. */
> > DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
> > diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> > index
> 51e7f05100de7ab31994f3b4a30f5c5be8d8afdd..7dee95951eef95622c955
> 4c1bbe01c3e02cd7894 100644
> > --- a/gcc/tree-vect-loop.cc
> > +++ b/gcc/tree-vect-loop.cc
> > @@ -7184,12 +7184,17 @@ vectorizable_reduction (loop_vec_info
> loop_vinfo,
> > gcc_unreachable ();
> > bool lane_reducing = lane_reducing_op_p (op.code);
> >
> > - if (!POINTER_TYPE_P (op.type) && !INTEGRAL_TYPE_P (op.type)
> > - && !SCALAR_FLOAT_TYPE_P (op.type))
> > + tree scalar_type = op.type;
> > + if (VECTOR_TYPE_P (scalar_type))
> > + scalar_type = TREE_TYPE (scalar_type);
> > +
> > + if (!POINTER_TYPE_P (scalar_type)
> > + && !INTEGRAL_TYPE_P (scalar_type)
> > + && !SCALAR_FLOAT_TYPE_P (scalar_type))
> > return false;
> >
> > /* Do not try to vectorize bit-precision reductions. */
> > - if (!type_has_mode_precision_p (op.type)
> > + if (!type_has_mode_precision_p (scalar_type)
> > && op.code != BIT_AND_EXPR
> > && op.code != BIT_IOR_EXPR
> > && op.code != BIT_XOR_EXPR)
> > @@ -7444,7 +7449,7 @@ vectorizable_reduction (loop_vec_info
> loop_vinfo,
> > if (!reduc_chain
> > && known_eq (LOOP_VINFO_VECT_FACTOR (loop_vinfo), 1u))
> > ;
> > - else if (needs_fold_left_reduction_p (op.type, orig_code))
> > + else if (needs_fold_left_reduction_p (scalar_type, orig_code))
> > {
> > /* When vectorizing a reduction chain w/o SLP the reduction PHI
> > is not directly used in stmt. */
> > @@ -7573,7 +7578,7 @@ vectorizable_reduction (loop_vec_info
> loop_vinfo,
> > else if (reduction_type == COND_REDUCTION)
> > {
> > int scalar_precision
> > - = GET_MODE_PRECISION (SCALAR_TYPE_MODE (op.type));
> > + = GET_MODE_PRECISION (SCALAR_TYPE_MODE (scalar_type));
> > cr_index_scalar_type = make_unsigned_type (scalar_precision);
> > cr_index_vector_type = get_same_sized_vectype (cr_index_scalar_type,
> > vectype_out);
> > diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
> > index
> 5f5e3fb6201da379460c46e9d7ff1d01625d0998..ca806f7d8d94333fc4301
> b8a8e0da5ad0a4fad0b 100644
> > --- a/gcc/tree-vect-patterns.cc
> > +++ b/gcc/tree-vect-patterns.cc
> > @@ -6243,6 +6243,140 @@ vect_recog_bool_pattern (vec_info *vinfo,
> > return NULL;
> > }
> >
> > +static gimple *
> > +vect_recog_vect_bool_pattern (vec_info *vinfo,
> > + stmt_vec_info stmt_vinfo, tree *type_out)
> > +{
> > + gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
> > + enum tree_code rhs_code;
> > + tree var, lhs, rhs, vectype;
> > + gimple *pattern_stmt;
> > +
> > + if (!is_gimple_assign (last_stmt))
> > + return NULL;
> > +
> > + var = gimple_assign_rhs1 (last_stmt);
> > + lhs = gimple_assign_lhs (last_stmt);
> > + rhs_code = gimple_assign_rhs_code (last_stmt);
> > +
> > + if (rhs_code == VIEW_CONVERT_EXPR)
> > + var = TREE_OPERAND (var, 0);
> > +
> > + if (!VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (var)))
> > + return NULL;
> > +
> > + if (rhs_code != VEC_COND_EXPR)
> > + return NULL;
> > +
> > + tree op0 = gimple_assign_rhs2 (last_stmt);
> > + tree op1 = gimple_assign_rhs3 (last_stmt);
> > + if (!integer_zerop (op1)
> > + || !integer_all_onesp (op0))
> > + return NULL;
> > +
> > + /* Find the comparison and make it part of the pattern we'll make. */
> > + auto cmp_info = vinfo->lookup_def (var);
> > + if (!cmp_info)
> > + return NULL;
> > +
> > + gimple *cmp_stmt = STMT_VINFO_STMT (cmp_info);
> > + if (!is_gimple_assign (cmp_stmt)
> > + || TREE_CODE_CLASS (gimple_assign_rhs_code (cmp_stmt)) !=
> tcc_comparison)
> > + return NULL;
> > +
> > + vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (lhs));
> > + if (vectype == NULL_TREE)
> > + return NULL;
> > +
> > + /* Build a scalar type for the boolean result that when
> > + vectorized matches the vector type of the result in
> > + size and number of elements. */
> > + unsigned prec
> > + = vector_element_size (tree_to_poly_uint64 (TYPE_SIZE (vectype)),
> > + TYPE_VECTOR_SUBPARTS (vectype));
> > +
> > + tree type
> > + = build_nonstandard_integer_type (prec, TYPE_UNSIGNED (TREE_TYPE
> (var)));
> > +
> > + tree new_vectype = get_mask_type_for_scalar_type (vinfo, type);
> > + if (new_vectype == NULL_TREE)
> > + return NULL;
> > +
> > + /* copy the scalar statement to get a new UID. */
> > + tree lhs_cmp
> > + = vect_recog_temp_ssa_var (TREE_TYPE (gimple_get_lhs (cmp_stmt)),
> NULL);
> > + gimple *copy_stmt
> > + = gimple_build_assign (lhs_cmp, gimple_assign_rhs_code (cmp_stmt),
> > + gimple_assign_rhs1 (cmp_stmt),
> > + gimple_assign_rhs2 (cmp_stmt));
> > + append_pattern_def_seq (vinfo, stmt_vinfo, copy_stmt, new_vectype,
> type);
> > +
> > + tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
> > + pattern_stmt = gimple_build_assign (lhs_ivar, COND_EXPR, lhs_cmp,
> > + build_all_ones_cst (type),
> > + build_zero_cst (type));
> > +
> > + *type_out = vectype;
> > + vect_pattern_detected ("vect_recog_vect_bool_pattern", last_stmt);
> > +
> > + return pattern_stmt;
> > +}
> > +
> > +static gimple *
> > +vect_recog_vect_scalar_pattern (vec_info *vinfo,
> > + stmt_vec_info stmt_vinfo, tree *type_out)
> > +{
> > + gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
> > + enum tree_code rhs_code;
> > + tree var, lhs, rhs, vectype;
> > + gimple *pattern_stmt;
> > + loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
> > + class loop *loop = loop_vinfo ? LOOP_VINFO_LOOP (loop_vinfo) : NULL;
> > +
> > +
> > + lhs = gimple_get_lhs (last_stmt);
> > + if (!lhs || !VECTOR_TYPE_P (TREE_TYPE (lhs)))
> > + return NULL;
> > +
> > + if (targetm.vectorize.recog_patterns_builtin_call)
> > + {
> > + auto_vec<gimple *> chain;
> > + gimple *res
> > + = targetm.vectorize.recog_patterns_builtin_call (vinfo, stmt_vinfo,
> > + chain, type_out);
> > + vect_pattern_detected ("vect_recog_vect_scalar_pattern", last_stmt);
> > + }
> > +
> > +#if 1
> > + /* First pattern is reductions, if we have a vector reduction then adjust
> > + any constant lanes if we can. TOOD: Maybe this should force HVLA? */
> > + if (loop && is_a <gphi *> (last_stmt))
> > + {
> > + /* Check the header edge. */
> > + gphi *phi = as_a <gphi *>(last_stmt);
> > + tree init = PHI_ARG_DEF_FROM_EDGE (phi, loop_preheader_edge
> (loop));
> > + if ((var = ssa_uniform_vector_p (init)))
> > + {
> > + tree type = TREE_TYPE (lhs);
> > + vectype = get_vectype_for_scalar_type (vinfo, type);
> > + if (vectype == NULL_TREE)
> > + return NULL;
> > +
> > + tree lhs_ivar = vect_recog_temp_ssa_var (type, NULL);
> > + gphi *res = make_phi_node (lhs_ivar, 2);
> > + tree step = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
> > + add_phi_arg (res, init, loop_preheader_edge (loop),
> UNKNOWN_LOCATION);
> > + add_phi_arg (res, step, loop_latch_edge (loop),
> UNKNOWN_LOCATION);
> > + pattern_stmt = res;
> > +
> > + *type_out = vectype;
> > + vect_pattern_detected ("vect_recog_vect_scalar_pattern",
> last_stmt);
> > + }
> > + }
> > +#endif
> > +
> > + return NULL;
> > +}
> >
> > /* Function vect_recog_mask_conversion_pattern
> >
> > @@ -7467,6 +7601,8 @@ static vect_recog_func
> vect_vect_recog_func_ptrs[] = {
> > { vect_recog_sat_trunc_pattern, "sat_trunc" },
> > { vect_recog_gcond_pattern, "gcond" },
> > { vect_recog_bool_pattern, "bool" },
> > + { vect_recog_vect_bool_pattern, "vect_bool" },
> > + { vect_recog_vect_scalar_pattern, "vect_scalar" },
> > /* This must come before mask conversion, and includes the parts
> > of mask conversion that are needed for gather and scatter
> > internal functions. */
> > diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
> > index
> 075e93f04a97cddd6bc4b7f82485f08ee22c953b..c1bf905f54786c3622a15
> 5f777e6889a635506d4 100644
> > --- a/gcc/tree-vect-slp.cc
> > +++ b/gcc/tree-vect-slp.cc
> > @@ -1950,7 +1950,7 @@ vect_build_slp_tree (vec_info *vinfo,
> > for (i = 0; i < group_size; ++i)
> > if (!matches[i])
> > break;
> > - gcc_assert (i < group_size);
> > + //gcc_assert (i < group_size);
> > }
> > memcpy (res->failed, matches, sizeof (bool) * group_size);
> > }
> > @@ -2107,6 +2107,9 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree
> node,
> > &vectype))
> > return NULL;
> >
> > + if (known_gt (group_size, this_max_nunits))
> > + return NULL;
> > +
> > /* If the SLP node is a load, terminate the recursion unless masked. */
> > if (STMT_VINFO_DATA_REF (stmt_info)
> > && DR_IS_READ (STMT_VINFO_DATA_REF (stmt_info)))
> > @@ -2133,6 +2136,7 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree
> node,
> > FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_STMTS (node), j, load_info)
> > {
> > int load_place;
> > + tree type;
> > if (! load_info)
> > {
> > if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
> > @@ -2140,6 +2144,11 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree
> node,
> > else
> > load_place = 0;
> > }
> > + /* ?? Not all vector loads are linear, model this. */
> > + else if ((type = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT
> (load_info))))
> > + && VECTOR_TYPE_P (type)
> > + && TYPE_VECTOR_SUBPARTS (type).is_constant ())
> > + load_place = j;
> > else if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
> > load_place = vect_get_place_in_interleaving_chain
> > (load_info, first_stmt_info);
> > @@ -3141,7 +3150,7 @@ fail:
> > n_vector_builds++;
> > }
> > }
> > - if (all_uniform_p
> > + if (all_uniform_p && false
> > || n_vector_builds > 1
> > || (n_vector_builds == children.length ()
> > && is_a <gphi *> (stmt_info->stmt)))
> > @@ -4282,7 +4291,9 @@ vect_build_slp_instance (vec_info *vinfo,
> > /* While we arrive here even with slp_inst_kind_store we should only
> > for group_size == 1. The code to split store groups is only in
> > vect_analyze_slp_instance now. */
> > +#if 0
> > gcc_assert (kind != slp_inst_kind_store || group_size == 1);
> > +#endif
> >
> > /* Free the allocated memory. */
> > scalar_stmts.release ();
> > @@ -5014,7 +5025,16 @@ vect_analyze_slp_instance (vec_info *vinfo,
> > stmt_vec_info next_info = stmt_info;
> > while (next_info)
> > {
> > - scalar_stmts.quick_push (vect_stmt_to_vectorize (next_info));
> > + scalar_stmts.safe_push (vect_stmt_to_vectorize (next_info));
> > +#if 1
> > + /* If we're a vector type, insert dummy nodes. */
> > + unsigned HOST_WIDE_INT vect_nunits = 0;
> > + tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT
> (next_info)));
> > + if (VECTOR_TYPE_P (vectype)
> > + && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
> > + for (auto i = 0; i < vect_nunits - 1UL; i++)
> > + scalar_stmts.safe_push (next_info);
> > +#endif
> > next_info = DR_GROUP_NEXT_ELEMENT (next_info);
> > }
> >
> > @@ -5038,6 +5058,13 @@ vect_analyze_slp_instance (vec_info *vinfo,
> >
> > /* Build the tree for the SLP instance. */
> > unsigned int group_size = scalar_stmts.length ();
> > +#if 0
> > + /* If we're a vector type, adjust group size by nunits. */
> > + unsigned HOST_WIDE_INT vect_nunits = 0;
> > + tree vectype = TREE_TYPE (gimple_get_lhs (STMT_VINFO_STMT
> (scalar_stmts[0])));
> > + if (TYPE_VECTOR_SUBPARTS (vectype).is_constant (&vect_nunits))
> > + group_size *= vect_nunits;
> > +#endif
> > bool *matches = XALLOCAVEC (bool, group_size);
> > poly_uint64 max_nunits = 1;
> > unsigned tree_size = 0;
> > @@ -9780,6 +9807,7 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
> > enum tree_code code = gimple_assign_rhs_code (assign);
> > use_operand_p use_p;
> > gimple *use_stmt;
> > + optab op;
> > if (code == CONSTRUCTOR)
> > {
> > if (!VECTOR_TYPE_P (TREE_TYPE (rhs))
> > @@ -9995,6 +10023,24 @@ vect_slp_check_for_roots (bb_vec_info
> bb_vinfo)
> > }
> > }
> > }
> > +#if 1
> > + else if (gimple_vdef (assign)
> > + && VECTOR_TYPE_P (TREE_TYPE (rhs))
> > + && ((op = optab_for_tree_code (code, TREE_TYPE (rhs),
> optab_default)) == unknown_optab
> > + || !can_implement_p (op, TYPE_MODE (TREE_TYPE (rhs)))))
> > + {
> > + vec<stmt_vec_info> roots = vNULL;
> > + auto stmt_vinfo = bb_vinfo->lookup_stmt (assign);
> > + roots.safe_push (stmt_vinfo);
> > + vec<stmt_vec_info> stmts;
> > + auto num_entries = TYPE_VECTOR_SUBPARTS (TREE_TYPE
> (rhs)).to_constant () / 2;
> > + stmts.create (num_entries);
> > + for (int i = 0; i < num_entries; i++)
> > + stmts.quick_push (stmt_vinfo);
> > + bb_vinfo->roots.safe_push (slp_root (slp_inst_kind_store,
> > + stmts, roots));
> > + }
> > +#endif
> > }
> > }
> >
> > @@ -10984,6 +11030,14 @@ vect_transform_slp_perm_load_1 (vec_info
> *vinfo, slp_tree node,
> > dr_group_size = DR_GROUP_SIZE (stmt_info);
> > }
> >
> > + /* We should probably adjust group size, but for now lets hack. */
> > + unsigned HOST_WIDE_INT vect_nunits = 0;
> > + auto vscalar_stmt = STMT_VINFO_STMT (stmt_info);
> > + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> > + if (VECTOR_TYPE_P (vscalar_type)
> > + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant
> (&vect_nunits))
> > + dr_group_size *= vect_nunits;
> > +
> > mode = TYPE_MODE (vectype);
> > poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > unsigned int nstmts = vect_get_num_copies (vinfo, node);
> > @@ -12431,6 +12485,8 @@ vect_schedule_slp (vec_info *vinfo, const
> vec<slp_instance> &slp_instances)
> > break;
> >
> > store_info = vect_orig_stmt (store_info);
> > + if (!store_info)
> > + continue;
> > /* Free the attached stmt_vec_info and remove the stmt. */
> > vinfo->remove_stmt (store_info);
> >
> > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
> > index
> 7687c107a6afc54741cb5e3f5756bcb65196b519..3cd8e4dfba4b3bf317375
> 3a08cb214493b802e13 100644
> > --- a/gcc/tree-vect-stmts.cc
> > +++ b/gcc/tree-vect-stmts.cc
> > @@ -2108,9 +2108,11 @@ get_load_store_type (vec_info *vinfo,
> stmt_vec_info stmt_info,
> > bool perm_ok = true;
> > poly_int64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1;
> >
> > +#if 1
> > if (SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
> > perm_ok = vect_transform_slp_perm_load (vinfo, slp_node, vNULL, NULL,
> > vf, true, n_perms, n_loads);
> > +#endif
> >
> > if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
> > {
> > @@ -2728,8 +2730,18 @@ get_load_store_type (vec_info *vinfo,
> stmt_vec_info stmt_info,
> > if (k > maxk)
> > maxk = k;
> > tree vectype = SLP_TREE_VECTYPE (slp_node);
> > +
> > + /* We should probably adjust group size, but for now lets hack. */
> > + unsigned HOST_WIDE_INT vect_nunits = 0;
> > + auto group_size = DR_GROUP_SIZE (group_info);
> > + auto vscalar_stmt = STMT_VINFO_STMT (group_info);
> > + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> > + if (VECTOR_TYPE_P (vscalar_type)
> > + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant
> (&vect_nunits))
> > + group_size *= vect_nunits;
> > +
> > if (!TYPE_VECTOR_SUBPARTS (vectype).is_constant (&nunits)
> > - || maxk >= (DR_GROUP_SIZE (group_info) & ~(nunits - 1)))
> > + || maxk >= (group_size & ~(nunits - 1)))
> > {
> > if (dump_enabled_p ())
> > dump_printf_loc (MSG_MISSED_OPTIMIZATION,
> vect_location,
> > @@ -6618,13 +6630,17 @@ vectorizable_operation (vec_info *vinfo,
> > }
> >
> > scalar_dest = gimple_assign_lhs (stmt);
> > + tree scalar_type = TREE_TYPE (scalar_dest);
> > + if (VECTOR_TYPE_P (scalar_type))
> > + scalar_type = TREE_TYPE (scalar_type);
> > +
> > vectype_out = SLP_TREE_VECTYPE (slp_node);
> >
> > /* Most operations cannot handle bit-precision types without extra
> > truncations. */
> > bool mask_op_p = VECTOR_BOOLEAN_TYPE_P (vectype_out);
> > if (!mask_op_p
> > - && !type_has_mode_precision_p (TREE_TYPE (scalar_dest))
> > + && !type_has_mode_precision_p (scalar_type)
> > /* Exception are bitwise binary operations. */
> > && code != BIT_IOR_EXPR
> > && code != BIT_XOR_EXPR
> > @@ -6658,7 +6674,7 @@ vectorizable_operation (vec_info *vinfo,
> > type. */
> > if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (op0)))
> > {
> > - if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (scalar_dest)))
> > + if (!VECT_SCALAR_BOOLEAN_TYPE_P (scalar_type))
> > {
> > if (dump_enabled_p ())
> > dump_printf_loc (MSG_MISSED_OPTIMIZATION,
> vect_location,
> > @@ -8297,6 +8313,14 @@ vectorizable_store (vec_info *vinfo,
> > group_size = 1;
> > }
> >
> > + /* We should probably adjust group size, but for now lets hack. */
> > + unsigned HOST_WIDE_INT vect_nunits = 0;
> > + auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
> > + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> > + if (VECTOR_TYPE_P (vscalar_type)
> > + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant
> (&vect_nunits))
> > + group_size *= vect_nunits;
> > +
> > if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) > 1 && cost_vec)
> > {
> > if (!check_scan_store (vinfo, stmt_info, vectype, rhs_dt, slp_node,
> > @@ -9887,6 +9911,9 @@ vectorizable_load (vec_info *vinfo,
> > padding bits in the type that might leak otherwise.
> > Refer to PR115336. */
> > tree scalar_type = TREE_TYPE (scalar_dest);
> > + if (VECTOR_TYPE_P (scalar_type))
> > + scalar_type = TREE_TYPE (scalar_type);
> > +
> > bool type_mode_padding_p
> > = TYPE_PRECISION (scalar_type) < GET_MODE_PRECISION
> (GET_MODE_INNER (mode));
> >
> > @@ -10509,6 +10536,7 @@ vectorizable_load (vec_info *vinfo,
> > first_stmt_info = stmt_info;
> > group_size = 1;
> > }
> > +
> > /* For SLP vectorization we directly vectorize a subchain
> > without permutation. */
> > if (! SLP_TREE_LOAD_PERMUTATION (slp_node).exists ())
> > @@ -10522,6 +10550,14 @@ vectorizable_load (vec_info *vinfo,
> > first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info);
> > group_gap_adj = 0;
> >
> > + /* We should probably adjust group size, but for now lets hack. */
> > + unsigned HOST_WIDE_INT vect_nunits = 0;
> > + auto vscalar_stmt = STMT_VINFO_STMT (first_stmt_info);
> > + tree vscalar_type = TREE_TYPE (gimple_get_lhs (vscalar_stmt));
> > + if (VECTOR_TYPE_P (vscalar_type)
> > + && TYPE_VECTOR_SUBPARTS (vscalar_type).is_constant
> (&vect_nunits))
> > + group_size *= vect_nunits;
> > +
> > /* VEC_NUM is the number of vect stmts to be created for this group.
> > */
> > grouped_load = false;
> > /* If an SLP permutation is from N elements to N elements,
> > @@ -12015,7 +12051,8 @@ vect_is_simple_cond (tree cond, vec_info
> *vinfo,
> >
> > /* Mask case. */
> > if (TREE_CODE (cond) == SSA_NAME
> > - && VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond)))
> > + && (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (cond))
> > + || VECTOR_BOOLEAN_TYPE_P (TREE_TYPE (cond))))
> > {
> > if (!vect_is_simple_use (vinfo, slp_node, 0, &cond,
> > &slp_op, &dts[0], comp_vectype)
> > @@ -13209,6 +13246,226 @@ vectorizable_early_exit (loop_vec_info
> loop_vinfo, stmt_vec_info stmt_info,
> > return true;
> > }
> >
> > +/* Intrinsics and vector types may not match even though the underlying
> modes
> > + are the same. This is particularly true of masks and SVE modes. As
> > such
> > + convert between the two. */
> > +static void
> > +vect_shim_call_arg (vec<tree> &types, vec_info *vinfo, stmt_vec_info
> stmt_info,
> > + gimple_stmt_iterator *gsi, gcall* call, int argno,
> > + tree newarg)
> > +{
> > + tree ty_dest = types[argno+1];
> > + tree ty_src = TREE_TYPE (newarg);
> > +
> > + if (ty_src != ty_dest)
> > + {
> > + tree lhs = vect_create_destination_var (newarg, ty_dest);
> > + tree op = build1 (VIEW_CONVERT_EXPR, ty_dest, newarg);
> > + gimple *tmp = gimple_build_assign (lhs, VIEW_CONVERT_EXPR, op);
> > + lhs = make_ssa_name (lhs, tmp);
> > + gimple_assign_set_lhs (tmp, lhs);
> > + vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
> > + newarg = lhs;
> > +
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim
> between: %T"
> > + " and %T due to mismatch between generic type and
> vector"
> > + " call.\n", ty_dest, ty_src);
> > + }
> > +
> > + gimple_call_set_arg (call, argno, newarg);
> > +}
> > +
> > +bool
> > +vectorizable_vect_call (vec_info *vinfo,
> > + stmt_vec_info stmt_info, gimple_stmt_iterator *gsi,
> > + slp_tree slp_node,
> > + stmt_vector_for_cost *cost_vec)
> > +{
> > + enum vect_def_type dt[5]
> > + = { vect_unknown_def_type, vect_unknown_def_type,
> vect_unknown_def_type,
> > + vect_unknown_def_type, vect_unknown_def_type };
> > + tree vectypes[ARRAY_SIZE (dt)] = {};
> > + slp_tree slp_op[ARRAY_SIZE (dt)] = {};
> > + tree op;
> > +
> > + /* If the target hook isn't configured there's nothing to do. */
> > + if (!targetm.vectorize.translate_builtin_call)
> > + return false;
> > +
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def
> > + && cost_vec)
> > + return false;
> > +
> > + /* Is STMT_INFO a vectorizable vector call? */
> > + gcall *stmt = dyn_cast <gcall *> (STMT_VINFO_STMT (stmt_info));
> > + if (!stmt || !gimple_call_builtin_p (stmt))
> > + return false;
> > +
> > + /* It must be a built in with a vector type as an argument. */
> > + bool has_vect_arg = false;
> > + int nargs = gimple_call_num_args (stmt);
> > +
> > + /* Check if it has more arguments than we support at the moment. */
> > + if (nargs > 5)
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> > + "vector function call has too many arguments.\n");
> > + return false;
> > + }
> > +
> > + for (int i = 0; i < nargs; i++)
> > + {
> > + has_vect_arg = has_vect_arg
> > + || VECTOR_TYPE_P (TREE_TYPE (gimple_call_arg (stmt, i)));
> > +
> > + if (!vect_is_simple_use (vinfo, slp_node,
> > + i, &op, &slp_op[i], &dt[i], &vectypes[i]))
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> > + "use not simple.\n");
> > + return false;
> > + }
> > + }
> > +
> > + if (!has_vect_arg)
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> > + "function is not a vectorizable vector function.\n");
> > + return false;
> > + }
> > +
> > + /* We only handle functions that do not read or clobber memory. */
> > + if (gimple_vuse (stmt))
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> > + "function reads from or writes to memory.\n");
> > + return false;
> > + }
> > +
> > + tree vectype_out;
> > + int maskopno = -1;
> > + auto_vec<tree> types;
> > + gcall *call
> > + = targetm.vectorize.translate_builtin_call (vinfo, slp_node, stmt_info,
> > + &maskopno, types,
> &vectype_out);
> > +
> > + if (!call)
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> > + "target did not have implementation for %G.\n",
> > + (gimple*)stmt);
> > + return false;
> > + }
> > +
> > + loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
> > + vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS
> (loop_vinfo) : NULL);
> > + vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) :
> NULL);
> > + bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P
> (loop_vinfo);
> > + bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P
> (loop_vinfo);
> > + unsigned int vect_nargs = nargs;
> > +
> > + if (maskopno != -1)
> > + {
> > + if (!loop_vinfo) // || (!masked_loop_p && !len_loop_p))
> > + {
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
> > + "target translation requires predication but "
> > + "masks not supported: %G.\n",
> > + (gimple*)stmt);
> > + return false;
> > + }
> > + vect_nargs += 1;
> > + }
> > +
> > + if (cost_vec) /* transformation not required. */
> > + {
> > + SLP_TREE_TYPE (slp_node) = vector_call_vec_info_type;
> > + DUMP_VECT_SCOPE ("vectorizable_vect_call");
> > + vect_model_simple_cost (vinfo, 1, slp_node, cost_vec);
> > + return true;
> > + }
> > +
> > + /* Transform. */
> > +
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_NOTE, vect_location, "transform vector call.\n");
> > +
> > + /* Resolve arguments. */
> > + auto_vec<vec<tree> > vec_defs (nargs);
> > + vect_get_slp_defs (vinfo, slp_node, &vec_defs);
> > +
> > + /* Handle def. */
> > + tree scalar_dest = gimple_call_lhs (stmt);
> > + tree vec_dest = vect_create_destination_var (scalar_dest, vectype_out);
> > + tree new_temp = make_ssa_name (vec_dest, stmt);
> > + gimple_call_set_lhs (call, new_temp);
> > + gimple_call_set_nothrow (call, true);
> > +
> > + /* Now fill in the new values. Intrinsics types and vector types may not
> > + match, so convert the types. */
> > + int shift = 0;
> > + for (int i = 0; i < nargs; i++)
> > + {
> > + if (maskopno == i)
> > + {
> > + tree mask;
> > + if (masked_loop_p)
> > + {
> > + /* ?? Handle decomposition here, can that happen??. */
> > + unsigned int vec_num = 1; //vec_oprnds0.length ();
> > + mask = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
> > + vectype_out, 0);
> > + }
> > + else
> > + {
> > + tree mask_vectype = truth_type_for (vectype_out);
> > + mask = vect_build_all_ones_mask (loop_vinfo, stmt_info,
> > + mask_vectype);
> > + }
> > + vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i, mask);
> > + shift++;
> > + }
> > + tree arg = vec_defs[i][0];
> > + vect_shim_call_arg (types, vinfo, stmt_info, gsi, call, i+shift,
> > arg);
> > + }
> > +
> > + gimple *res = call;
> > + /* Check if the result types match, if not, shim them. */
> > + if (vectype_out != types[0])
> > + {
> > + /* First update the call. */
> > + tree lhs = vect_create_destination_var (scalar_dest, types[0]);
> > + tree new_lhs = make_ssa_name (lhs, NULL);
> > + gimple_call_set_lhs (call, new_lhs);
> > + vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
> > + tree op = build1 (VIEW_CONVERT_EXPR, vectype_out, new_lhs);
> > + gimple *tmp = gimple_build_assign (vec_dest, VIEW_CONVERT_EXPR,
> op);
> > + gimple_assign_set_lhs (tmp, new_temp);
> > + vect_finish_stmt_generation (vinfo, stmt_info, tmp, gsi);
> > + res = tmp;
> > +
> > + if (dump_enabled_p ())
> > + dump_printf_loc (MSG_NOTE, vect_location, "inserted type shim
> between: %T"
> > + " and %T due to mismatch between generic type and
> vector"
> > + " call.\n", types[0], vectype_out);
> > + }
> > + else
> > + vect_finish_stmt_generation (vinfo, stmt_info, call, gsi);
> > +
> > + slp_node->push_vec_def (res);
> > + return true;
> > +}
> > +
> > /* If SLP_NODE is nonnull, return true if vectorizable_live_operation
> > can handle all live statements in the node. Otherwise return true
> > if STMT_INFO is not live or if vectorizable_live_operation can handle
> > it.
> > @@ -13338,6 +13595,7 @@ vect_analyze_stmt (vec_info *vinfo,
> > || vectorizable_shift (vinfo, stmt_info, NULL, node, cost_vec)
> > || vectorizable_condition (vinfo, stmt_info, NULL, node, cost_vec)
> > || vectorizable_comparison (vinfo, stmt_info, NULL, node, cost_vec)
> > + || vectorizable_vect_call (vinfo, stmt_info, NULL, node, cost_vec)
> > || (bb_vinfo
> > && vectorizable_phi (bb_vinfo, stmt_info, node, cost_vec))
> > || (is_a <loop_vec_info> (vinfo)
> > @@ -13454,6 +13712,10 @@ vect_transform_stmt (vec_info *vinfo,
> > done = vectorizable_call (vinfo, stmt_info, gsi, slp_node, NULL);
> > break;
> >
> > + case vector_call_vec_info_type:
> > + done = vectorizable_vect_call (vinfo, stmt_info, gsi, slp_node,
> > NULL);
> > + break;
> > +
> > case call_simd_clone_vec_info_type:
> > done = vectorizable_simd_clone_call (vinfo, stmt_info, gsi,
> > slp_node, NULL);
> > @@ -13572,6 +13834,10 @@ get_related_vectype_for_scalar_type
> (machine_mode prevailing_mode,
> > machine_mode simd_mode;
> > tree vectype;
> >
> > + /* For vector types, get the inner most type. */
> > + if (VECTOR_TYPE_P (scalar_type))
> > + scalar_type = TREE_TYPE (scalar_type);
> > +
> > if ((!INTEGRAL_TYPE_P (scalar_type)
> > && !POINTER_TYPE_P (scalar_type)
> > && !SCALAR_FLOAT_TYPE_P (scalar_type))
> > @@ -14801,6 +15067,10 @@ vect_get_vector_types_for_stmt (vec_info
> *vinfo, stmt_vec_info stmt_info,
> > else
> > scalar_type = TREE_TYPE (gimple_get_lhs (stmt));
> >
> > + /* If scalar_type is a vector type, get the innner typ. */
> > + if (VECTOR_TYPE_P (scalar_type))
> > + scalar_type = TREE_TYPE (scalar_type);
> > +
> > if (dump_enabled_p ())
> > {
> > if (group_size)
> > @@ -14901,4 +15171,3 @@ vect_gen_len (tree len, tree start_index, tree
> end_index, tree len_limit)
> >
> > return stmts;
> > }
> > -
> > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> > index
> 35e531c11c178839bcf3218e58ad7521662d3b42..1bba7702b628cc40128
> 48e1d887be03fe3d2533d 100644
> > --- a/gcc/tree-vectorizer.h
> > +++ b/gcc/tree-vectorizer.h
> > @@ -229,6 +229,7 @@ enum stmt_vec_info_type {
> > shift_vec_info_type,
> > op_vec_info_type,
> > call_vec_info_type,
> > + vector_call_vec_info_type,
> > call_simd_clone_vec_info_type,
> > assignment_vec_info_type,
> > condition_vec_info_type,
> >
> >
> >
>
> --
> Richard Biener <[email protected]>
> SUSE Software Solutions Germany GmbH,
> Frankenstrasse 146, 90461 Nuernberg, Germany;
> GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG
> Nuernberg)
