This is a summary of discussions relative to the merge request created by
Christopher Bazley (chris.bazley) <[email protected]> titled
AArch64: Add scalar-to-vector costs for vec_construct
since its creation.
Description: An anti-pattern found in compiled code when predicated tails were
enabled for basic block SLP vectorization was triggered by
byte-reversing patterns in source code, such as:
uint8_t *dst;
int size;
dst[0] = size >> 24;
dst[1] = size >> 16;
dst[2] = size >> 8;
dst[3] = size >> 0;
which would previously have compiled to:
rev w1, w1
str w1, [x0]
but (with tail-predication) was vectorized as:
mov z31.b, w1
ptrue p7.s, vl4
fmov s30, w1
sshr v29.2s, v30.2s, #8
insr z31.s, s29
sshr v30.2s, v30.2s, #16
insr z31.s, s30
fmov s30, w1
sshr v30.2s, v30.2s, #24
insr z31.s, s30
st1b {z31.s}, p7, [x0]
One reason is that the SLP pass runs before the store-merging
pass gets a chance to coalesce 4 stores into 1 and substitute a
32 bit bswap implementation. Even ignoring that, costing of the
vectorized version (cost: 4) compared to the scalar version
(also 4) was not realistic:
_2 1 times vector_store costs 1 in body
node 0x32ee6d0 1 times vec_construct costs 3 in prologue
There were a couple of contributing issues:
1. the cost of mask construction for the vector_store (ptrue) was
omitted for BB SLP, whereas the loop vectorizer explicitly charges
for it.
2. the cost of vec_construct (elements / 2 + 1) did not incorporate
any GPR-to-SIMD register transfer costs (mov, fmov).
Since the supposed cost of the vectorised code only just reached parity
with the scalar code, addressing either of the above issues would be
sufficient to prevent vectorisation (in this specific case). It is also
less risky than changing the order of passes, and less hacky than
teaching the SLP pass about store-merging.
This commit addresses only the second issue, by adding code in
vector_costs::add_stmt_cost to charge scalar_to_vec_cost for each
element of an external def of kind vec_construct (with specific
exceptions noted below). This cost is added to the base cost
already charged by aarch64_builtin_vectorization_cost for a
vec_construct (which is assumed to cover the cost of the INSR or
equivalent instructions).
This is justifiable because SIMD-to-SIMD insertions into a vector
register generally have lower latency and higher throughput than
GPR-to-SIMD insertions.
The basic structure of the code was copied from commit
90d693bdc9d71841f51d68826ffa5bd685d7f0bc which modified the x86
backend in a similar way, but adapted to use a hash_set<tree>
instead of TREE_VISITED to guard against charging twice or more for
the same scalar op feeding an external def.
This commit assumes that constructing a vector from memory
is no more costly than the equivalent set of scalar loads (or at least
that any difference is incorporated in the cost returned by
aarch64_builtin_vectorization_cost for vec_construct). It also assumes
that constructing a vector from scalar values of floating point type,
from a BIT_FIELD_REF/lastb that extracts from a vector register, or
from the result of a call to an inbuilt reduction function, does not
incur GPR-to-SIMD register transfer costs because such scalars are
typically already in FP/SIMD registers on AArch64.
CC: [email protected], [email protected], [email protected]
The full and up to date discussion can be found at
https://forge.sourceware.org/gcc/gcc-TEST/pulls/154
The merge request has been closed without being merged directly on the forge
repository.
On 2026-05-19 15:13:47+00:00, Tamar Christina (tnfchris) <[email protected]>
approved the changes:
As discussed internally this is fine.
