Hi,
We are trying to update our patches on complex numbers to take into
account what has been discussed.
The main change from our previous patches consists of replacing vectors
of complex types with classical vectors of real types (ex V4SF instead
of V2SC) associated with existing complex opcodes (like .COMPLEX_MUL)
when vectorizing. Non vectored complex modes are also replaced by
vectors of two reals at the end of the middle-end (ex SC to V2SF), so
that it can reuse already existing patterns. Indeed, non complex
specific operations like an addition does not require an specific
pattern anymore, and already implementing patterns like cmul, cmul_conj,
cadd90,... can be used.
To do so, the cplxlower pass has been cut into two passes:
- The first one replace complex specific opcodes with dedicated
opcodes (like .COMPLEX_MUL replacing MUL_EXPR with SC mode), but complex
modes are kept at this point. Unsupported native operations are also
lowered, because we assume that it's better to lower and hope for
standard optimizations in the middle-end than trying to vectorize with
near-zero chance, and then lower only after.
- The second one almost only remaps non vectored complex modes into
vector of two reals (like SC to V2SF).
So the vectorizer takes complex modes as input but vectorize with
vectors of real modes (ex V4SF vector mode for SC). Because complex
specific opcodes have been set before, no confusion with real operations
is possible. We also may use vectors of two reals as inputs, but
vectorizing small vector modes into bigger ones (like V2SF to V4SF) is
not possible.
Here are some advantages of this new approach:
- No more vectors of complex modes
- The vectorization of complex operations is improved, because split
and unified vectored statements can easely be mixed as it uses the same
vector type. We can also imagine to test multiple options (First: native
vectored, second: split vectored, third: unified scalar,...).
- It reuses patterns for vectors of two reals for non complex
specific operations, and also already existing complex patterns like
cmul implemented on aarch64, which could mean almost free performance
gains on many targets.
On the performance side, we can still exploit the full potential of
complex instructions on KVX. To illustrate the gains on aarch64 without
rewriting any patterns (except a mov), here is the assembly generated
for a vector complex mul mul add with -O2 -mcpu=neoverse-v1 (and without
ffast-math like with SLP):
void vfmma (_Complex float a[restrict N], _Complex float b[restrict N],
_Complex float c[restrict N], _Complex float
d[restrict N])
{
for (int i = 0; i < N; i++)
c[i] += a[i] * b[i] * d[i];
}
vfmma:
movi v3.4s, 0
mov x4, 0
.align 5
.L2:
ldr q2, [x1, x4]
mov v1.16b, v3.16b
ldr q0, [x0, x4]
fcmla v1.4s, v0.4s, v2.4s, #0
fcmla v1.4s, v0.4s, v2.4s, #90
ldr q0, [x2, x4]
ldr q2, [x3, x4]
fcmla v0.4s, v2.4s, v1.4s, #0
fcmla v0.4s, v2.4s, v1.4s, #90
str q0, [x2, x4]
add x4, x4, 16
cmp x4, 256
bne .L2
ret
We have only done some experimentation with this approach. If you think
that it could be interesting we will try to develop it more.
Thanks,
Sylvain
