Hi all, In the sink pass, if-convert a diamond that selects between two loads at data-dependent addresses (PHI <*P, *Q>) into a single reg-offset load behind branchless selects, reusing phi-opt's factoring helpers (exported in the previous patch) to common the load and its address. Guarded so it only fires on non-vectorisable in-loop load-address recurrences, leaving affine vectorisable loops alone. This was needed to avoid regressing the marian benchmark in SPEC2026 that otherwise ended up using expensive gathers to vectorise. We can look to relax this condition in the future for cases where we think it's beneficial.
The load is commoned only when both arms are guaranteed to reduce to pure
speculatable scalar (scc_arms_speculatable_p), so the branchless finish always
succeeds and no commoned-but-still-branching diamond is ever left behind. The
recurrence walk is bounded by a new --param=sink-diamond-recurrence-limit
(default 256).
With this patch the Snappy hot loop gets if-converted optimally and performance
improves by 25% on my aarch64 machine, making it a bit better than what LLVM
gets today.
The code goes from:
.L8: ; ---type==0 arm
lsr x3, x1, 2 ; tag >> 2
add x1, x0, x3
add x3, x3, 2
add x0, x0, x3 ; ip += (tag>>2)+2
ldrb w1, [x1, 1] ; LOAD #1: tag = ip[(tag>>2)+1]
cmp x2, x0
bls .L2
.L5: ; loop head / else arm
ands x3, x1, 3 ; type = tag & 3
beq .L8 ; <-- DATA-DEPENDENT BRANCH (mispredicts)
ldrb w1, [x0, x3] ; LOAD #2: tag = ip[type]
add w3, w3, 1
add x0, x0, x3 ; ip += type+1
cmp x2, x0
bhi .L5
to:
.L3: ; --- single loop body, no diamond branch
lsr x4, x1, 2 ; tag >> 2
ands x3, x1, 3 ; type = tag & 3 (Z = type==0)
csinc x1, x3, x4, ne ; offset = (type!=0) ? type : (tag>>2)+1
ldrb w1, [x0, x1] ; ONE reg-offset LOAD: tag = ip[offset]
add x4, x4, 2
csinc x3, x4, x3, eq ; advance = (type==0) ? (tag>>2)+2 : type+1
add x0, x0, x3 ; ip += advance
cmp x2, x0
bhi .L3
crucially avoiding the badly-predicted branch.
Bootstrapped and tested on aarch64-none-linux-gnu.
Ok for trunk after Andrea’s patches land?
Thanks,
Kyrill
Signed-off-by: Kyrylo Tkachov <[email protected]>
gcc/
PR tree-optimization/125557
* params.opt (sink-diamond-recurrence-limit): New param.
* doc/params.texi (sink-diamond-recurrence-limit): Document it.
* tree-ssa-sink.cc (scc_recurrence_p, scc_arms_speculatable_p)
(scc_try_ifconvert, sink_common_computations_to_bb): New.
(pass_sink_code::execute): Call it at the early sink.
gcc/testsuite/
PR tree-optimization/125557
* gcc.dg/tree-ssa/scc-diamond-1.c: New test.
* gcc.dg/tree-ssa/scc-diamond-3.c: New test.
* gcc.target/aarch64/scc-diamond-2.c: New test.
0002-2-2-PR-tree-optimization-125557-if-convert-data-depe.patch
Description: 0002-2-2-PR-tree-optimization-125557-if-convert-data-depe.patch
