From: Kyrylo Tkachov <[email protected]>
vect_slp_analyze_data_ref_dependence conservatively reports a dependence
whenever the classical (affine) data-dependence test returns chrec_dont_know,
e.g. when one of the accesses has a non-affine or runtime array subscript.
In the BB SLP region check this is overly pessimistic: the unanalyzable
subscript says nothing about whether the two references can actually alias,
and the alias oracle can frequently still prove they cannot (distinct restrict
parameters, distinct non-escaping objects, TBAA-incompatible types, and so on).
When that happens a perfectly good SLP group can be torn down.
Consult the alias oracle (refs_may_alias_p) before assuming a dependence on
chrec_dont_know: if the two references cannot alias they are independent
regardless of the unanalyzable subscript, and BB SLP may proceed.
On the new gcc.dg/vect/bb-slp-dep-oracle.c the 8-lane reciprocal group is torn
down and emitted scalar without the patch
fmov d23, 1.0e+0
fdiv d8, d23, d8
fdiv d9, d23, d9
fdiv d10, d23, d10
fdiv d11, d23, d11
fdiv d12, d23, d12
fdiv d13, d23, d13
fdiv d14, d23, d14
fdiv d15, d23, d15
and vectorizes with it:
fmov v27.2d, 1.0e+0
fdiv v28.2d, v27.2d, v28.2d
fdiv v29.2d, v27.2d, v29.2d
fdiv v30.2d, v27.2d, v30.2d
fdiv v31.2d, v27.2d, v31.2d
Bootstrapped and tested on aarch64-none-linux-gnu.
Ok for trunk?
Thanks,
Kyrill
Signed-off-by: Kyrylo Tkachov <[email protected]>
gcc/ChangeLog:
* tree-vect-data-refs.cc (vect_slp_analyze_data_ref_dependence): On an
unknown (chrec_dont_know) dependence, consult the alias oracle on the
references and treat them as independent when they provably cannot
alias.
gcc/testsuite/ChangeLog:
* gcc.dg/vect/bb-slp-dep-oracle.c: New test.
---
gcc/testsuite/gcc.dg/vect/bb-slp-dep-oracle.c | 40 +++++++++++++++++++
gcc/tree-vect-data-refs.cc | 8 ++++
2 files changed, 48 insertions(+)
create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-dep-oracle.c
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-dep-oracle.c
b/gcc/testsuite/gcc.dg/vect/bb-slp-dep-oracle.c
new file mode 100644
index 00000000000..b2551ef644c
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-dep-oracle.c
@@ -0,0 +1,40 @@
+/* BB SLP must not abandon a vectorizable group when the classical (affine)
+ data-dependence test cannot analyze a runtime array subscript but the alias
+ oracle can still prove the two references do not alias.
+
+ The per-lane reciprocals are discovered as an SLP group, then the group is
+ torn down by vect_slp_analyze_data_ref_dependence reporting "can't determine
+ dependence" between the restrict output store and a runtime-indexed input
+ load, even though the distinct restrict objects provably do not alias. */
+
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_double } */
+/* { dg-additional-options "-O3 -ffast-math -fno-trapping-math" } */
+
+struct VA { double data[8]; };
+struct Tensor { struct VA comp[4]; };
+
+/* Opaque: the returned index is a runtime value the affine subscript test
+ cannot analyze. */
+unsigned __attribute__((noipa)) pick (unsigned k) { return k & 3; }
+
+void f (const struct Tensor *in, double *__restrict out,
+ unsigned nq, unsigned base)
+{
+ for (unsigned q = 0; q < nq; q++)
+ {
+ const struct VA *rho = &in[q].comp[0]; /* divisor: contiguous
.data[i] */
+ double inv[8];
+ for (unsigned i = 0; i < 8; i++)
+ inv[i] = 1.0 / rho->data[i]; /* reciprocal group, reused
below */
+ for (unsigned d = 0; d < 3; d++)
+ {
+ const struct VA *mom = &in[q].comp[pick (base + d)]; /*
runtime-indexed numerator */
+ for (unsigned i = 0; i < 8; i++)
+ out[(q * 3 + d) * 8 + i] = mom->data[i] * inv[i];
+ }
+ }
+}
+
+/* The reciprocal group must survive the dependence check and vectorize. */
+/* { dg-final { scan-tree-dump "basic block part vectorized" "slp1" } } */
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 0e57e1068d6..07d0d860d44 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -1011,6 +1011,14 @@ vect_slp_analyze_data_ref_dependence (vec_info *vinfo,
/* Unknown data dependence. */
if (DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
{
+ /* The classical (affine) dependence test gave up, e.g. a non-affine
+ or runtime array subscript. Before conservatively assuming a
+ dependence (which prevents SLP), consult the alias oracle on the
+ actual references: if they provably cannot alias (e.g. based on
+ distinct restrict / non-escaping objects) they are independent
+ regardless of the unanalyzable subscript. */
+ if (!refs_may_alias_p (DR_REF (dra), DR_REF (drb)))
+ return false;
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
"can't determine dependence between %T and %T\n",
--
2.50.1 (Apple Git-155)
