[PATCH] PR49720, infinite recursion in RTX simplification

[Chung-Lin Tang]

Hi,
this patch is for a specific case in RTX simplification where infinite
recursion ensues, causing the out-of-stack segfault in PR49720.

Tracking back the origin of this bug, the exact revision causing this
was rev.92429, back in 2004.

The clause in simplify-rtx.c:simplify_relational_operation_1(),
transforming X + cst1 == cst2 into X == cst2 - cst1 can cause an
oscillating recursive cycle like this:

  A + B == C    <===>   C - B == A

When all A, B, and C are CONSTANT_P, but not simplifiable (in the PR
testcase, they are all SYMBOL_REFs).  The switching of XEXP(op0,1) to
the other side of the equation, plus commutative canonicalizing then
completes the cycle.

This can be solved by using simplify_binary_relation() and returning
NULL_RTX when simplification of the constant PLUS/MINUS fails, but I'm
not sure that further working on X == CST2-CST1 will never yield more
optimization, so the patch here just tries to plug the recursion hole
for the exact condition.

Bootstrapped and tested on i686 and x86_64 without regressions. Is this
okay for trunk?

Thanks,
Chung-Lin

2011-10-27  Chung-Lin Tang  <clt...@codesourcery.com>

        PR rtl-optimization/49720
        * simplify-rtx.c (simplify_relational_operation_1): Detect
        infinite recursion condition in "(eq/ne (plus x cst1) cst2)
        simplifies to (eq/ne x (cst2 - cst1))" case.

        testsuite/
        * g++.dg/torture/pr49720.C: New test.

Index: trunk/gcc/simplify-rtx.c
===================================================================
--- trunk/gcc/simplify-rtx.c    (revision 180421)
+++ trunk/gcc/simplify-rtx.c    (working copy)
@@ -4352,10 +4352,20 @@
     {
       rtx x = XEXP (op0, 0);
       rtx c = XEXP (op0, 1);
+      enum rtx_code invcode = op0code == PLUS ? MINUS : PLUS;
+      rtx tem = simplify_gen_binary (invcode, cmp_mode, op1, c);
 
-      c = simplify_gen_binary (op0code == PLUS ? MINUS : PLUS,
-                              cmp_mode, op1, c);
-      return simplify_gen_relational (code, mode, cmp_mode, x, c);
+      /* Detect an infinite recursive condition, where we oscillate at this
+        simplification case between:
+           A + B == C  <--->  C - B == A,
+        where A, B, and C are all constants with non-simplifiable expressions,
+        usually SYMBOL_REFs.  */
+      if (GET_CODE (tem) == invcode
+         && CONSTANT_P (x)
+         && rtx_equal_p (c, XEXP (tem, 1)))
+       return NULL_RTX;
+
+      return simplify_gen_relational (code, mode, cmp_mode, x, tem);
     }
 
   /* (ne:SI (zero_extract:SI FOO (const_int 1) BAR) (const_int 0))) is
Index: trunk/gcc/testsuite/g++.dg/torture/pr49720.C
===================================================================
--- trunk/gcc/testsuite/g++.dg/torture/pr49720.C        (revision 0)
+++ trunk/gcc/testsuite/g++.dg/torture/pr49720.C        (revision 0)
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+
+extern char t_start[], t_end[], t_size[];
+bool foo (void)
+{
+  long size = reinterpret_cast<long>(t_size);
+  return (size == t_end - t_start);
+}