[PATCH] Fix UB in store-merging or bswap opt (PR tree-optimization/87826)

[Jakub Jelinek]

Hi!

When compiling code with UB shifts/rotates where the shift count is out of
range constant, the store-merging and bswap opts will invoke UB in the
compiler too.
The following patch prevents that.

Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk/8.x?

2018-11-01  Jakub Jelinek  <ja...@redhat.com>

        PR tree-optimization/87826
        * gimple-ssa-store-merging.c (do_shift_rotate): Punt if count is
        negative or larger or equal to type's precision.

        * gcc.dg/pr87826.c: New test.

--- gcc/gimple-ssa-store-merging.c.jj   2018-10-23 10:13:25.199876583 +0200
+++ gcc/gimple-ssa-store-merging.c      2018-10-31 11:07:50.961296026 +0100
@@ -262,7 +262,9 @@ do_shift_rotate (enum tree_code code,
   int i, size = TYPE_PRECISION (n->type) / BITS_PER_UNIT;
   unsigned head_marker;
 
-  if (count % BITS_PER_UNIT != 0)
+  if (count < 0
+      || count >= TYPE_PRECISION (n->type)
+      || count % BITS_PER_UNIT != 0)
     return false;
   count = (count / BITS_PER_UNIT) * BITS_PER_MARKER;
 
--- gcc/testsuite/gcc.dg/pr87826.c.jj   2018-10-31 11:08:56.907209723 +0100
+++ gcc/testsuite/gcc.dg/pr87826.c      2018-10-31 11:25:48.078579206 +0100
@@ -0,0 +1,13 @@
+/* PR tree-optimization/87826 */
+/* { dg-do compile } */
+/* { dg-options "-O3 -w" } */
+
+int c;
+
+void
+foo (int *b)
+{
+  int e;
+  for (e = 0; e < 16; ++e)
+    b[e] = c >> e * 8;
+}

        Jakub