[Bug target/110557] New: Wrong code for x86_64-linux-gnu with -O3 -mavx2: vectorized loop mishandles signed bit-fields

[xry111 at gcc dot gnu.org via Gcc-bugs]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110557

            Bug ID: 110557
           Summary: Wrong code for x86_64-linux-gnu with -O3 -mavx2:
                    vectorized loop mishandles signed bit-fields
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: xry111 at gcc dot gnu.org
  Target Milestone: ---

$ cat test.cc

#include <cstddef>

size_t max(size_t a, size_t b) { return a < b ? b : a; }

struct Item {
  int x : 8;
  std::ptrdiff_t y : 56;
};

__attribute__((noipa)) std::size_t test(Item *a, int cnt) {
  std::size_t size = 0;
  for (int i = 0; i < cnt; i++)
    size = max(static_cast<size_t>(-(a[i].y * 4)), size);
  return size;
}

int main() {
  struct Item items[] = {
      {1, -1},
      {2, -2},
      {3, -3},
      {4, -4},
  };

  if (test(items, 4) != 16)
    __builtin_trap();
}

$ g++ test.cc -Wall -Wextra -O3 -fsanitize=undefined
$ ./a.out && echo ok
ok
$ g++ test.cc -Wall -Wextra -O3 -mavx2
$ ./a.out && echo ok
Illegal instruction (core dumped)

Reproducible with GCC 13.1 and trunk.  Interestingly if I convert the test case
to  C (instead of C++) the issue won't reproduce.

This test case is reduced from WebKit, it crashes with GCC 13.1 and -O3 -mavx2.
 The WebKit code has "<< 2" instead of "* 4", thus invoking an undefined
behavior.  But it still crashes even if I change "<< 2" to "* 4".