[Bug c/87248] New: Bad code for masked operations involving signed ints

[david at westcontrol dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87248

            Bug ID: 87248
           Summary: Bad code for masked operations involving signed ints
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: david at westcontrol dot com
  Target Milestone: ---

I am always wary of saying there might be a compiler bug - usually it is a bug
in the user code.  But this time I am very suspicious.  The example here comes
from a discussion in the comp.lang.c Usenet group.

Here is the code I have been testing:


unsigned char foo_u(unsigned int v) {
    return (v & 0x7f) | (v & ~0x7f ? 0x80 : 0);
}

unsigned char foo_s(int v) {
    return (v & 0x7f) | (v & ~0x7f ? 0x80 : 0);
}

unsigned char bar_s(int v) {
    int x = v & ~0x7f;
    return (v & 0x7f) | (x ? 0x80 : 0);
}


int test_u_1(void) {
    return foo_u(0x01);        // Expect 0x01 = 1
}

int test_u_2(void) {
    return foo_u(0x1001);    // Expect 0x81 = 129
}

int test_s_1(void) {
    return foo_s(0x01);        // Expect 0x01 = 1
}

int test_s_2(void) {
    return foo_s(0x1001);    // Expect 0x81 = 129
}


The assembly code generated for this is:

foo_u(unsigned int):
        mov     eax, edi
        mov     edx, edi
        or      edx, -128
        and     eax, 127
        and     edi, -128
        cmovne  eax, edx
        ret
foo_s(int):
        mov     eax, edi
        ret
bar_s(int):
        mov     eax, edi
        mov     edx, edi
        or      edx, -128
        and     eax, 127
        and     edi, 127
        cmovne  eax, edx
        ret
test_u_1():
        mov     eax, 1
        ret
test_u_2():
        mov     eax, 129
        ret
test_s_1():
        mov     eax, 1
        ret
test_s_2():
        mov     eax, 1
        ret



When the code uses "int" rather than "unsigned int", in "foo_s", the compiler
thinks the function can be optimised to a simple byte extraction.  I cannot see
how that interpretation is valid.  And gcc cannot see it either, if there is an
intermediate variable (in "bar_s").  (The type of the intermediate "x" in
"bar_s" does not matter - "int", "unsigned int", "auto", "__auto_type", const
or not).

This effect is happening in the front-end.  It is independent of optimisation
levels (-O gives the same results), the target processor (I tried a half-dozen
targets), the compiler version (from gcc 4.4 upwards - gcc 4.1 gives the same
code for foo_s as foo_u.  I haven't tried gcc 4.2 or 4.3).  There are minor
variations in the generated code (such as the use of conditional move
instructions) between different versions - those don't affect the results.  The
"-fwrapv" flag has no effect either, nor does the choice of C or C++.

The results from the compiler evaluations in the "test_" functions shows that
it this happens in the compiler analysis - it is not a code generation issue.

(<https://godbolt.org> is great for this kind of testing!)

(This bug report was first sent to the mailing list
<https://gcc.gnu.org/ml/gcc/2018-09/msg00028.html> - the report here has a few
minor corrections.)