This is a bug report for gcc 4_3 branch. I will attach a test case, slightly
reduced from zlib code. When compiling this test case for the x86_64-linux
target with -O2 -fomit-frame-pointer, I see this at the start of the function:
adler32:
pushq %rbp
movq %rdi, %rax
andl $65535, %edi
shrq $16, %rax
movq %rsp, %rbp
pushq %r15
andl $65535, %eax
movl %edx, -140(%rbp)
After %rsp was copied to %rbp, %r15 was pushed. So now %rsp is 8 bytes less
than %rbp. The red zone is 128 bytes, so this means that any reference down to
-136(%rbp) is valid. However, the code actually stores a value into
-140(%rbp). This is invalid, and can cause subtle and unpredictable bugs
depending upon when the kernel interrupts this code.
This error happens because the scheduler moves this movl instruction ahead of
some other pushq instructions. If all the pushq instructions happened first,
the stack pointer would have been decremented sufficiently that the reference
to -140(%rbp) would be safely within the red zone.
4.4 generates completely different, and seemingly better, code for this test
case. However, I don't see anything in the 4.4 source code which would prevent
this same bug from occurring given the right test case. So this is potentially
a serious error for the 4.4 release as well, unless it has been fixed by some
scheduling dependency that I haven't uncovered. However, I have not been able
to find a test case.
--
Summary: x86_64 red zone violation
Product: gcc
Version: 4.3.4
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: target
AssignedTo: unassigned at gcc dot gnu dot org
ReportedBy: ian at airs dot com
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39118
