[Bug c/18260] New: generated code uses memory beyond allocated stack frame.

[jtc at acorntoolworks dot com]

A co-worker testing MySQL 4 on NetBSD/amd64 was having trouble with random
crashes.  He tracked the crashes down to mf_radix.c:radixsort_for_str_ptr(), and
handed the problem off to me for further analysis.

I cut down mf_radix.c (attached to this bug report), and looked at the code
generated with "gcc -O2 -S mf_radix.c".  It appears that the memory for local
variables is not accounted for entirely correctly.  There are 1024+ bytes of
local variables, but the stack pointer (%rsp) is only adjusted by 912 bytes. 
And it appears local variable "count" is referenced before the stack pointer
(-120(%rsp)).

I believe, but have not yet confirmed, that the crashes occur because something
in NetBSD's thread/task switch code is overwriting the values below the stack
pointer assuming that it is free for temporary use.  This may also effect other
systems --I've done the same in embedded kernels I've written in the past.

NetBSD 2.0RC4 uses a modified version of gcc 3.3.3, so I have confirmed that the
problem still exists with unmodified gcc 3.3.5.  Linking in versions of
mf_radix.c compiled without optimization or with gcc 3.4 eliminates the crashes. 

My attempts to build MySQL 4 entirely with gcc 3.4.X have not been successful,
but I have not had enough time to investigate why.

-- 
   Summary: generated code uses memory beyond allocated stack frame.
   Product: gcc
   Version: 3.3.5
Status: UNCONFIRMED
  Severity: normal
  Priority: P2
 Component: c
AssignedTo: unassigned at gcc dot gnu dot org
    ReportedBy: jtc at acorntoolworks dot com
CC: gcc-bugs at gcc dot gnu dot org
  GCC host triplet: i386-netbsd
GCC target triplet: amd64-netbsd


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18260

[Bug rtl-optimization/18260] generated code uses memory beyond allocated stack frame.

[jtc at acorntoolworks dot com]

--- Additional Comments From jtc at acorntoolworks dot com  2004-11-01 00:21 
---
Created an attachment (id=7449)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=7449&action=view)
cut down test case

This is a cut down test case derived from MySQL's mf_radix.c.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18260

[Bug rtl-optimization/18260] generated code uses memory beyond allocated stack frame.

[jtc at acorntoolworks dot com]

--- Additional Comments From jtc at acorntoolworks dot com  2004-11-01 22:45 
---
I seem to have missed the description of the "red-zone" when I read the ABI.  It's a 
neat optimization 
for leaf functions, but violates the principal of least suprise.

I have confirmed that the MySQL4 crashes that we were experiencing went away with 
-mno-red-zone.

I have done further investigation, and it appears that NetBSD/amd64's thread context 
switch code did 
not preserve the red-zone until recently.  This has been fixed in -current, and 
hopefully the changes 
will be pulled up to the release branch before 2.0 is released.


-- 
   What|Removed |Added

 Status|UNCONFIRMED |RESOLVED
 Resolution||INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18260