[Bug c/47533] New: [m68k] stack corruption with -Os (probably due to tail call optimisation)

[tg at mirbsd dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47533

           Summary: [m68k] stack corruption with -Os (probably due to tail
                    call optimisation)
           Product: gcc
           Version: 4.4.5
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
        AssignedTo: unassig...@gcc.gnu.org
        ReportedBy: t...@mirbsd.org


I just discovered a stack corruption in klibc in m68k and tracked it down to
this reduced testcase. When using -O2 or -Os, the problem happens. The
component may be wrong, but I don’t know the correct one to use (backend?
*-optimisation? target?) so please reassign. I didn’t choose target because
this _might_ happen on !m68k too (I don’t know, but if so, it’s a major
problem).

GNU C (Debian 4.4.5-10) version 4.4.5 (m68k-linux-gnu)
        compiled by GNU C version 4.4.5, GMP version 4.3.2, MPFR version
3.0.0-p3.

root@ara2:~ # gcc tst1.S tst2.c -Os
root@ara2:~ # ./a.out
want: C0000010
got1: C0000010
syscall with 1, 258, 16 reached
got2: 00000010

The problem appears to be this:

80000448 <baz>:
80000448:       4e56 0000       linkw %fp,#0
8000044c:       4280            clrl %d0
8000044e:       302e 0012       movew %fp@(18),%d0
80000452:       2d40 0010       movel %d0,%fp@(16)
80000456:       006e 0100 000e  oriw #256,%fp@(14)
8000045c:       4e5e            unlk %fp
8000045e:       60ff ffff ffb0  bral 80000410 <__baz>

This basically does the equivalent of:
        return __baz(a, b | 0x100, (int)(c & 0x0000FFFF));
However, "c" actually wasn’t passed (pushed onto the stack) and is in the
caller’s frame a local variable…

The issue comes from the C library function open(int, int, short) which calls
the syscall __open(int, int | O_LARGEFILE, short) and argument promotion. I
think when doing tail call optimisation, the calling function’s stack should
not be touched. The problem is: baz(int, int, short) takes 3 arguments, but
_can_ be called with two arguments only (how many people use the 3-argument
form of open() unless it’s for O_CREAT anyway?).

Here’s the testcase – the caller is an assembly function, to display the
problem. I don’t “really” speak m68k assembly, only i8086/i386, so feel free to
direct stylistic flames to me, I’m just teaching this myself as I go along. (No
preprocessed source because cpp doesn’t even touch the two files, I think.)

root@ara2:~ # cat tst1.S
        .text

        .globl  foo
        .type   foo, @function
foo:
        /* create a stack frame, reserve 4 bytes for local variables */
        linkw   %fp, #-4
        /* imagine we have locvarname = func() == 0xC0000010 */
        move.l  #0xC0000010, %d0
        move.l  %d0, -4(%fp)
        /* here is the point where foo and bar differ */
        /* now return this value from locvarname */
        move.l  -4(%fp), %d0
        unlk    %fp
        rts   
        .size   foo, . - foo

        .globl  bar
        .type   bar, @function
bar:
        /* create a stack frame, reserve 4 bytes for local variables */
        linkw   %fp, #-4
        /* imagine we have locvarname = func() == 0xC0000010 */
        move.l  #0xC0000010, %d0
        move.l  %d0, -4(%fp)
        /* now call baz(1, 2) */
        move.l  #2, %d1
        move.l  %d1, -(%sp)
        move.l  #1, %d1
        move.l  %d1, -(%sp)
        jsr     baz
        add.l   #8, %sp
        /* now return this value from locvarname */
        move.l  -4(%fp), %d0
        unlk    %fp
        rts
        .size   bar, . - bar
root@ara2:~ # cat tst2.c
extern int printf(const char *, ...);

extern unsigned int foo(void);
extern unsigned int bar(void);

/* imagine open() here */
int __baz(int, int, unsigned short) __attribute__((noinline));
int baz(int, int, unsigned short) __attribute__((noinline));

int
main(void)
{
        unsigned int i;

        i = foo();
        printf("want: C0000010\ngot1: %08X\n", i);
        i = bar();
        printf("got2: %08X\n", i);
        return (0);
}

int
__baz(int a, int b, unsigned short c)
{
        printf("syscall with %d, %d, %d reached\n", a, b, c);
        return (a + b + c);
}

int
baz(int a, int b, unsigned short c)
{
        /* imagine O_LARGEFILE here */
        return (__baz(a, b | 0x100, c));
}