Re: US-CERT Vulnerability Note VU#162289

[Nicola Musatti]

Sorry to be so late in joining this discussion. I'm the person who 
originally notified Mark Mitchell about Microsoft's compiler performing 
this same optimization under certain conditions. Since mailing Mark on 
the subject I tried also VC++ 2008 and it behaves exactly like its 
predecessor. Here's my test program, directly derived from Mark's test case:


int f(char *buf, int len) {
  len = 1 << 30;
  if (buf + len < buf)
return 1;


  return 0;
}

int main()
{
char * b = "0123456789";
for ( int l = 0; l < 1 << 30; ++l )
f(b, l);
}

This is the command line shown by the IDE:
/Ox /GL /D "WIN32" /D "_CONSOLE" /D "_UNICODE" /D "UNICODE" /Gm /EHsc 
/MDd /GS- /Za /FAs /Fa"Debug\\" /Fo"Debug\\" /Fd"Debug\vc90.pdb" /W4 /c 
/Wp64 /Zi /TP .\cert.cpp


This is the assembler listing generated by VC++ 2008. VC++ 2005's 
listing differs only in the first line:


; Listing generated by Microsoft (R) Optimizing Compiler Version 
15.00.21022.08


TITLE   d:\src\cert\cert.cpp
.686P
.XMM
include listing.inc
.model  flat


$SG-5   DB  '0123456789', 00H
PUBLIC  ?f@@[EMAIL PROTECTED]   ; f
; Function compile flags: /Ogtpy
; File d:\src\cert\cert.cpp
_TEXT   SEGMENT
?f@@[EMAIL PROTECTED] PROC  ; f

; 2:   len = 1 << 30;
; 3:   if (buf + len < buf)
; 4: return 1;
; 5:
; 6:
; 7:   return 0;

xor eax, eax

; 8: }

ret 0
?f@@[EMAIL PROTECTED] ENDP  ; f
_TEXT   ENDS
PUBLIC  _main
; Function compile flags: /Ogtpy
_TEXT   SEGMENT
_main   PROC

; 12   :char * b = "0123456789";
; 13   :for ( int l = 0; l < 1 << 30; ++l )
; 14   :f(b, l);
; 15   : }

xor eax, eax
ret 0
_main   ENDP
_TEXT   ENDS
END

I can make my project files available if anybody is interested.

I also tested CodeGear (Borland)'s C++ Builder 2007 compiler and as far 
as I can tell it doesn't perform this optimization. Maybe we should all 
switch to their product ;-)


Cheers,
Nicola Musatti
--
Nicola.Musatti  gmail  com
Home: http://nicola.musatti.googlepages.com/home
Blog: http://wthwdik.wordpress.com/

Re: US-CERT Vulnerability Note VU#162289

[Nicola Musatti]

David Edelsohn wrote:

Nicola,

Please send the project files to Robert Seacord.


Done.

Cheers,
Nicola
--
Nicola.Musatti  gmail  com
Home: http://nicola.musatti.googlepages.com/home
Blog: http://wthwdik.wordpress.com/

Re: US-CERT Vulnerability Note VU#162289

[Nicola Musatti]

Rupert Wood wrote:

Nicola Musatti wrote:


_main   PROC

; 12   :char * b = "0123456789"; ; 13   : for ( int l = 0; l < 1
<< 30; ++l ) ; 14   : f(b, l); ; 15   : }

xor eax, eax ret0 _main ENDP


Note that it optimised away your whole program! It could blank out
f() because it never needed to call it.


That's true, although f() was still compiled to the equivalent of 
'return 0;'.

 This can be made more evident by changing f() to

#include 

int f(char *buf, int len) {
int res = 0;
len = 1 << 30;
if (buf + len < buf)
res =  1;
std::cout << res << '\n';
return res;
}

The resulting f() amounts to

std::cout << 0 << '\n';
return 0;

Which is still inlined into main().
Cheers,
Nicola
--
Nicola.Musatti  gmail  com
Home: http://nicola.musatti.googlepages.com/home
Blog: http://wthwdik.wordpress.com/