[cfe-users] Problem with address sanitizer stack traces

[Rainer Gerhards via cfe-users]

Hi all,

first, thanks for the great tool, especially the address sanitizer. I
have a multi-threaded program, where I get a segfault due to access to
free'd memory in one thread. In the report, I see references to where
this memory block was malloc'ed and free'ed. Unfortunately, these
later two stack traces just include 2 and 3 stack frames, where it
would need to be around 15. Most importantly, the information I really
need is missing due to that.

I tried to explicitely set

ASAN_OPTIONS=verbosity=1:malloc_context_size=20

even though malloc_context_size should be 30 by default. I know
ASAN_OPTIONS is used, because my output now is verbose.

I compile with

-g -W -Wall -Wformat-security -Wshadow -Wcast-align -Wpointer-arith
-Wmissing-format-attribute -fsanitize=address -fno-omit-frame-pointer
-O0

Note that I added -O0 in a desparate effort to try to get to larger
stack traces. Nothing changes if I leave it out.

I than wrote a small single-threaded test program, and there large
stack traces are reported.

Does anyone have any advise on how to get to larger stack traces? Or
at least what could be the root cause of this problem?

Any help would be deeply appreciated.

Thanks,
Rainer
___
cfe-users mailing list
cfe-users@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users

Re: [cfe-users] Segmentation fault with memory sanitizer and OpenMPI's mpirun

[Schlottke-Lakemper, Michael via cfe-users]

Hi Evgenii,

I can confirm that adding 

LIBS=“-lutil”

to the configure command for OpenMPI resolved the reported issue. Thanks again 
for your help!

Michael


> On 23 Nov 2015, at 20:05 , Evgenii Stepanov  wrote:
> 
> I think so. What probably happens here is MSan confuses the configure
> script into thinking that openpty is available without -lutil, but
> what's actually available is just a stub that tries calling the real
> openpty and fails, unless libutil is linked.
> 
> On Mon, Nov 23, 2015 at 10:41 AM, Schlottke-Lakemper, Michael
>  wrote:
>> Hi Evgenii,
>> 
>> Just to clarify: you mean I should re-compile OpenMPI and add “-lutil” to 
>> LDFLAGS at configure time?
>> 
>> Yours
>> 
>> Michael
>> 
>>> On 23 Nov 2015, at 18:07 , Evgenii Stepanov  
>>> wrote:
>>> 
>>> This is caused by missing -lutil.
>>> FTR, http://llvm.org/viewvc/llvm-project?rev=245619&view=rev
>>> 
>>> On Mon, Nov 23, 2015 at 7:33 AM, Schlottke-Lakemper, Michael via
>>> cfe-users  wrote:
 Hi folks,
 
 When running “mpirun” of an msan-instrumented installation of OpenMPI, I 
 get the following error:
 
 $> mpirun -n 1 hostname
 [aia308:48324] *** Process received signal ***
 [aia308:48324] Signal: Segmentation fault (11)
 [aia308:48324] Signal code: Address not mapped (1)
 [aia308:48324] Failing at address: (nil)
 [aia308:48324] [ 0] 
 /pds/opt/openmpi-1.8.7-clang-msan/lib64/libopen-pal.so.6(+0x123ca1)[0x7f9e21c90ca1]
 [aia308:48324] [ 1] mpirun[0x42b602]
 [aia308:48324] [ 2] /lib64/libpthread.so.0(+0xf890)[0x7f9e21250890]
 [aia308:48324] *** End of error message ***
 Segmentation fault
 
 Running it through gdb and printing the stacktrace, I get the following 
 additional information:
 $> gdb -ex r --args mpirun -n 1 hostname
 #0  0x in ?? ()
 #1  0x0042bd5b in __interceptor_openpty () at 
 /pds/opt/install/llvm/llvm-20151121-r253770-src/projects/compiler-rt/lib/msan/msan_interceptors.cc:1355
 #2  0x77705a7c in opal_openpty (amaster=0x7fffacf8, 
 aslave=0x7fffacfc, name=0x0, termp=0x0, winp=0x0) at 
 ../../../openmpi-1.8.7/opal/util/opal_pty.c:116
 #3  0x77b31e9a in orte_iof_base_setup_prefork 
 (opts=0x7ffface8) at 
 ../../../../openmpi-1.8.7/orte/mca/iof/base/iof_base_setup.c:89
 #4  0x7fffefea465a in odls_default_fork_local_proc 
 (context=0x7240bb80, child=0x7240b880, 
 environ_copy=0x7341ec00, jobdat=0x72209d80) at 
 ../../../../../openmpi-1.8.7/orte/mca/odls/default/odls_default_module.c:860
 #5  0x77b3cfb8 in orte_odls_base_default_launch_local (fd=-1, 
 sd=4, cbdata=0x7062da80) at 
 ../../../../openmpi-1.8.7/orte/mca/odls/base/odls_base_default_fns.c:1544
 #6  0x777459d7 in event_process_active_single_queue 
 (base=0x72a0fc80, activeq=0x7100bdc0) at 
 ../../../../../../openmpi-1.8.7/opal/mca/event/libevent2021/libevent/event.c:1367
 #7  0x7773bb92 in event_process_active (base=0x72a0fc80) at 
 ../../../../../../openmpi-1.8.7/opal/mca/event/libevent2021/libevent/event.c:1437
 #8  0x77738fd7 in opal_libevent2021_event_base_loop 
 (base=0x72a0fc80, flags=1) at 
 ../../../../../../openmpi-1.8.7/opal/mca/event/libevent2021/libevent/event.c:1647
 #9  0x0048f651 in orterun (argc=4, argv=0x7fffcea8) at 
 ../../../../openmpi-1.8.7/orte/tools/orterun/orterun.c:1133
 #10 0x0048b20c in main (argc=4, argv=0x7fffcea8) at 
 ../../../../openmpi-1.8.7/orte/tools/orterun/main.c:13
 
 I suspect that has something to do with using some non-msan-instrumented 
 system libraries, but how can I find out which library is the problem and 
 what to do to fix it? Any ideas?
 
 Regards,
 
 Michael
 
 P.S.: I compiled OpenMPI with the following configure command (the 
 wildcard blacklist was necessary because OpenMPI just has too many issues 
 with msan…):
 
 printf "fun:*\n" > blacklist.txt && \
 CC=clang CXX=clang++ \
 CFLAGS="-g -fsanitize=memory -fno-omit-frame-pointer 
 -fsanitize-memory-track-origins -fsanitize-blacklist=`pwd`/blacklist.txt" \
 CXXFLAGS="-g -fsanitize=memory -fno-omit-frame-pointer 
 -fsanitize-memory-track-origins -fsanitize-blacklist=`pwd`/blacklist.txt" \
 ../openmpi-1.8.7/configure --prefix=/pds/opt/openmpi-1.8.7-clang-msan 
 --disable-mpi-fortran
 ___
 cfe-users mailing list
 cfe-users@lists.llvm.org
 http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users
>> 

___
cfe-users mailing list
cfe-users@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users

[cfe-users] Clang memory sanitizer: llvm-symbolizer problem

[Schlottke-Lakemper, Michael via cfe-users]

Hi folks,

When running our msan-instrumented simulation program, instead of a proper 
output I get the following error:

==12089==WARNING: MemorySanitizer: use-of-uninitialized-value
==12089==WARNING: Can't read from symbolizer at fd 14
/pds/opt/llvm/bin/llvm-symbolizer: symbol lookup error: 
/pds/opt/libcxx-20151121-r253770-clang-msan/lib/libc++abi.so.1: undefined 
symbol: __msan_origin_tls
==12089==WARNING: external symbolizer didn't start up correctly!

My setup for using the memory sanitizer is as follows:
- compile libc++/libc++abi with -fsanitize=memory
- compile test program with normal (=uninstrumented) LLVM/Clang installation 
using -fsanitize=memory
- put instrumented libcxx/libcxxabi library dirs into LD_LIBRARY_PATH
- run program

Any idea what I am doing wrong? Thanks a lot in advance!

Michael



--
Michael Schlottke-Lakemper

Chair of Fluid Mechanics and Institute of Aerodynamics
RWTH Aachen University
Wüllnerstraße 5a
52062 Aachen
Germany

Phone: +49 (241) 80 95188
Fax: +49 (241) 80 92257
Mail: 
m.schlottke-lakem...@aia.rwth-aachen.de
Web: http://www.aia.rwth-aachen.de

___
cfe-users mailing list
cfe-users@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users

Re: [cfe-users] Clang memory sanitizer: llvm-symbolizer problem

[Schlottke-Lakemper, Michael via cfe-users]

Short update: I thought the behavior reported below it might be related to the 
fact that llvm-symbolizer picks up the “wrong” (i.e. the msan-implemented) 
version of libc++ once I put it in the LD_LIBRARY_PATH. Thus I tried to compile 
a complete LLVM/Clang stack (with compiler-rt, libcxx, libcxxabi, libomp) using 
-DLLVM_USE_SANITIZER=MemoryWithOrigins. However, this did not work either, as 
apparently during the compilation process the memory sanitizer already comes to 
life and complains about use-of-unitialized values… Thus this approach seems to 
be a dead end.

Michael
On 24 Nov 2015, at 15:28 , Michael Schlottke-Lakemper 
mailto:m.schlottke-lakem...@aia.rwth-aachen.de>>
 wrote:

Hi folks,

When running our msan-instrumented simulation program, instead of a proper 
output I get the following error:

==12089==WARNING: MemorySanitizer: use-of-uninitialized-value
==12089==WARNING: Can't read from symbolizer at fd 14
/pds/opt/llvm/bin/llvm-symbolizer: symbol lookup error: 
/pds/opt/libcxx-20151121-r253770-clang-msan/lib/libc++abi.so.1: undefined 
symbol: __msan_origin_tls
==12089==WARNING: external symbolizer didn't start up correctly!

My setup for using the memory sanitizer is as follows:
- compile libc++/libc++abi with -fsanitize=memory
- compile test program with normal (=uninstrumented) LLVM/Clang installation 
using -fsanitize=memory
- put instrumented libcxx/libcxxabi library dirs into LD_LIBRARY_PATH
- run program

Any idea what I am doing wrong? Thanks a lot in advance!

Michael



--
Michael Schlottke-Lakemper

Chair of Fluid Mechanics and Institute of Aerodynamics
RWTH Aachen University
Wüllnerstraße 5a
52062 Aachen
Germany

Phone: +49 (241) 80 95188
Fax: +49 (241) 80 92257
Mail: 
m.schlottke-lakem...@aia.rwth-aachen.de
Web: http://www.aia.rwth-aachen.de


___
cfe-users mailing list
cfe-users@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users

Re: [cfe-users] Problem with address sanitizer stack traces

[Alexey Samsonov via cfe-users]

Try to add ASAN_OPTIONS=fast_unwind_on_malloc=0. It's possible that stack
traces go through smth. like libstdc++ which is built without frame
pointers.

On Tue, Nov 24, 2015 at 12:49 AM, Rainer Gerhards via cfe-users <
cfe-users@lists.llvm.org> wrote:

> Hi all,
>
> first, thanks for the great tool, especially the address sanitizer. I
> have a multi-threaded program, where I get a segfault due to access to
> free'd memory in one thread. In the report, I see references to where
> this memory block was malloc'ed and free'ed. Unfortunately, these
> later two stack traces just include 2 and 3 stack frames, where it
> would need to be around 15. Most importantly, the information I really
> need is missing due to that.
>
> I tried to explicitely set
>
> ASAN_OPTIONS=verbosity=1:malloc_context_size=20
>
> even though malloc_context_size should be 30 by default. I know
> ASAN_OPTIONS is used, because my output now is verbose.
>
> I compile with
>
> -g -W -Wall -Wformat-security -Wshadow -Wcast-align -Wpointer-arith
> -Wmissing-format-attribute -fsanitize=address -fno-omit-frame-pointer
> -O0
>
> Note that I added -O0 in a desparate effort to try to get to larger
> stack traces. Nothing changes if I leave it out.
>
> I than wrote a small single-threaded test program, and there large
> stack traces are reported.
>
> Does anyone have any advise on how to get to larger stack traces? Or
> at least what could be the root cause of this problem?
>
> Any help would be deeply appreciated.
>
> Thanks,
> Rainer
> ___
> cfe-users mailing list
> cfe-users@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users
>



-- 
Alexey Samsonov
vonos...@gmail.com
___
cfe-users mailing list
cfe-users@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users

Re: [cfe-users] Clang memory sanitizer: llvm-symbolizer problem

[Alexey Samsonov via cfe-users]

On Tue, Nov 24, 2015 at 9:12 AM, Schlottke-Lakemper, Michael via cfe-users <
cfe-users@lists.llvm.org> wrote:

> Short update: I thought the behavior reported below it might be related to
> the fact that llvm-symbolizer picks up the “wrong” (i.e. the
> msan-implemented) version of libc++ once I put it in the LD_LIBRARY_PATH.
>

Yes, I think that's what happening.


> Thus I tried to compile a complete LLVM/Clang stack (with compiler-rt,
> libcxx, libcxxabi, libomp) using -DLLVM_USE_SANITIZER=MemoryWithOrigins.
> However, this did not work either, as apparently during the compilation
> process the memory sanitizer already comes to life and complains about
> use-of-unitialized values… Thus this approach seems to be a dead end.
>
> Michael
>
> On 24 Nov 2015, at 15:28 , Michael Schlottke-Lakemper <
> m.schlottke-lakem...@aia.rwth-aachen.de> wrote:
>
> Hi folks,
>
> When running our msan-instrumented simulation program, instead of a proper
> output I get the following error:
>
> ==12089==WARNING: MemorySanitizer: use-of-uninitialized-value
> ==12089==WARNING: Can't read from symbolizer at fd 14
> /pds/opt/llvm/bin/llvm-symbolizer: symbol lookup error:
> /pds/opt/libcxx-20151121-r253770-clang-msan/lib/libc++abi.so.1: undefined
> symbol: __msan_origin_tls
> ==12089==WARNING: external symbolizer didn't start up correctly!
>
> My setup for using the memory sanitizer is as follows:
> - compile libc++/libc++abi with -fsanitize=memory
> - compile test program with normal (=uninstrumented) LLVM/Clang
> installation using -fsanitize=memory
> - put instrumented libcxx/libcxxabi library dirs into LD_LIBRARY_PATH
>
> Can you pass -Wl,-rpath when you link your executable, to specify the path
to instrumented libc++/libc++abi?


> - run program
>
> Any idea what I am doing wrong? Thanks a lot in advance!
>
> Michael
>
>
>
> --
> Michael Schlottke-Lakemper
>
> Chair of Fluid Mechanics and Institute of Aerodynamics
> RWTH Aachen University
> Wüllnerstraße 5a
> 52062 Aachen
> Germany
>
> Phone: +49 (241) 80 95188
> Fax: +49 (241) 80 92257
> Mail: m.schlottke-lakem...@aia.rwth-aachen.de
> Web: http://www.aia.rwth-aachen.de
>
>
>
> ___
> cfe-users mailing list
> cfe-users@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users
>
>


-- 
Alexey Samsonov
vonos...@gmail.com
___
cfe-users mailing list
cfe-users@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-users