[lldb-dev] Some API test failures are really opaque/could be improved
Wondering if anyone else has encountered/dealt with debugging lldb test
failures like the one shown at the end of this email ("AssertionError: 10
!= 5" in "test.assertEqual(process.GetState(), lldb.eStateStopped)" while
checking that a breakpoint was reached)
Is there anything that could be done to improve the debuggability of such
failures? Logging standard output/error from the lldb process or any other
logging it might have? At least for one of these I managed to figure it out
by running lldb directly on the binary and finding that the binary couldn't
find libc++.so (that's a story for another bug/email thread, looks like
maybe lldb libc++ pretty printer tests are using the system installed
libc++, not the just-built libc++ (& thus not running if there is no system
installed libc++)). But my current failure like this seems a bit more
inscrutible and I'm still looking into it.
Looks like dotest.py has some sense of logging (it has a --log-success
option which says preserves the logs even on failure - though the output of
dotest.py, at least for me, has no mention of logs, log files, or where
they might be located, and looking at the source points to some sort of
".log" files... ah, found some)
So, yeah, there do seem to be some Failure.log, SkippedTest.log, etc -
should dotest print something about the paths to those files when it exits
with failure, maybe?
FAIL: lldb-api ::
functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py
(23 of 23)
TEST 'lldb-api ::
functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py'
FAILED
Script:
--
/usr/bin/python3
/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u
CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env
OBJCOPY=/usr/bin/objcopy --env
LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib
--arch x86_64 --build-dir
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex
--lldb-module-cache-dir
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api
--clang-module-cache-dir
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api
--executable
/usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler
/usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang
--dsymutil
/usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil
--llvm-tools-dir
/usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir
/usr/local/google/home/blaikie/dev/llvm/build/release/./lib
/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set
-p TestDataFormatterLibcxxSet.py
--
Exit Code: 1
Command Output (stdout):
--
lldb version 14.0.0 (g...@github.com:llvm/llvm-project.git revision
6176fda3f992b5086302b3826aa0636135cc4cc0)
clang revision 6176fda3f992b5086302b3826aa0636135cc4cc0
llvm revision 6176fda3f992b5086302b3826aa0636135cc4cc0
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver',
'objc']
--
Command Output (stderr):
--
UNSUPPORTED: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_ref_and_ptr_dsym
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
not fall in any category of interest for this run)
FAIL: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_ref_and_ptr_dwarf
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_ref_and_ptr_dwo
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_ref_and_ptr_gmodules
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
not fall in any category of interest for this run)
UNSUPPORTED: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_with_run_command_dsym
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
not fall in any category of interest for this run)
FAIL: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_with_run_command_dwarf
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_with_run_command_dwo
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB
(/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
test_with_run_command_gmodules
(TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
not fall in any category of interest for this run)
==
[lldb-dev] libc++ pretty printer test dependencies
So I'm trying to run a clean "ninja check-lldb" and I'm running into some difficulties with the libc++ pretty printer tests. 1) They're "unsupported" if my host compiler is gcc: $ ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx [2/3] Running lit suite .../src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx Testing Time: 1.57s Unsupported: 23 Looking at the logs (see other thread: perhaps those logs should actually be part of the test output - especially for buildbots where you'd have no ability to read separate log files): " unittest2.case.SkipTest: could not find library matching 'libc\+\+' in target a.out" So, looks like it built with the just-built clang, but without libc++? (since libc++ isn't built yet in this tree) - but the logs don't show the commands that built the binary - should I be able to find that somewhere? It seems important for debugging exactly what's under test, etc. 2) Oh, but if I explicitly `ninja cxx` the tests fail instead of "unsupported" Now they fail, rather than unsupported. The log isn't especially helpful so far as I can see: ... runCmd: setting set target.prefer-dynamic-value no-dynamic-values output: FAIL >>: success Traceback (most recent call last): File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method return attrvalue(self) File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py", line 53, in test_with_run_command (self.target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint( File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint return run_to_breakpoint_do_run(test, target, breakpoint, launch_info, File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run test.assertEqual(process.GetState(), lldb.eStateStopped) AssertionError: 10 != 5 Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang Session info generated @ Sun Oct 17 22:23:34 2021 But if I run lldb on the binary (which isn't mentioned in the logs... probably should be?) under test: $ ./bin/lldb ./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.test_ref_and_ptr_dwarf/a.out (lldb) target create "./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.test_ref_and_ptr_dwarf/a.out" Current executable set to 'build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.test_ref_and_ptr_dwarf/a.out' (x86_64). (lldb) r Process 1896861 launched: 'build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.test_ref_and_ptr_dwarf/a.out' (x86_64) build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.test_ref_and_ptr_dwarf/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory Process 1896861 exited with status = 127 (0x007f) OK, so this looks like it's related to libc++.so.1 not being in the ld library path - but there's no rpath or LD_LIBRARY_PATH to find the library. The libc++ tests build test binaries with "-Wl,-rpath,/usr/local/google/home/blaikie/dev/llvm/build/release/./lib/x86_64-unknown-linux-gnu" Ah, here we go - by modifying the test's source file so it would fail to compile, I am able to observe the compilation command: build/release/bin/clang -std=c++11 -g -O0 -fno-builtin -m64 -Isrc/lldb/packages/Python/lldbsuite/test/make/../../../../../include -Isrc/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/vector -Isrc/lldb/packages/Python/lldbsuite/test/make -include src/lldb/packages/Python/lldbsuite/test/make/test_common.h -fno-limit-debug-info -gsplit-dwarf -O0 -DLLDB_USING_LIBCPP -stdlib=libc++ --driver-mode=g++ -MT main.o -MD -MP -MF main.d -c -o main.o src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/vector/main.cpp So... how does this work for everyone else? I'm not sure how it's meant to work. >From some offline discussion with Pavel: This is generally broken - it either uses the system libc++.so (making the build non-hermetic), or if you don't enable libc++ in CMake the tests flag as unsupported/don't fail (though other tests not specifically testing libc++ but using any standard library features are still non-hermetic, they'll be using the system C++ standard library) That's all unfortunate... would love to know if anyone's got plans/ideas/priorit
