https://llvm.org/bugs/show_bug.cgi?id=24869
Bug ID: 24869
Summary: dotest.py reports the wrong number of test cases in
mutiprocess mode.
Product: lldb
Version: unspecified
Hardware: PC
OS: All
Status: NEW
Severity: normal
Priority: P
Component: All Bugs
Assignee: lldb-dev@lists.llvm.org
Reporter: dawn+bugzi...@burble.org
CC: llvm-b...@lists.llvm.org
Classification: Unclassified
This appears to result from dotest.py failing to count the test cases from
entire test files, as described in a run on a narrowed example below (following
the problem description).
** problem description and steps **
Run "dotest.py -v --output-on-success" and count the total test cases via
adding
up all the Ns in the lines:
Ran N tests in .*
and you get the correct total, 1388 on OSX.
Compare that to the final count of 838 from:
Ran N test cases .*
and you see a difference of 550 tests.
Commands used to run the tests and collect the results are as follows:
cd llvm/tools/lldb/test
./dotest.py -v --output-on-success 2>&1 | tee $INSTALLDIR/lldb_test_out.log
export lldb_failures=`grep -E "^RESULT:" lldb_test_out.log | grep failures
| awk '{count+=$5} END {print count}'` || true
export lldb_errors=`grep -E "^RESULT:" lldb_test_out.log | grep errors |
awk '{count+=$7} END {print count}'` || true
export lldb_total=`grep -E "^Ran [0-9]+ tests? in" lldb_test_out.log | awk
'{count+=$2} END {print count}'`
export lldb_total_cases=`grep -E "^Ran [0-9]+ test cases" lldb_test_out.log
| awk '{count+=$2} END {print count}'`
On OSX, I get:
lldb_failures=4
lldb_errors=0
lldb_total=1388
lldb_total_cases=838
** narrowed example **
When run on the following narrowed subset of tests, you can see that dotest.py
is not counting any test cases from an entire test file.
Test on just 4 types test suites:
rm -rf api arm_emulation attic benchmarks driver
rm -rf example expression_command functionalities help lang
rm -rf linux logging macosx python_api tools
rm -rf settings source-manager terminal warnings
rm -f types/TestIntegerTypes.py
You should have only the following tests left:
types/AbstractBase.py
types/HideTestFailures.py
types/TestFloatTypes.py
types/TestFloatTypesExpr.py
types/TestIntegerTypesExpr.py
types/TestRecursiveTypes.py
When dotest.py is run with the --no-multiprocess option, it correctly counts
46, but counts only 16 otherwise, as can been seen in the relevant test output:
./dotest.py -v --output-on-success --no-multiprocess:
Ran 46 tests in 76.124s
./dotest.py -v --output-on-success:
Testing: 4 test suites, 4 threads
Ran 6 tests in 7.090s
[TestFloatTypes.py PASSED]
Ran 6 tests in 11.230s
[TestFloatTypesExpr.py PASSED]
Ran 30 tests in 55.200s
[TestIntegerTypesExpr.py PASSED]
Ran 4 tests in 3.608s
[TestRecursiveTypes.py PASSED]
Ran 4 test suites (0 failed) (0.000000%)
Ran 16 test cases (0 failed) (0.000000%)
All 30 test cases from TestIntegerTypesExpr.py appear to have been omitted from
the total.
With run on only 3 test suites, both correctly count 16 test cases:
rm -f types/TestIntegerTypesExpr.py
./dotest.py -v --output-on-success --no-multiprocess:
Ran 16 tests in 21.426s
./dotest.py -v --output-on-success:
Testing: 3 test suites, 4 threads
Ran 6 tests in 7.245s
[TestFloatTypes.py PASSED]
Ran 6 tests in 11.474s
[TestFloatTypesExpr.py PASSED]
Ran 4 tests in 3.599s
[TestRecursiveTypes.py PASSED]
3 out of 3 test suites processed - TestRecursiveTypes.py
Ran 3 test suites (0 failed) (0.000000%)
Ran 16 test cases (0 failed) (0.000000%)
--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
