On 2025-08-28 16:18, Thiago Jung Bauermann via Gcc wrote:
> "Richard Earnshaw (lists)" <[email protected]> writes:
>
>> On 28/08/2025 15:01, Iain Sandoe wrote:
>>>
>>> … the classification is useful, but the false positive “new fail / old fail
>>> went away”
>>> pairs are a real nuiscance .. hopefully we can have some brainstorming
>>> @cauldron about
>>> ways to deal with this (e.g. fuzzy matching or some smart way to discard
>>> the twinkling
>>> line numbers).
>>>
>>
>> Well really, the compare-tests script should report duplicate results as a
>> problem as
>> well, since
>>
>> PASS: abcd
>> ...
>> PASS: abcd
>>
>> is just a dup pass/fail waiting to happen.
>
> GDB has some magic to detect and report duplicate test names during the
> DejaGNU run:
>
> https://sourceware.org/git?p=binutils-gdb.git;a=blob;f=gdb/testsuite/lib/check-test-names.exp;h=049addd49a93744a832ec0fb5249cd59f3db74f1;hb=refs/heads/master#l64
>
It might be worth to show how it looks in practice too.
And I'll show other neat features the GDB testsuite has, in case they pique
your interest.
Here's an example of the duplicates detection:
$ cat testsuite/gdb.base/dup.exp
clean_restart
gdb_test "print 1"
gdb_test "print 1"
$ make check TESTS="gdb.base/dup.exp"
Running .../src/gdb/testsuite/gdb.base/dup.exp ...
DUPLICATE: gdb.base/dup.exp: print 1 <<< NEW (goes on screen in
addition to gdb.sum for extra visibility)
=== gdb Summary ===
# of expected passes 2
# of duplicate test names 1 <<< NEW
$ cat testsuite/gdb.sum
...
Running .../src/gdb/testsuite/gdb.base/dup.exp ...
PASS: gdb.base/dup.exp: print 1
PASS: gdb.base/dup.exp: print 1
DUPLICATE: gdb.base/dup.exp: print 1
...
$
GDB used to have hundreds of duplicated test names, and we've since fixed them
all, and it is now
hard to introduce any new instance because that mechanism above flags it
immediately.
GDB's testsuite also has similar machinery to flag if a source or build path
appears in gdb.sum,
as those makes it more difficult to compare results across builds. E.g.:
$ make check TESTS="gdb.base/sometest.exp"
...
PASS: gdb.base/sometest.exp: Loaded /path/to/build/testsuite/foo.exe
PATH: gdb.base/sometest.exp: Loaded /path/to/build/testsuite/foo.exe
=== gdb Summary ===
# of paths in test names 1 <<< NEW
It also detects if we end up with stray core dumps after a test run (meaning,
GDB crashed), the
latter useful because sometimes gdb crashes _after_ dejagnu closes gdb. e.g.:
$ make check TESTS="gdb.server/connect-with-no-symbol-file.exp"
...
=== gdb Summary ===
# of unexpected core files 2 <<<< NEW
# of expected passes 8
All of this works with parallel testing as well, where we invoke multiple
expect processes in parallel,
and have a mechanism that lets you mark some tcl procedures as special such
that if any of the expect
processes calls such a proc, the result is cached and reused by the other
processes that later call the
same procedure with the same arguments.
Another thing the GDB testsuite does is treat any info in trailing parenthesis
as "extra" info. For
example, these three are all considered the same test:
PASS: foo
FAIL: foo (timeout)
FAIL: foo (internal-error)
Pedro Alves
