Arsen Arsenović wrote:
Hi,
On Sunday, 25 September 2022 06:24:06 CEST you wrote:
Running ${tool}-dg-prune earlier works both ways, allowing the
callback prune routine to also remove or modify any output that might
confuse the message-matching mechanism. Really, given the documented
(such as it is) feature for ${tool}-dg-prune to be able to declare
the entire test unsupported, I would expect it to filter the results
before messages are checked.
Re-reading the docs, I think you're right in terms of the semantics that
the ${tool}-dg-prune callback should implement.
As a test, I re-ran the GCC test suite after moving the invocation of
${tool}-dg-prune output to be right after ${tool}-dg-test (so, line 700
of lib/dg.exp), which introduced 14361 new FAILs (but also fixed a few
tests).
The tests that it fixed probably need some attention in the form of
moving some "preclean" steps to GCC's ${tool}-dg-test.
It'd appear that all but sixteen of these new fails are a
result of the GCC-specific dg-message and multiline.exp, and some of
those sixteen fail due to trying to test for the presence of strings
they prune (i.e. they want to detect that there's at least one of some
diagnostic, AFAICT). I'll see if I can do anything to remedy these.
Looking at this again, I now agree with your earlier position on the
timing of ${tool}-dg-prune, since ${tool}-dg-test can perform any needed
immediate post-processing itself before returning the results.
Keeping the current interface while fixing the {PASS,FAIL}+UNSUPPORTED
issue is also possible: we can build up a list of results from scanning
the messages, but not actually report them to the framework until after
checking the return value of ${tool}-dg-prune for an "abandon test"
flag. Note that ${tool}-dg-test could produce this flag in lieu of
actual output and ${tool}-dg-prune simply pass it along. (I suspect
that the current Tcl list facilities that would enable this were not yet
available when dg.exp was written more than 20 years ago.)
The main reason I am reluctant to simply accept the "abandon test" flags
directly from ${tool}-dg-test is that that would completely break
backwards compatibility: testsuites depending on this would absolutely
need DejaGnu 1.6.4, and I think we may be able to get most of the way
there (at least getting an UNSUPPORTED result reliably into the summary)
with existing code.
In the meanwhile, I filed a bug so that this issue can be tracked in
the GNU bug tracker.
Excellent! It is now tracked as PR58065. I expect to have a regression
test and a preliminary patch in a few days.
-- Jacob
