Eric Gallager wrote:
Many command-line utilities have a "--dry-run" flag to print what would be done by running them without actually performing any changes. I was thinking one would be particularly useful for running the GCC testsuite (to put in RUNTESTFLAGS), since it is quite large and running it can take quite a long time, yet sometimes there are reasons to run it besides actually running it (e.g., getting a list of tests run). For example, https://gcc.gnu.org/install/test.html says: "To get a list of the possible *.exp files, pipe the output of ‘make check’ into a file and look at the ‘Running … .exp’ lines," which is something that I thought being able to do "RUNTEST_FLAGS=--dry-run" would be useful for.
The main problem with this suggestion is technical: the DejaGnu framework does not know what tests are in the testsuite (adding means to declare this is one of my eventual plans, if I can find a way to make it fit DejaGnu's architecture). The problem is that it is possible for a test to silently misfire and produce no result at all, depending on how the test scripts are written, which comes back to the problem that makes '--dry-run' difficult for DejaGnu: we could list the test scripts that would be run without actually running them, but without support in the testsuite there is no way to know what tests each script would do, and there is infrastructure in DejaGnu to support test scripts that use some (potentially variable) set of external files. Without actually running the tests, the framework knows nothing about those files.
Another possibility would be to replace host_execute, target_compile, and others with stubs when in dry-run mode and ignore results (replacing the procedures that actually run tests _should_ cause numerous failures in most testsuites); this would allow to evaluate the test scripts, but may cause confusion, especially in more advanced testsuites, like GCC.
For your use case, is this really needed? DejaGnu has logic that is (or should be) very similar to `find testsuite/ -type d -name testsuite -prune -o -type d -name config -prune -o -type d -name lib -prune -o -name '*.exp' -print` for locating test scripts to run, except that DejaGnu additionally narrows the list to files appropriate for the selected tool to be tested and also excludes directories associated with Git, Subversion, CVS, RCS, and SCCS from the search. In a large package like GCC that has many testsuites, you could first find the testsuites with a simple `find . -type d -name testsuite -print`.
DejaGnu is beginning to accumulate a selection of auxiliary tools, and finding the scripts that runtest could possibly use might be a helpful feature there. Perhaps `dejagnu list testsuites` to wrap the second find command I mentioned and `dejagnu list test scripts` for the more detailed search of one specific testsuite (always or usually ./testsuite)?
Let me know if you need me to find specific examples of software that already has the kind of "--dry-run" flag I'm thinking of.
I suspect that GNU make is a good example. -- Jacob
