Ralf Wildenhues <[EMAIL PROTECTED]> wrote: > Hello everyone, > > These four patches implement parallel execution of TESTS in Automake, > adapted from the check.mk file Akim Demaille posted earlier. I would > be delighted about review, comments, bug reports, any feedback. There > are a couple of questions at the end.
Great! Thanks for all the work, Ralf. > Design considerations: > > 1) should be similar to current TESTS interface, and not enabled without > developer consent (because of inevitable semantic differences) > 2) should allow running of a subset of tests easily, preferably as > make check TESTS="..." > 3) should work with different test naming styles: > - *.test (with log files named s/\.test$/.log/) > - no particular suffix (log files will be s/$/.log/) > - should work with executables (e.g., TESTS = $(check_PROGRAMS)) > - arbitrary other suffixes > 4) should work with @substituted@ values in TESTS, > (here a limitation upon nonempty suffixes is ok) > 5) parallel test execution, > 6) lazy test completion (do not rerun already-run tests), > 7) portable make (of course!), portable shell, etc., > 8) should allow for hard errors that stop test suite execution, > 9) should allow for dependency relations (ordering) between tests, > 10) test names may have directory components, > ------ > 11) allow for extensibility like multiple test suites, each with their > own set of logs. > 12) allow for additional output on stdout/stderr? > example: test was skipped because of $reason. This would be nice. It'd would be useful also to mark as "surprising" or "highly undesirable" the results of certain tests. Better than a simple "yes" or "no". ... > Questions, and issues I'm not sure about yet: > > - improvements on the naming scheme would be good; some of the new > variables seem quite ad-hoc > > - shouldn't test-suite.log also be created if all tests passed? Sounds reasonable. > - should skipped tests show up in test-suite.log? I'd find that useful. I'll give it a spin in coreutils soon.
