Hi, I've been working on some automake tests for grub, and ran in to a bug. I noticed that when specifying a subset of tests to run via the TESTS environment variable, I could have all tests complete successfully and yet still have make return an error.
Here's a sequence of commands that should reproduce the error: git clone https://git.savannah.gnu.org/git/grub.git cd grub ./bootstrap ./configure make env TESTS=grub_cmd_echo make -e check VERBOSE=yes It turns out that make check is being recursively run in the grub-core directory, as expected. The TESTS environment variable is the same in both invocations, also as expected. However, the test program is not in the grub-core subdirectory. So the test is run successfully in the root make, and not found in the subdirectory, thus generating an error. In fact, grub has no tests in the grub-core subdirectory, but the harness is run because its part of automake, even though I do not need or want it to. I've attached a patch which fixes this issue by ignoring tests which do not exist. I'm not sure this patch takes care of everything needed, for instance EXEEXT, to check for the existence of the test program. Grub does not use any TEST_EXTENSIONS, so this works. This will change make it so TESTS which should exist, but do not, will not cause an error, whereas currently it will. I'm not sure how this situation would arise, because make should require the existence as a dependency. If there is an error in the TESTS variable (eg. a typo in a test name), this will not cause an error either, which I deem undesirable. However, I think this is an acceptable trade-off. Perhaps a better solution would be to not recursively run make check if TESTS is defined and run all given tests from the root make. Tests which are run in recursive make invocations when TESTS is not defined can be run via TESTS by specifying a path component (eg. TESTS="grub-core/test-program"). This could be a problem if test programs assume they are run from the directory in which they reside (in which case, we could chdir to the directory and run the program). Any thoughts? Glenn
diff --git a/lib/am/check.am b/lib/am/check.am index 449708742..94021775e 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -416,8 +416,8 @@ check-TESTS: %CHECK_DEPS% ## we rely on .PHONY to work portably. @test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) @set +e; $(am__set_TESTS_bases); \ - log_list=`for i in $$bases; do echo $$i.log; done`; \ - trs_list=`for i in $$bases; do echo $$i.trs; done`; \ + log_list=`for i in $$bases; do test -x $$i && echo $$i.log; done`; \ + trs_list=`for i in $$bases; do test -x $$i && echo $$i.trs; done`; \ ## Remove newlines and normalize whitespace. Trailing (and possibly ## leading) whitespace is known to cause segmentation faults on ## Solaris 10 XPG4 make.
make[3]: Leaving directory '/home/user/grub.git' make check-TESTS make[3]: Entering directory '/home/user/grub.git' make[4]: Entering directory '/home/user/grub.git' PASS: grub_cmd_echo ============================================================================ Testsuite summary for GRUB 2.05 ============================================================================ # TOTAL: 1 # PASS: 1 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ make[4]: Leaving directory '/home/user/grub.git' make[3]: Leaving directory '/home/user/grub.git' make[2]: Leaving directory '/home/user/grub.git' Making check in grub-core make[2]: Entering directory '/home/user/grub.git/grub-core' make check-am make[3]: Entering directory '/home/user/grub.git/grub-core' make make[4]: Entering directory '/home/user/grub.git/grub-core' make all-am make[5]: Entering directory '/home/user/grub.git/grub-core' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/user/grub.git/grub-core' make[4]: Leaving directory '/home/user/grub.git/grub-core' make check-TESTS make[4]: Entering directory '/home/user/grub.git/grub-core' make[5]: Entering directory '/home/user/grub.git/grub-core' make[6]: Entering directory '/home/user/grub.git/grub-core' make[6]: Nothing to be done for 'grub_cmd_echo.log'. make[6]: Leaving directory '/home/user/grub.git/grub-core' fatal: making test-suite.log: failed to create grub_cmd_echo.trs fatal: making test-suite.log: failed to create grub_cmd_echo.log make[5]: *** [Makefile:43107: test-suite.log] Error 1 make[5]: Leaving directory '/home/user/grub.git/grub-core' make[4]: *** [Makefile:43216: check-TESTS] Error 2 make[4]: Leaving directory '/home/user/grub.git/grub-core' make[3]: *** [Makefile:43283: check-am] Error 2 make[3]: Leaving directory '/home/user/grub.git/grub-core' make[2]: *** [Makefile:43285: check] Error 2 make[2]: Leaving directory '/home/user/grub.git/grub-core' make[1]: *** [Makefile:11920: check-recursive] Error 1 make[1]: Leaving directory '/home/user/grub.git' make: *** [Makefile:12944: check] Error 2
