Allow suffix-based definition of custom "driver script" for the
test scripts. These driver scripts will be responsible of
launching the tests (or their corresponding $(LOG_COMPILER), if
they have an associated one), interpreting and displaying the
test results, and writing the `.log' files.
This new API should allow easy and flexible use of different
test protocols in the future; in particular, we plan to use it
to implement TAP and SubUnit harnesses.
Note that no new documentation is added by this change; that is
be left for follow-up changes.
* automake.in (handle_tests): Define default for $(LOG_DRIVER),
and, for any registered test extension `<ext>', define defaults
for $(<ext>_LOG_DRIVER). Substitute %DRIVER% using these new
variables, instead of the old internal $(am__test_driver). When
processing check2.am, also substitute %DRIVER_FLAGS%.
Require auxiliary script `test-driver' only if no driver has been
explicitly defined for the test script kinds.
* am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
?!GENERIC?%OBJ%): Pass the %DRIVER_FLAGS% to the %DRIVER% call.
* tests/parallel-tests-no-extra-driver.test: New test.
* tests/test-driver-custom.test: Likewise.
* tests/test-driver-custom-xfail-tests.test: Likewise.
* tests/test-driver-fail.test: Likewise.
* tests/Makefile.am: Update.
* NEWS: Update.
---
ChangeLog | 29 +++++
NEWS | 12 ++-
automake.in | 41 ++++++--
lib/Automake/tests/Makefile.in | 4 +-
lib/am/check2.am | 4 +-
tests/Makefile.am | 4 +
tests/Makefile.in | 8 +-
tests/test-driver-custom-no-extra-driver.test | 63 +++++++++++
tests/test-driver-custom-xfail-tests.test | 147 +++++++++++++++++++++++++
tests/test-driver-custom.test | 140 +++++++++++++++++++++++
tests/test-driver-fail.test | 65 +++++++++++
11 files changed, 500 insertions(+), 17 deletions(-)
create mode 100755 tests/test-driver-custom-no-extra-driver.test
create mode 100755 tests/test-driver-custom-xfail-tests.test
create mode 100755 tests/test-driver-custom.test
create mode 100755 tests/test-driver-fail.test
diff --git a/ChangeLog b/ChangeLog
index 7f0133e..ebd7e19 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,34 @@
2011-06-21 Stefano Lattarini <stefano.lattar...@gmail.com>
+ parallel-tests: allow custom driver scripts
+ Allow suffix-based definition of custom "driver script" for the
+ test scripts. These driver scripts will be responsible of
+ launching the tests (or their corresponding $(LOG_COMPILER), if
+ they have an associated one), interpreting and displaying the
+ test results, and writing the `.log' files.
+ This new API should allow easy and flexible use of different
+ test protocols in the future; in particular, we plan to use it
+ to implement TAP and SubUnit harnesses.
+ Note that no new documentation is added by this change; that is
+ be left for follow-up changes.
+ * automake.in (handle_tests): Define default for $(LOG_DRIVER),
+ and, for any registered test extension `<ext>', define defaults
+ for $(<ext>_LOG_DRIVER). Substitute %DRIVER% using these new
+ variables, instead of the old internal $(am__test_driver). When
+ processing check2.am, also substitute %DRIVER_FLAGS%.
+ Require auxiliary script `test-driver' only if no driver has been
+ explicitly defined for the test script kinds.
+ * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
+ ?!GENERIC?%OBJ%): Pass the %DRIVER_FLAGS% to the %DRIVER% call.
+ * tests/parallel-tests-no-extra-driver.test: New test.
+ * tests/test-driver-custom.test: Likewise.
+ * tests/test-driver-custom-xfail-tests.test: Likewise.
+ * tests/test-driver-fail.test: Likewise.
+ * tests/Makefile.am: Update.
+ * NEWS: Update.
+
+2011-06-21 Stefano Lattarini <stefano.lattar...@gmail.com>
+
parallel-tests: add auxiliary script 'test-driver', refactor
This refactoring should cause no API of functionality change,
and is meant only to simplify the future implementation of TAP
diff --git a/NEWS b/NEWS
index 41d0dfe..37af53a 100644
--- a/NEWS
+++ b/NEWS
@@ -13,9 +13,10 @@ New in 1.11a:
* Changes to Automake-generated testsuite harnesses:
- - The parallel-tests driver is now implemented (partly at least) with
- the help of automake-provided auxiliary scripts (e.g., `test-driver'),
- instead of relying entirely on code in the generated Makefile.in.
+ - The default testsuite driver offered by the 'parallel-tests' option is
+ now implemented (partly at least) with the help of automake-provided
+ auxiliary scripts (e.g., `test-driver'), instead of relying entirely
+ on code in the generated Makefile.in.
This has two noteworthy implications. The first one is that projects
using the `parallel-tests' option should now either run automake with
the `--add-missing' option, or manually copy the `test-driver' script
@@ -52,6 +53,11 @@ New in 1.11a:
}
LOG_COMPILER = run_with_per_or_shell
+ - The package authors can now use customary testsuite drivers within
+ the framework provided by the 'parallel-tests' testsuite harness.
+ Consistently with the existing syntax, this can be done by defining
+ special makefile variables `LOG_DRIVER' and `<ext>_LOG_DRIVER'.
+
* WARNING: Future backward-incompatibilities!
- The Automake support for automatic de-ANSI-fication will be removed in
diff --git a/automake.in b/automake.in
index e1c7a92..340d4e3 100644
--- a/automake.in
+++ b/automake.in
@@ -4991,11 +4991,6 @@ sub handle_tests
if (my $parallel_tests = option 'parallel-tests')
{
- require_conf_file ($parallel_tests->{position}, FOREIGN,
- 'test-driver');
- define_variable ('am__test_driver',
- "\$(SHELL) $am_config_aux_dir/test-driver",
- INTERNAL);
define_variable ('TEST_SUITE_LOG', 'test-suite.log', INTERNAL);
define_variable ('TEST_SUITE_HTML', '$(TEST_SUITE_LOG:.log=.html)',
INTERNAL);
my $suff = '.test';
@@ -5039,6 +5034,20 @@ sub handle_tests
if substr ($obj, - length ($test_suffix)) eq $test_suffix;
}
$obj .= '.log';
+ # The "test driver" program, deputed to handle tests protocol
used by
+ # test scripts. By default, it's assumed that no protocol is
used,
+ # so we fall back to the old "parallel-tests" behaviour,
implemented
+ # by the `test-driver' auxiliary script.
+ if (! var 'LOG_DRIVER')
+ {
+ require_conf_file ($parallel_tests->{position}, FOREIGN,
+ 'test-driver');
+ define_variable ('LOG_DRIVER',
+ "\$(SHELL)
$am_config_aux_dir/test-driver",
+ INTERNAL);
+ }
+ my $driver = '$(LOG_DRIVER)';
+ my $driver_flags = '$(AM_LOG_DRIVER_FLAGS) $(LOG_DRIVER_FLAGS)';
my $compile = 'LOG_COMPILE';
define_variable ($compile,
'$(LOG_COMPILER) $(AM_LOG_FLAGS)
$(LOG_FLAGS)', INTERNAL);
@@ -5046,7 +5055,8 @@ sub handle_tests
GENERIC => 0,
OBJ => $obj,
SOURCE => $val,
- DRIVER => '$(am__test_driver)',
+ DRIVER => $driver,
+ DRIVER_FLAGS => $driver_flags,
COMPILE =>'$(' . $compile . ')',
EXT => '',
am__EXEEXT => 'FALSE');
@@ -5077,6 +5087,18 @@ sub handle_tests
{
(my $ext = $test_suffix) =~ s/^\.//;
$ext = uc $ext;
+ # See comments about definition of LOG_DRIVER, above.
+ if (! var "${ext}_LOG_DRIVER")
+ {
+ require_conf_file ($parallel_tests->{position}, FOREIGN,
+ 'test-driver');
+ define_variable ("${ext}_LOG_DRIVER",
+ "\$(SHELL)
$am_config_aux_dir/test-driver",
+ INTERNAL);
+ }
+ my $driver = '$(' . $ext . '_LOG_DRIVER)';
+ my $driver_flags = '$(AM_' . $ext . '_LOG_DRIVER_FLAGS) ' .
+ '$(' . $ext . '_LOG_DRIVER_FLAGS)';
my $compile = $ext . '_LOG_COMPILE';
define_variable ($compile,
'$(' . $ext . '_LOG_COMPILER) $(AM_' . $ext
. '_LOG_FLAGS)'
@@ -5086,16 +5108,19 @@ sub handle_tests
GENERIC => 1,
OBJ => '',
SOURCE => '$<',
- DRIVER =>
'$(am__test_driver)',
+ DRIVER => $driver,
+ DRIVER_FLAGS => $driver_flags,
COMPILE => '$(' . $compile .
')',
EXT => $test_suffix,
am__EXEEXT => $am_exeext);
}
}
+ # FIXME: this is partly out-of-date w.r.t. the rest of the
+ # FIXME: code now ... what is the best fix?
define_variable ('TEST_LOGS_TMP', '$(TEST_LOGS:.log=.log-t)',
INTERNAL);
-
$clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN;
+
$clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_HTML)'} = MOSTLY_CLEAN;
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 6ae57f1..ed1ddbe 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -171,11 +171,11 @@ esac;
\
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
-am__test_driver = $(SHELL) $(top_srcdir)/lib/test-driver
TEST_SUITE_LOG = test-suite.log
TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
am__test_logs1 = $(TESTS:=.log)
TEST_LOGS = $(am__test_logs1:.pl.log=.log)
+PL_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
PL_LOG_COMPILE = $(PL_LOG_COMPILER) $(AM_PL_LOG_FLAGS) $(PL_LOG_FLAGS)
TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
@@ -467,7 +467,7 @@ recheck recheck-html:
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.pl.log:
@p='$<'; $(am__check_pre) \
- $(am__test_driver) $(am__test_driver_flags) -- \
+ $(PL_LOG_DRIVER) $(am__test_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS)
$(PL_LOG_DRIVER_FLAGS) -- \
$(PL_LOG_COMPILE) "$$tst"
distdir: $(DISTFILES)
diff --git a/lib/am/check2.am b/lib/am/check2.am
index f3116c8..b7ed0ee 100644
--- a/lib/am/check2.am
+++ b/lib/am/check2.am
@@ -18,7 +18,7 @@
?GENERIC?%EXT%.log:
?!GENERIC?%OBJ%: %SOURCE%
@p='%SOURCE%'; $(am__check_pre) \
- %DRIVER% $(am__test_driver_flags) -- \
+ %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
%COMPILE% "$$tst"
## If no programs are built in this package, then this rule is removed
@@ -28,6 +28,6 @@
if %am__EXEEXT%
?GENERIC?%EXT%$(EXEEXT).log:
@p='%SOURCE%'; $(am__check_pre) \
- %DRIVER% $(am__test_driver_flags) -- \
+ %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
%COMPILE% "$$tst"
endif %am__EXEEXT%
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 0d2c514..9c61022 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -724,6 +724,10 @@ parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
parallel-test-driver-install.test \
+test-driver-custom-no-extra-driver.test \
+test-driver-custom.test \
+test-driver-custom-xfail-tests.test \
+test-driver-fail.test \
parse.test \
percent.test \
percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 196ee5c..c054423 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -176,12 +176,12 @@ esac;
\
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
-am__test_driver = $(SHELL) $(top_srcdir)/lib/test-driver
TEST_SUITE_LOG = test-suite.log
TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
TEST_EXTENSIONS = .test
am__test_logs1 = $(TESTS:=.log)
TEST_LOGS = $(am__test_logs1:.test.log=.log)
+TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
TEST_LOG_COMPILE = $(TEST_LOG_COMPILER) $(AM_TEST_LOG_FLAGS) \
$(TEST_LOG_FLAGS)
TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
@@ -984,6 +984,10 @@ parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
parallel-test-driver-install.test \
+test-driver-custom-no-extra-driver.test \
+test-driver-custom.test \
+test-driver-custom-xfail-tests.test \
+test-driver-fail.test \
parse.test \
percent.test \
percent2.test \
@@ -1497,7 +1501,7 @@ recheck recheck-html:
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.test.log:
@p='$<'; $(am__check_pre) \
- $(am__test_driver) $(am__test_driver_flags) -- \
+ $(TEST_LOG_DRIVER) $(am__test_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- \
$(TEST_LOG_COMPILE) "$$tst"
distdir: $(DISTFILES)
diff --git a/tests/test-driver-custom-no-extra-driver.test
b/tests/test-driver-custom-no-extra-driver.test
new file mode 100755
index 0000000..f4c996e
--- /dev/null
+++ b/tests/test-driver-custom-no-extra-driver.test
@@ -0,0 +1,63 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that auxiliary script 'pt-driver' doesn't get needlessly
+# installed or referenced if it's not used, i.e., if the user has
+# defined his own `*LOG_DRIVER' variables.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<'END'
+AC_PROG_CC
+AC_CONFIG_FILES([sub1/Makefile sub2/Makefile])
+AC_SUBST([X_LOG_DRIVER], [none])
+AC_OUTPUT
+END
+
+mkdir sub1 sub2
+
+cat > Makefile.am <<'END'
+SUBDIRS = sub1 sub2
+LOG_DRIVER =
+TEST_LOG_DRIVER = :
+TESTS = foo bar.test
+END
+
+cat > sub1/Makefile.am <<'END'
+TEST_EXTENSIONS = .x .sh .pl
+SH_LOG_DRIVER = dummy1
+PL_LOG_DRIVER = dummy2
+TESTS = a.pl b.sh c.x
+END
+
+cat > sub2/Makefile.am <<'END'
+TEST_EXTENSIONS = .bar
+LOG_DRIVER = x
+BAR_LOG_DRIVER = y
+TESTS = 1 2.bar 3.test 4.t 5.tt $(check_PROGRAMS)
+check_PROGRAMS = p1 p2$(EXEEXT) p3.bar p4.suf
+END
+
+$ACLOCAL
+
+for opts in '' '--add-missing' '-a -c'; do
+ $AUTOMAKE $opts
+ $FGREP pt-driver Makefile.in sub1/Makefile.in sub2/Makefile.in && Exit 1
+ find . | $FGREP pt-driver && Exit 1
+done
+
+:
diff --git a/tests/test-driver-custom-xfail-tests.test
b/tests/test-driver-custom-xfail-tests.test
new file mode 100755
index 0000000..0d10594
--- /dev/null
+++ b/tests/test-driver-custom-xfail-tests.test
@@ -0,0 +1,147 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: "abstract" XFAIL_TESTS support.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<'END'
+AC_SUBST([nihil], [])
+AC_SUBST([ac_xfail_tests], ['x5.test x6$(test_suffix)'])
+AC_CONFIG_FILES([sub1/Makefile sub2/Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+SUBDIRS = . sub1 sub2
+TEST_LOG_DRIVER = $(srcdir)/td
+TESTS = pass.test xfail.test
+XFAIL_TESTS = xfail.test
+END
+
+mkdir sub1 sub2
+
+cat > sub1/Makefile.am <<END
+empty =
+
+TEST_LOG_DRIVER = \$(top_srcdir)/td
+
+# XFAIL_TESTS should gracefully handle TAB characters, and multiple
+# whitespaces.
+XFAIL_TESTS =\$(empty)${tab}x1.test x2.test${tab}x3.test${tab}\
+x4.test ${tab} x5.test x6.test${tab}\$(empty)
+
+TESTS = pass.test x1.test x2.test x3.test x4.test x5.test x6.test
+END
+
+cat > sub2/Makefile.am <<'END'
+AUTOMAKE_OPTIONS = -Wno-portability-recursive
+
+TEST_LOG_DRIVER = $(srcdir)/../td
+
+# XFAIL_TESTS should gracefully AC_SUBST @substitution@ and
+# make variables indirections.
+an_xfail_test = x1.test
+test_suffix = .test
+v0 = x3.test
+v1 = v
+v2 = 0
+XFAIL_TESTS = $(an_xfail_test) x2.test @nihil@ x3${test_suffix}
+XFAIL_TESTS += $($(v1)$(v2)) x4.test @ac_xfail_tests@
+
+TESTS = pass.test x1.test x2.test x3.test x4.test x5.test x6.test
+END
+
+cat > pass.test <<'END'
+#!/bin/sh
+exit 0
+END
+
+cat > xfail.test <<'END'
+#!/bin/sh
+exit 1
+END
+
+chmod a+x pass.test xfail.test
+
+cp pass.test sub1/pass.test
+cp pass.test sub2/pass.test
+
+for i in 1 2 3 4 5 6; do
+ cp xfail.test sub1/x$i.test
+ cp xfail.test sub2/x$i.test
+done
+
+cat > td <<'END'
+#! /bin/sh
+set -e
+test_name=INVALID
+log_file=/dev/null
+expect_failure=no
+while test $# -gt 0; do
+ case $1 in
+ --test-name) test_name=$2; shift;;
+ --expect-failure) expect_failure=$2; shift;;
+ --log-file) log_file=$2; shift;;
+ # Ignored.
+ --color-tests) shift;;
+ --enable-hard-errors) shift;;
+ # Explicitly terminate option list.
+ --) shift; break;;
+ # Shouldn't happen
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+st=0
+"$@" || st=$?
+rm -f "$log_file"
+case $st,$expect_failure in
+ 0,no) echo "PASS: $test_name"; exit 0;;
+ 1,no) echo "FAIL: $test_name"; exit 1;;
+ 0,yes) echo "XPASS: $test_name"; exit 1;;
+ 1,yes) echo "XFAIL: $test_name"; exit 0;;
+ *) echo "UNEXPECTED OUTCOME: $test_name"; exit 99;;
+esac | tee "$log_file"
+END
+chmod a+x td
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+VERBOSE=yes $MAKE check
+test -f test-suite.log
+test -f sub1/test-suite.log
+test -f sub2/test-suite.log
+
+for dir in sub1 sub2; do
+ cd $dir
+ cp pass.test x1.test
+ cp x2.test pass.test
+ $MAKE check && { cat test-suite.log; Exit 1; }
+ cat test-suite.log
+ grep '^FAIL: pass\.test$' test-suite.log
+ grep '^XPASS: x1\.test$' test-suite.log
+ test `grep -c '^FAIL' test-suite.log` -eq 1
+ test `grep -c '^XPASS' test-suite.log` -eq 1
+ cd ..
+done
+
+:
diff --git a/tests/test-driver-custom.test b/tests/test-driver-custom.test
new file mode 100755
index 0000000..b6a33d2
--- /dev/null
+++ b/tests/test-driver-custom.test
@@ -0,0 +1,140 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: per-extension test drivers.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+# Automake shouldn't match the '/test' in 'sub/test' as '.test' suffix.
+TESTS = 1.chk 2.test 3 4.c.chk 5.suf sub/test
+
+TEST_EXTENSIONS = .chk .test
+
+CHK_LOG_DRIVER = ./chk-wrapper
+TEST_LOG_DRIVER = $(SHELL) $(srcdir)/test-wrapper
+LOG_DRIVER = noext-wrapper
+
+AM_CHK_LOG_DRIVER_FLAGS = --am-chk
+CHK_LOG_DRIVER_FLAGS = --chk
+AM_TEST_LOG_DRIVER_FLAGS = -am-test
+TEST_LOG_DRIVER_FLAGS = -test
+AM_LOG_DRIVER_FLAGS = am
+LOG_DRIVER_FLAGS = _
+END
+
+mkdir sub bin
+PATH=`pwd`/bin$PATH_SEPARATOR$PATH; export PATH
+
+cat > wrapper.skel <<'END'
+#! /bin/sh
+set -e
+
+me=`echo "$0" | sed 's,^.*/,,'`
+if test -z "$me"; then
+ echo "$0: cannot determine program name" >&2
+ exit 99
+fi
+
+am_log_wflags='@am_log_wflags@'
+log_wflags='@log_wflags@'
+
+test_name=INVALID
+log_file=BAD.log
+extra_opts=
+while test $# -gt 0; do
+ case $1 in
+ --test-name) test_name=$2; shift;;
+ --log-file) log_file=$2; shift;;
+ # Ignored.
+ --expect-failure) shift;;
+ --color-tests) shift;;
+ --enable-hard-errors) shift;;
+ # Remembered in the same order they're passed in.
+ $am_log_wflags|$log_wflags) extra_opts="$extra_opts $1";;
+ # Explicitly terminate option list.
+ --) shift; break;;
+ # Shouldn't happen
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+
+echo "$me" "$test_name" $extra_opts > "$log_file"
+
+exec "$@"
+exit 127
+END
+
+sed -e 's|@am_log_wflags@|--am-chk|' \
+ -e 's|@log_wflags@|--chk|' \
+ < wrapper.skel > chk-wrapper
+
+sed -e 's|@am_log_wflags@|-am-test|' \
+ -e 's|@log_wflags@|-test|' \
+ < wrapper.skel > test-wrapper
+
+sed -e 's|@am_log_wflags@|am|' \
+ -e 's|@log_wflags@|_|' \
+ < wrapper.skel > bin/noext-wrapper
+
+# `test-wrapper' is deliberately not made executable.
+chmod a+x chk-wrapper bin/noext-wrapper
+
+# Not needed anymore.
+rm -f wrapper.skel
+
+cat > 1.chk << 'END'
+#! /bin/sh
+exit 0
+END
+chmod a+x 1.chk
+cp 1.chk 2.test
+cp 1.chk 3
+cp 1.chk 4.c.chk
+cp 1.chk 5.suf
+cp 1.chk sub/test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+$MAKE
+VERBOSE=yes $MAKE check
+ls -l . sub
+
+test ! -r BAD.log
+
+echo 'chk-wrapper 1.chk --am-chk --chk' > 1.exp
+echo 'test-wrapper 2.test -am-test -test' > 2.exp
+echo 'noext-wrapper 3 am _' > 3.exp
+echo 'chk-wrapper 4.c.chk --am-chk --chk' > 4.c.exp
+echo 'noext-wrapper 5.suf am _' > 5.suf.exp
+echo 'noext-wrapper sub/test am _' > sub/test.exp
+
+st=0
+for x in 1 2 3 4.c 5.suf sub/test; do
+ cat $x.log
+ diff $x.exp $x.log || st=1
+done
+
+Exit $st
diff --git a/tests/test-driver-fail.test b/tests/test-driver-fail.test
new file mode 100755
index 0000000..e1d2f9c
--- /dev/null
+++ b/tests/test-driver-fail.test
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: what happens when a test driver fails? Well,
+# "make check" should at least fail too, and the test-suite.log
+# shouldn't be created. Unfortunately, we cannot truly control also
+# the (non-)creation of individual test logs, since those are expected
+# to be created by the drivers themselves, and an ill-behaved driver
+# (like our dummy one in this test) might leave around a test log even
+# in case of internal failures.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TEST_LOG_DRIVER = ./oops
+TESTS = foo.test
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+exit 0
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# The testsuite driver does not exist.
+$MAKE check && Exit 1
+test ! -f test-suite.log
+
+# The testsuite driver exists and create the test log files, but fails.
+
+cat > oops <<'END'
+#!/bin/sh
+: > foo.log
+echo 'Oops, I fail!' >&2
+exit 1
+END
+chmod a+x oops
+
+$MAKE check && Exit 1
+test ! -f test-suite.log
+
+:
--
1.7.2.3
