* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'. Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99. When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* NEWS: Update.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise. Also, make stricter, and also test
from VPATH.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: Likewise.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 31 ++++++
NEWS | 4 +
doc/automake.texi | 8 +-
lib/Automake/tests/Makefile.in | 4 +-
lib/am/check.am | 8 +-
lib/test-driver | 5 +-
tests/Makefile.am | 2 +
tests/Makefile.in | 6 +-
tests/color.test | 101 +++++++++++++++-----
tests/color2.test | 108 ++++++++++++++++------
tests/parallel-tests-console-output.test | 104 +++++++++++++++++++++
tests/parallel-tests-exit-statuses.test | 108 ++++++++++++++++++++++
tests/parallel-tests-harderror.test | 6 +-
tests/parallel-tests-no-spurious-summary.test | 2 +
tests/parallel-tests.test | 9 +-
tests/test-driver-custom-multitest-recheck.test | 27 +++---
tests/test-driver-custom-multitest-recheck2.test | 56 ++++++++---
tests/test-driver-custom-multitest.test | 33 ++++---
tests/test-driver-custom-no-html.test | 2 +-
tests/test-driver-end-test-results.test | 1 +
tests/test-driver-global-log.test | 4 +-
tests/test-driver-recheck.test | 2 +-
tests/trivial-test-driver | 11 +-
23 files changed, 524 insertions(+), 118 deletions(-)
create mode 100755 tests/parallel-tests-console-output.test
create mode 100755 tests/parallel-tests-exit-statuses.test
diff --git a/ChangeLog b/ChangeLog
index f7da174..6a062e3 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,36 @@
2011-07-07 Stefano Lattarini <stefano.lattar...@gmail.com>
+ parallel-tests: new recognized test result 'ERROR'
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
+ `ERROR'. Use it when encountering unreadable test logs (previously
+ a simple `FAIL' was used in this situations).
+ * lib/test-driver: Set the global test result to `ERROR' when the
+ test exit status is 99. When doing colorized output, color `ERROR'
+ results in magenta.
+ * doc/automake.texi (Log files generation and test results
+ recording): Update by listing `ERROR' too among the list of valid
+ `:test-results:' arguments.
+ * NEWS: Update.
+ * tests/trivial-test-driver: Update.
+ * tests/parallel-tests.test: Likewise.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-no-html.test: Likewise.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/color.test: Likewise. Also, make stricter, and also test
+ from VPATH.
+ * tests/color2.test: Likewise, and improve syncing with color.test.
+ * tests/parallel-tests-exit-statuses.test: New test.
+ * tests/parallel-tests-console-output.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-07 Stefano Lattarini <stefano.lattar...@gmail.com>
+
parallel-tests: make parsing of test results safer
The new code for parsing the testsuite-generated `.log' files,
as introduced in commit `v1.11-872-gc96b881', considers each
diff --git a/NEWS b/NEWS
index 37af53a..56250ab 100644
--- a/NEWS
+++ b/NEWS
@@ -13,6 +13,10 @@ New in 1.11a:
* Changes to Automake-generated testsuite harnesses:
+ - The parallel-tests harness has a new test result 'ERROR', that can be
+ used to signal e.g., unexpected or internal errors, or failures to set
+ up test case scenarios.
+
- The default testsuite driver offered by the 'parallel-tests' option is
now implemented (partly at least) with the help of automake-provided
auxiliary scripts (e.g., `test-driver'), instead of relying entirely
diff --git a/doc/automake.texi b/doc/automake.texi
index 1e251db..16804cd 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9248,10 +9248,10 @@ leading whitespace will not be ignored.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
-@code{SKIP}, @code{FAIL} and @code{XPASS}. These results, when declared
-with @code{:test-result:}, can be optionally followed by text holding
-the name and/or a brief description of the corresponding test; the
-@option{parallel-tests} harness will ignore such extra text when
+@code{SKIP}, @code{FAIL}, @code{XPASS} and @code{ERROR}. These results,
+when declared with @code{:test-result:}, can be optionally followed by
+text holding the name and/or a brief description of the corresponding
+test; the @option{parallel-tests} harness will ignore such extra text when
generating @file{test-suite.log} and preparing the testsuite summary.
Also, @code{:test-result:} can be used with a special ``pseudo-result''
@code{END}, that will instruct the testsuite harness to stop scanning
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 5d3c39f..82ef9e1 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -348,7 +348,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
results2=''; \
exec 5<&0; \
for f in $$list2; do \
@@ -369,6 +369,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
skip=`echo "$$results" | grep -c '^SKIP'`; \
xfail=`echo "$$results" | grep -c '^XFAIL'`; \
xpass=`echo "$$results" | grep -c '^XPASS'`; \
+ error=`echo "$$results" | grep -c '^ERROR'`; \
+ fail=`expr $$fail + $$error`; \
failures=`expr $$fail + $$xpass`; \
all=`expr $$all - $$skip`; \
if test "$$all" -eq 1; then tests=test; All=; \
diff --git a/lib/am/check.am b/lib/am/check.am
index f2528e0..3cedd46 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -66,7 +66,7 @@ include inst-vars.am
## appended.
##
## In addition to the magic "exit 77 means SKIP" feature (which was
-## imported from automake), there is a magic "exit 99 means FAIL" feature
+## imported from automake), there is a magic "exit 99 means ERROR" feature
## which is useful if you need to issue a hard error no matter whether the
## test is XFAIL or not. You can disable this feature by setting the
## variable DISABLE_HARD_ERRORS to a nonempty value.
@@ -153,7 +153,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
## Readable test logs.
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
## Each unreadable test log counts as a failed test.
- results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
## Now we're going to extract the outcome of all the testcases from the
## test logs.
results2=''; \
@@ -187,6 +187,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
skip=`echo "$$results" | grep -c '^SKIP'`; \
xfail=`echo "$$results" | grep -c '^XFAIL'`; \
xpass=`echo "$$results" | grep -c '^XPASS'`; \
+ error=`echo "$$results" | grep -c '^ERROR'`; \
+## FIXME: for the moment, we count errors as failures, otherwise the code
+## that displays the testsuite summary will become too complicated.
+ fail=`expr $$fail + $$error`; \
failures=`expr $$fail + $$xpass`; \
all=`expr $$all - $$skip`; \
if test "$$all" -eq 1; then tests=test; All=; \
diff --git a/lib/test-driver b/lib/test-driver
index 145bbea..aa39c5b 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -85,9 +85,10 @@ if test $color_tests = yes; then
grn='[0;32m' # Green.
lgn='[1;32m' # Light green.
blu='[1;34m' # Blue.
+ mgn='[0;35m' # Magenta.
std='[m' # No color.
else
- red= grn= lgn= blu= std=
+ red= grn= lgn= blu= mgn= std=
fi
tmpfile=$logfile-t
@@ -109,7 +110,7 @@ case $estatus:$expect_failure in
0:yes) col=$red; res=XPASS;;
0:*) col=$grn; res=PASS ;;
77:*) col=$blu; res=SKIP ;;
- 99:*) col=$red; res=FAIL ;;
+ 99:*) col=$mgn; res=ERROR;;
*:yes) col=$lgn; res=XFAIL;;
*:*) col=$red; res=FAIL ;;
esac
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 9c0605d..64299b8 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -731,6 +731,8 @@ parallel-tests-log-override-recheck.test \
parallel-tests-empty-testlogs.test \
parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
+parallel-tests-exit-statuses.test \
+parallel-tests-console-output.test \
test-driver-end-test-results.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index dc03179..5ef82da 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -991,6 +991,8 @@ parallel-tests-log-override-recheck.test \
parallel-tests-empty-testlogs.test \
parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
+parallel-tests-exit-statuses.test \
+parallel-tests-console-output.test \
test-driver-end-test-results.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
@@ -1400,7 +1402,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
results2=''; \
exec 5<&0; \
for f in $$list2; do \
@@ -1421,6 +1423,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
skip=`echo "$$results" | grep -c '^SKIP'`; \
xfail=`echo "$$results" | grep -c '^XFAIL'`; \
xpass=`echo "$$results" | grep -c '^XPASS'`; \
+ error=`echo "$$results" | grep -c '^ERROR'`; \
+ fail=`expr $$fail + $$error`; \
failures=`expr $$fail + $$xpass`; \
all=`expr $$all - $$skip`; \
if test "$$all" -eq 1; then tests=test; All=; \
diff --git a/tests/color.test b/tests/color.test
index ecd4f94..2524324 100755
--- a/tests/color.test
+++ b/tests/color.test
@@ -22,11 +22,14 @@
TERM=ansi
export TERM
-red='[0;31m'
-grn='[0;32m'
-lgn='[1;32m'
-blu='[1;34m'
-std='[m'
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
# Check that grep can parse nonprinting characters.
# BSD 'grep' works from a pipe, but not a seekable file.
@@ -43,7 +46,7 @@ END
cat >Makefile.am <<'END'
AUTOMAKE_OPTIONS = color-tests
TESTS = $(check_SCRIPTS)
-check_SCRIPTS = pass fail skip xpass xfail
+check_SCRIPTS = pass fail skip xpass xfail error
XFAIL_TESTS = xpass xfail
END
@@ -62,39 +65,89 @@ cat >skip <<END
exit 77
END
+cat >error <<END
+#! /bin/sh
+exit 99
+END
+
cp fail xfail
cp pass xpass
-chmod +x pass fail skip xpass xfail
+chmod +x pass fail skip xpass xfail error
$ACLOCAL
$AUTOCONF
$AUTOMAKE --add-missing
-./configure
-
test_color ()
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn"
- cat stdout | grep ": fail" | $FGREP "$red"
- cat stdout | grep ": xfail" | $FGREP "$lgn"
- cat stdout | grep ": xpass" | $FGREP "$red"
- cat stdout | grep ": skip" | $FGREP "$blu"
+ cat stdout | grep "^${grn}PASS${std}: .*pass"
+ cat stdout | grep "^${red}FAIL${std}: .*fail"
+ cat stdout | grep "^${blu}SKIP${std}: .*skip"
+ cat stdout | grep "^${lgn}XFAIL${std}: .*xfail"
+ cat stdout | grep "^${red}XPASS${std}: .*xpass"
+ # The old serial testsuite driver doesn't distinguish between failures
+ # and hard errors.
+ if test x"$parallel_tests" = x"yes"; then
+ cat stdout | grep "^${mgn}ERROR${std}: .*error"
+ else
+ cat stdout | grep "^${red}FAIL${std}: .*error"
+ fi
+ :
}
test_no_color ()
{
- # Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn" && Exit 1
- cat stdout | grep ": fail" | $FGREP "$red" && Exit 1
- cat stdout | grep ": xfail" | $FGREP "$lgn" && Exit 1
- cat stdout | grep ": xpass" | $FGREP "$red" && Exit 1
- cat stdout | grep ": skip" | $FGREP "$blu" && Exit 1
- :
+ # With make implementations that, like Solaris make, in case of errors
+ # print the whole failing recipe on standard output, we should content
+ # ourselves with a laxer check, to avoid false positives.
+ # Keep this in sync with lib/am/check.am:$(am__color_tests).
+ if $FGREP '= Xalways || test -t 1 ' stdout; then
+ # Extra verbose make, resort to laxer checks.
+ (
+ set +e # In case some grepped regex below isn't matched.
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ # To check that the testsuite summary is not unduly colorized.
+ cat stdout | grep '===='
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
+ ) | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ else
+ cat stdout | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ fi
}
-AM_COLOR_TESTS=always $MAKE -e check >stdout && { cat stdout; Exit 1; }
-cat stdout
-test_color
+for vpath in false :; do
+
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+
+ $srcdir/configure
+
+ AM_COLOR_TESTS=always $MAKE -e check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ test_color
+
+ $MAKE -e check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ test_no_color
+
+ $MAKE distclean
+ cd $srcdir
+
+done
:
diff --git a/tests/color2.test b/tests/color2.test
index 4f8862e..758ccd4 100755
--- a/tests/color2.test
+++ b/tests/color2.test
@@ -22,11 +22,14 @@
TERM=ansi
export TERM
-red='[0;31m'
-grn='[0;32m'
-lgn='[1;32m'
-blu='[1;34m'
-std='[m'
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
# Check that grep can parse nonprinting characters.
# BSD 'grep' works from a pipe, but not a seekable file.
@@ -56,7 +59,7 @@ END
cat >Makefile.am <<'END'
AUTOMAKE_OPTIONS = color-tests
TESTS = $(check_SCRIPTS)
-check_SCRIPTS = pass fail skip xpass xfail
+check_SCRIPTS = pass fail skip xpass xfail error
XFAIL_TESTS = xpass xfail
END
@@ -75,34 +78,64 @@ cat >skip <<END
exit 77
END
+cat >error <<END
+#! /bin/sh
+exit 99
+END
+
cp fail xfail
cp pass xpass
-chmod +x pass fail skip xpass xfail
+chmod +x pass fail skip xpass xfail error
$ACLOCAL
-$AUTOMAKE -a
$AUTOCONF
-./configure
+$AUTOMAKE --add-missing
test_color ()
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn"
- cat stdout | grep ": fail" | $FGREP "$red"
- cat stdout | grep ": xfail" | $FGREP "$lgn"
- cat stdout | grep ": xpass" | $FGREP "$red"
- cat stdout | grep ": skip" | $FGREP "$blu"
+ cat stdout | grep "^${grn}PASS${std}: .*pass"
+ cat stdout | grep "^${red}FAIL${std}: .*fail"
+ cat stdout | grep "^${blu}SKIP${std}: .*skip"
+ cat stdout | grep "^${lgn}XFAIL${std}: .*xfail"
+ cat stdout | grep "^${red}XPASS${std}: .*xpass"
+ # The old serial testsuite driver doesn't distinguish between failures
+ # and hard errors.
+ if test x"$parallel_tests" = x"yes"; then
+ cat stdout | grep "^${mgn}ERROR${std}: .*error"
+ else
+ cat stdout | grep "^${red}FAIL${std}: .*error"
+ fi
+ :
}
test_no_color ()
{
- # Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn" && Exit 1
- cat stdout | grep ": fail" | $FGREP "$red" && Exit 1
- cat stdout | grep ": xfail" | $FGREP "$lgn" && Exit 1
- cat stdout | grep ": xpass" | $FGREP "$red" && Exit 1
- cat stdout | grep ": skip" | $FGREP "$blu" && Exit 1
- :
+ # With make implementations that, like Solaris make, in case of errors
+ # print the whole failing recipe on standard output, we should content
+ # ourselves with a laxer check, to avoid false positives.
+ # Keep this in sync with lib/am/check.am:$(am__color_tests).
+ if $FGREP '= Xalways || test -t 1 ' stdout; then
+ # Extra verbose make, resort to laxer checks.
+ (
+ set +e # In case some grepped regex below isn't matched.
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ # To check that the testsuite summary is not unduly colorized.
+ cat stdout | grep '===='
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
+ ) | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ else
+ cat stdout | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ fi
}
cat >expect-make <<'END'
@@ -111,14 +144,31 @@ eval spawn $env(MAKE) -e check
expect eof
END
-MAKE=$MAKE expect -f expect-make >stdout \
- || { cat stdout; Exit 1; }
-cat stdout
-test_color
+for vpath in false :; do
+
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+
+ $srcdir/configure
+
+ MAKE=$MAKE expect -f $srcdir/expect-make >stdout \
+ || { cat stdout; Exit 1; }
+ cat stdout
+ test_color
+
+ AM_COLOR_TESTS=no MAKE=$MAKE expect -f $srcdir/expect-make >stdout \
+ || { cat stdout; Exit 1; }
+ cat stdout
+ test_no_color
+
+ $MAKE distclean
+ cd $srcdir
-AM_COLOR_TESTS=no MAKE=$MAKE expect -f expect-make >stdout \
- || { cat stdout; Exit 1; }
-cat stdout
-test_no_color
+done
:
diff --git a/tests/parallel-tests-console-output.test
b/tests/parallel-tests-console-output.test
new file mode 100755
index 0000000..8122bfd
--- /dev/null
+++ b/tests/parallel-tests-console-output.test
@@ -0,0 +1,104 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# parallel-tests: some checks on console output about testsuite
+# progress.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+XFAIL_TESTS = sub/xpass.test xfail.test error.test
+TESTS = $(XFAIL_TESTS) fail.test pass.test a/b/skip.test sub/error2.test
+pass.log: fail.log
+error.log: pass.log
+sub/xpass.log: error.log
+sub/error2.log: xfail.log
+a/b/skip.log: sub/error2.log
+END
+
+cat > exp <<'END'
+FAIL: fail.test
+PASS: pass.test
+ERROR: error.test
+XPASS: sub/xpass.test
+XFAIL: xfail.test
+ERROR: sub/error2.test
+SKIP: a/b/skip.test
+END
+
+mkdir sub a a/b
+
+cat > pass.test << 'END'
+#!/bin/sh
+exit 0
+END
+cp pass.test sub/xpass.test
+
+cat > fail.test << 'END'
+#!/bin/sh
+exit 1
+END
+
+cat > xfail.test << 'END'
+#!/bin/sh
+# The sleep should ensure expected execution order of tests
+# even when make is run in parallel mode.
+# FIXME: quotes below required by maintainer-check.
+sleep '10'
+exit 1
+END
+
+cat > error.test << 'END'
+#!/bin/sh
+exit 99
+END
+cp error.test sub/error2.test
+
+cat > a/b/skip.test << 'END'
+#!/bin/sh
+exit 77
+END
+
+chmod a+x pass.test fail.test xfail.test sub/xpass.test \
+ a/b/skip.test error.test sub/error2.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+for vpath in : false; do
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+ $srcdir/configure
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+ cat got
+ diff $srcdir/exp got
+ cd $srcdir
+done
+
+:
diff --git a/tests/parallel-tests-exit-statuses.test
b/tests/parallel-tests-exit-statuses.test
new file mode 100755
index 0000000..531526a
--- /dev/null
+++ b/tests/parallel-tests-exit-statuses.test
@@ -0,0 +1,108 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: normal and special exit statuses
+# in the test scripts.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+# $failure_statuses should be defined to the list of all integers between
+# 1 and 255 (inclusive), excluded 77 and 99.
+# Let's use `seq' if available, it's faster than the loop.
+failure_statuses=`seq 1 255 \
+ || { i=1; while test $i -le 255; do echo $i; i=\`expr $i + 1\`; done; }`
+failure_statuses=`
+ for i in $failure_statuses; do
+ test $i -eq 77 || test $i -eq 99 || echo $i
+ done | tr "$nl" ' '`
+# For debugging.
+echo "failure_statuses: $failure_statuses"
+# Sanity check.
+test `for st in $failure_statuses; do echo $st; done | wc -l` -eq 253 \
+ || fatal_ "initializing list of exit statuses for simple failures"
+
+cat > Makefile.am <<END
+LOG_COMPILER = ./do-exit
+fail_tests = $failure_statuses
+TESTS = 0 77 99 $failure_statuses
+\$(TESTS):
+END
+
+cat > do-exit <<'END'
+#!/bin/sh
+echo "$0: $1"
+case $1 in
+ [0-9]|[0-9][0-9]|[0-9][0-9][0-9]) st=$1;;
+ */[0-9]|*/[0-9][0-9]|*/[0-9][0-9][0-9]) st=`echo x"$1" | sed 's|.*/||'`;;
+ *) st=99;;
+esac
+exit $st
+END
+chmod a+x do-exit
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+{
+ echo PASS: 0
+ echo SKIP: 77
+ echo ERROR: 99
+ for st in $failure_statuses; do
+ echo "FAIL: $st"
+ done
+} | LC_ALL=C sort > exp-fail
+
+sed 's/^FAIL:/XFAIL:/' exp-fail | LC_ALL=C sort > exp-xfail-1
+sed '/^ERROR:/d' exp-xfail-1 > exp-xfail-2
+
+sort exp-fail
+sort exp-xfail-1
+sort exp-xfail-2
+
+./configure
+
+st=1
+$MAKE check >stdout && st=0
+cat stdout
+cat test-suite.log
+test $st -gt 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-fail
+diff exp-fail got-fail
+
+st=1
+XFAIL_TESTS="$failure_statuses 99" $MAKE -e check >stdout && st=0
+cat stdout
+cat test-suite.log
+test $st -gt 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-1
+diff exp-xfail-1 got-xfail-1
+
+st=0
+XFAIL_TESTS="$failure_statuses" TESTS="0 77 $failure_statuses" \
+ $MAKE -e check >stdout || st=$?
+cat stdout
+cat test-suite.log
+test $st -eq 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-2
+diff exp-xfail-2 got-xfail-2
+
+:
diff --git a/tests/parallel-tests-harderror.test
b/tests/parallel-tests-harderror.test
index 8680e29..f30fd52 100755
--- a/tests/parallel-tests-harderror.test
+++ b/tests/parallel-tests-harderror.test
@@ -69,13 +69,13 @@ DISABLE_HARD_ERRORS=x $MAKE check
$MAKE check DISABLE_HARD_ERRORS='' && Exit 1
cat test-suite.log
-grep '^FAIL: foo\.test .*exit.*99' test-suite.log
+grep '^ERROR: foo\.test .*exit.*99' test-suite.log
cd sub
# The `-e' is wanted here.
DISABLE_HARD_ERRORS='' $MAKE -e check && Exit 1
cat test-suite.log
-grep '^FAIL: bar\.test .*exit.*99' test-suite.log
+grep '^ERROR: bar\.test .*exit.*99' test-suite.log
cd ..
# Check the distributions.
@@ -90,7 +90,7 @@ $AUTOMAKE Makefile
./config.status Makefile
VERBOSE=yes $MAKE check && Exit 1
grep '^FAIL' test-suite.log && Exit 1
-grep '^FAIL: bar\.test .*exit.*99' sub/test-suite.log
+grep '^ERROR: bar\.test .*exit.*99' sub/test-suite.log
echo 'DISABLE_HARD_ERRORS = zardoz' >> sub/Makefile
VERBOSE=yes $MAKE check
diff --git a/tests/parallel-tests-no-spurious-summary.test
b/tests/parallel-tests-no-spurious-summary.test
index 43c54a5..8b9547a 100755
--- a/tests/parallel-tests-no-spurious-summary.test
+++ b/tests/parallel-tests-no-spurious-summary.test
@@ -32,10 +32,12 @@ cat > foo.test <<'END'
#! /bin/sh
echo :test-result:XFAIL
echo :test-result: SKIP
+echo :test-result:ERROR
exit 0
END
cat > bar.test <<'END'
#! /bin/sh
+echo :test-result: ERROR
echo :test-result:FAIL
echo :test-result: XPASS
exit 0
diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
index f046c3f..95f78e7 100755
--- a/tests/parallel-tests.test
+++ b/tests/parallel-tests.test
@@ -68,10 +68,13 @@ $AUTOMAKE -a
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-# There should be two errors: bar.test is a hard error.
-test `grep -c '^FAIL' stdout` -eq 2
+# There should be one failure and one hard error.
+test `grep -c '^FAIL:' stdout` -eq 1
+test `grep -c '^ERROR:' stdout` -eq 1
test -f mylog.log
-test `grep -c '^FAIL' mylog.log` -eq 2
+cat mylog.log
+test `grep -c '^FAIL:' mylog.log` -eq 1
+test `grep -c '^ERROR:' mylog.log` -eq 1
test -f baz.log
test -f bar.log
test -f foo.log
diff --git a/tests/test-driver-custom-multitest-recheck.test
b/tests/test-driver-custom-multitest-recheck.test
index e0f010f..c178fa0 100755
--- a/tests/test-driver-custom-multitest-recheck.test
+++ b/tests/test-driver-custom-multitest-recheck.test
@@ -49,7 +49,7 @@ echo PASS:
if test -f b.ok; then
echo PASS:
else
- echo FAIL:
+ echo ERROR:
fi
: > b.run
END
@@ -97,14 +97,15 @@ do_recheck ()
do_count ()
{
- pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
+ pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
eval "$@"
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
test `grep -c '^PASS:' stdout` -eq $pass
test `grep -c '^FAIL:' stdout` -eq $fail
test `grep -c '^XPASS:' stdout` -eq $xpass
test `grep -c '^XFAIL:' stdout` -eq $xfail
test `grep -c '^SKIP:' stdout` -eq $skip
+ test `grep -c '^ERROR:' stdout` -eq $error
}
for vpath in : false; do
@@ -129,7 +130,7 @@ for vpath in : false; do
test ! -r c.log
test ! -r d.run
test ! -r d.log
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
: Run the tests for the first time.
$MAKE check >stdout && { cat stdout; Exit 1; }
@@ -140,7 +141,7 @@ for vpath in : false; do
test -f b.run
test -f c.run
test -f d.run
- do_count pass=3 fail=3 xpass=1 xfail=1 skip=1
+ do_count pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
: Let us make b.test pass.
echo OK > b.ok
@@ -151,7 +152,7 @@ for vpath in : false; do
test -f b.run
test -f c.run
test -f d.run
- do_count pass=2 fail=2 xpass=1 xfail=1 skip=1
+ do_count pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
: Let us make the first part of c.test pass.
echo OK > c.pass
@@ -160,7 +161,7 @@ for vpath in : false; do
test ! -r b.run
test -f c.run
test -f d.run
- do_count pass=1 fail=1 xpass=1 xfail=1 skip=1
+ do_count pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
: Let us make also the second part of c.test pass.
echo KO > c.xfail
@@ -169,7 +170,7 @@ for vpath in : false; do
test ! -r b.run
test -f c.run
test -f d.run
- do_count pass=1 fail=1 xpass=0 xfail=2 skip=1
+ do_count pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
: Nothing changed, so only d.test should be run.
for i in 1 2; do
@@ -178,7 +179,7 @@ for vpath in : false; do
test ! -r b.run
test ! -r c.run
test -f d.run
- do_count pass=0 fail=1 xpass=0 xfail=0 skip=1
+ do_count pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
done
: Let us make d.test run more testcases, and experience _more_ failures.
@@ -190,14 +191,16 @@ for vpath in : false; do
echo XPASS: xp
echo FAIL: f 3
echo FAIL: f 4
+ echo ERROR: e 1
echo PASS: p 2
+ echo ERROR: e 2
END
do_recheck --fail
test ! -r a.run
test ! -r b.run
test ! -r c.run
test -f d.run
- do_count pass=2 fail=4 xpass=1 xfail=0 skip=2
+ do_count pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
: Let us finally make d.test pass.
echo : > d.extra
@@ -206,7 +209,7 @@ END
test ! -r b.run
test ! -r c.run
test -f d.run
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=1
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
: All tests have been successful or skipped, nothing should be re-run.
do_recheck --pass
@@ -214,7 +217,7 @@ END
test ! -r b.run
test ! -r c.run
test ! -r d.run
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
cd $srcdir
diff --git a/tests/test-driver-custom-multitest-recheck2.test
b/tests/test-driver-custom-multitest-recheck2.test
index a18bdd8..36cb302 100755
--- a/tests/test-driver-custom-multitest-recheck2.test
+++ b/tests/test-driver-custom-multitest-recheck2.test
@@ -56,7 +56,13 @@ END
cat > c.test << 'END'
#! /bin/sh
-echo XPASS: xp
+if test -f c.err; then
+ echo ERROR: xxx
+elif test -f c.ok; then
+ echo PASS: ok
+else
+ echo XPASS: xp
+fi
: > c.run
END
@@ -68,14 +74,15 @@ $AUTOMAKE
do_count ()
{
- pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
+ pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
eval "$@"
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
test `grep -c '^PASS:' stdout` -eq $pass
test `grep -c '^FAIL:' stdout` -eq $fail
test `grep -c '^XPASS:' stdout` -eq $xpass
test `grep -c '^XFAIL:' stdout` -eq $xfail
test `grep -c '^SKIP:' stdout` -eq $skip
+ test `grep -c '^ERROR:' stdout` -eq $error
}
for vpath in : false; do
@@ -96,7 +103,7 @@ for vpath in : false; do
test -f a.run
test -f b.run
test -f c.run
- do_count pass=2 fail=1 xpass=1 xfail=0 skip=1
+ do_count pass=2 fail=1 xpass=1 xfail=0 skip=1 error=0
rm -f *.run
@@ -104,7 +111,7 @@ for vpath in : false; do
for var in TESTS TEST_LOGS; do
env "$var=" $MAKE -e recheck >stdout || { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
@@ -115,7 +122,7 @@ for vpath in : false; do
env TESTS=a.test $MAKE -e recheck >stdout \
|| { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
@@ -128,7 +135,7 @@ for vpath in : false; do
test ! -r a.run
test -f b.run
test ! -r c.run
- do_count pass=0 fail=0 xpass=0 xfail=1 skip=1
+ do_count pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
rm -f *.run
@@ -136,35 +143,54 @@ for vpath in : false; do
TEST_LOGS=b.log $MAKE -e recheck >stdout \
|| { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
TESTS='a.test b.test' $MAKE -e recheck >stdout \
|| { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
- # An XPASS should count a failure.
+ : No need to re-run a.test anymore, but c.test should be rerun,
+ : as it contained an XPASS. And this time, make it fail with
+ : an hard error.
+ : > c.err
env TEST_LOGS='a.log c.log' $MAKE -e recheck >stdout \
&& { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
test ! -r a.run
test ! -r b.run
test -f c.run
- rm -f *.run
- env TESTS='c.test b.test' $MAKE -e recheck >stdout \
- && { cat stdout; Exit 1; }
+
+ rm -f *.run *.err
+
+ : c.test contained and hard error the last time, so it should be re-run.
+ : This time, make it pass
+ : > c.ok
+ env TESTS='c.test a.test' $MAKE -e recheck >stdout \
+ || { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
+ do_count pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test -f c.run
+ rm -f *.run *.err *.ok
+
+ : Nothing should be rerun anymore, as all tests have been eventually
+ : succesful.
+ $MAKE recheck >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+
cd $srcdir
done
diff --git a/tests/test-driver-custom-multitest.test
b/tests/test-driver-custom-multitest.test
index a1c1010..2366b5b 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -40,7 +40,7 @@ TESTS = \
pass-fail.t \
pass4-skip.t \
pass3-skip2-xfail.t \
- pass-xpass-fail-xfail-skip.t
+ pass-xpass-fail-xfail-skip-error.t
END
expected_pass=10
@@ -48,6 +48,7 @@ expected_fail=5
expected_skip=4
expected_xfail=2
expected_xpass=1
+expected_error=1
cat > pass.t << 'END'
echo %% pass %%
@@ -99,13 +100,14 @@ mv || echo "XFAIL: mv cannot read users' mind yet"
exit 127
END
-cat > pass-xpass-fail-xfail-skip.t << 'END'
+cat > pass-xpass-fail-xfail-skip-error.t << 'END'
echo PASS:
echo FAIL:
echo XFAIL:
echo XPASS:
echo SKIP:
-echo %% pass-xpass-fail-xfail-skip %%
+echo ERROR:
+echo %% pass-xpass-fail-xfail-skip-error %%
END
chmod a+x *.t
@@ -139,25 +141,28 @@ for vpath in : false; do
cat pass-fail.log
cat pass4-skip.log
cat pass3-skip2-xfail.log
- cat pass-xpass-fail-xfail-skip.log
+ cat pass-xpass-fail-xfail-skip-error.log
# For debugging.
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
test `grep -c '^PASS:' stdout` -eq $expected_pass
test `grep -c '^FAIL:' stdout` -eq $expected_fail
test `grep -c '^XPASS:' stdout` -eq $expected_xpass
test `grep -c '^XFAIL:' stdout` -eq $expected_xfail
test `grep -c '^SKIP:' stdout` -eq $expected_skip
+ test `grep -c '^ERROR:' stdout` -eq $expected_error
- grep '^PASS: pass-xpass-fail-xfail-skip.t\, testcase 1' stdout
- grep '^FAIL: pass-xpass-fail-xfail-skip\.t, testcase 2' stdout
- grep '^XFAIL: pass-xpass-fail-xfail-skip\.t, testcase 3' stdout
- grep '^XPASS: pass-xpass-fail-xfail-skip\.t, testcase 4' stdout
- grep '^SKIP: pass-xpass-fail-xfail-skip\.t, testcase 5' stdout
+ tst=pass-xpass-fail-xfail-skip-error
+ grep "^PASS: $tst\.t, testcase 1" stdout
+ grep "^FAIL: $tst\.t, testcase 2" stdout
+ grep "^XFAIL: $tst\.t, testcase 3" stdout
+ grep "^XPASS: $tst\.t, testcase 4" stdout
+ grep "^SKIP: $tst\.t, testcase 5" stdout
+ grep "^ERROR: $tst\.t, testcase 6" stdout
# Check testsuite summary printed on console.
sed -e 's/[()]/ /g' -e 's/^/ /' stdout > t
- grep ' 6 of 18 ' t
+ grep ' 7 of 19 ' t
grep ' 1 unexpected pass' t
grep ' 4 test.* not run' t
@@ -170,7 +175,7 @@ for vpath in : false; do
grep '%% fail %%' test-suite.log
grep '%% fail2 %%' test-suite.log
grep '%% pass-fail %%' test-suite.log
- grep '%% pass-xpass-fail-xfail-skip %%' test-suite.log
+ grep '%% pass-xpass-fail-xfail-skip-error %%' test-suite.log
test `grep -c '%% ' test-suite.log` -eq 4
TESTS='pass.t pass3-skip2-xfail.t' $MAKE -e check >stdout \
@@ -178,11 +183,11 @@ for vpath in : false; do
cat test-suite.log
cat stdout
# For debugging.
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
test `grep -c '^PASS:' stdout` -eq 4
test `grep -c '^SKIP:' stdout` -eq 2
test `grep -c '^XFAIL:' stdout` -eq 1
- $EGREP '^(FAIL|XPASS)' stdout && Exit 1
+ $EGREP '^(FAIL|XPASS|ERROR)' stdout && Exit 1
cd $srcdir
diff --git a/tests/test-driver-custom-no-html.test
b/tests/test-driver-custom-no-html.test
index 9aceae6..683659b 100755
--- a/tests/test-driver-custom-no-html.test
+++ b/tests/test-driver-custom-no-html.test
@@ -62,6 +62,6 @@ $FGREP 'dummy title' test-suite.log
# Sanity check: trying to produce HTML output should fail.
$MAKE check-html >output 2>&1 && { cat output; Exit 1; }
cat output
-$EGREP 'SEVERE|ERROR' output
+$FGREP SEVERE output
:
diff --git a/tests/test-driver-end-test-results.test
b/tests/test-driver-end-test-results.test
index 97c1f29..03af51b 100755
--- a/tests/test-driver-end-test-results.test
+++ b/tests/test-driver-end-test-results.test
@@ -57,6 +57,7 @@ PASS: b.test
:test-result:PASS
:test-result:END
:test-result:FAIL
+:test-result:ERROR
END
cat > c.test <<END
diff --git a/tests/test-driver-global-log.test
b/tests/test-driver-global-log.test
index cdaaff6..35b6fe1 100755
--- a/tests/test-driver-global-log.test
+++ b/tests/test-driver-global-log.test
@@ -44,12 +44,13 @@ END
echo SKIP: skip.test > skip.test
echo FAIL: fail.test > fail.test
echo XPASS: xpass.test > xpass.test
+echo ERROR: error.test > error.test
echo :test-result: PASS > fake-pass.test
echo "$tab $tab$tab" > empty.test
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = ./dummy-driver
-TESTS = pass.test skip.test fail.test xfail.test xpass.test \
+TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \
fake-pass.test empty.test
END
@@ -82,6 +83,7 @@ cat test-suite.log
grep '^SKIP: skip\.test$' test-suite.log
grep '^FAIL: fail.test$' test-suite.log
grep '^XPASS: xpass.test$' test-suite.log
+grep '^ERROR: error.test$' test-suite.log
grep '^:test-result: PASS$' test-suite.log
grep "^$tab $tab$tab$" test-suite.log
$EGREP 'not seen' test-suite.log && Exit 1
diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
index 2ae75b9..f9482b1 100755
--- a/tests/test-driver-recheck.test
+++ b/tests/test-driver-recheck.test
@@ -54,7 +54,7 @@ END
rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
done
-for R in FAIL XPASS UNKNOWN; do
+for R in FAIL XPASS ERROR UNKNOWN; do
echo $R: > $R-1
echo $R:foo > $R-2
echo $R: bar baz > $R-3
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index f06eb1c..113e158 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -20,9 +20,10 @@
# test results per test script.
#
# The exit status of the wrapped script is ignored. Lines in its stdout
-# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS' or `SKIP'
-# count as a test case result with the obviously-corresponding outcome.
-# Every other line is ignored for what concerns the testsuite outcome.
+# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS', `SKIP' or
+# `ERROR' count as a test case result with the obviously-corresponding
+# outcome. Every other line is ignored for what concerns the testsuite
+# outcome.
#
# This script is used at least by the `driver-custom-multitest*.test'
# tests.
@@ -61,10 +62,10 @@ tmp_res=$log_file-res.tmp
: > $tmp_res
while read line; do
case $line in
- PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*)
+ PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*|ERROR:*)
i=`expr $i + 1`
result=`LC_ALL=C expr "$line" : '\([A-Z]*\):.*'`
- case $result in FAIL|XPASS) st=1;; esac
+ case $result in FAIL|XPASS|ERROR) st=1;; esac
# Output testcase result to console.
echo "$result: $test_name, testcase $i"
# Register testcase outcome for the log file.
--
1.7.2.3
