On Tuesday 28 June 2011, Stefano Lattarini wrote:
> > Please go ahead after addressing my issues above. The questions I wrote
> > should still be discussed (and might need changing stuff again) but that
> > shouldn't hold up the patch.
> >
> OK, will push this evening or tomorrow.
>
I've pushed to 'test-protocols' now, and I've deleted the temporary branch
'GSoC/experimental/docs-custom-test-drivers'. Attached is the patch that
has been pushed, for reference.
Regards,
Stefano
From 1402741313cf6c2fe8611eb294f9a8111c994a32 Mon Sep 17 00:00:00 2001
Message-Id: <1402741313cf6c2fe8611eb294f9a8111c994a32.1309354565.git.stefano.lattar...@gmail.com>
From: Stefano Lattarini <stefano.lattar...@gmail.com>
Date: Wed, 22 Jun 2011 21:59:34 +0200
Subject: [PATCH] docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
use suggested here is not portable to 'parallel-tests'.
(Simple Tests using parallel-tests): Document new restrictions on
the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
(Custom Test Drivers): New section and node.
(Overview of Custom Test Drivers Support): New subsection.
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
* lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
moved into the manual.
(recheck, recheck-html): Minor adjustments to better conform to the
documentation (this should cause no semantic changes w.r.t. the
former behaviour); minor improvements and extensions to existing
comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 24 ++
doc/automake.texi | 378 +++++++++++++++++++++++++++++++--
lib/Automake/tests/Makefile.in | 2 +-
lib/am/check.am | 15 +-
tests/Makefile.am | 4 +
tests/Makefile.in | 6 +-
tests/test-driver-create-log-dir.test | 74 +++++++
tests/test-driver-global-log.test | 110 ++++++++++
tests/test-driver-recheck.test | 126 +++++++++++
tests/test-driver-strip-vpath.test | 96 +++++++++
10 files changed, 804 insertions(+), 31 deletions(-)
create mode 100755 tests/test-driver-create-log-dir.test
create mode 100755 tests/test-driver-global-log.test
create mode 100755 tests/test-driver-recheck.test
create mode 100755 tests/test-driver-strip-vpath.test
diff --git a/ChangeLog b/ChangeLog
index c99944a..6203f89 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,27 @@
+2011-06-29 Stefano Lattarini <stefano.lattar...@gmail.com>
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
2011-06-29 Stefano Lattarini <stefano.lattar...@gmail.com>
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
diff --git a/doc/automake.texi b/doc/automake.texi
index 481ada4..a26f2a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -319,9 +319,23 @@ Support for test suites
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
+* Custom Test Drivers:: Writing and using custom test drivers
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
+Custom Test Drivers
+
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
+
+API for Custom Test Drivers
+
+* Command-line arguments for test drivers::
+* Log files generation and test results recording::
+* Testsuite progress output::
+* HTML generation from testsuite logs::
+
Changing Automake's Behavior
* Options generalities:: Semantics of Automake option
@@ -8641,6 +8655,7 @@ are very similar.
@menu
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
+* Custom Test Drivers:: Writing and using custom test drivers
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@end menu
@@ -8670,7 +8685,9 @@ given test program exits with a status of 77, then its result is ignored
in the final count. This feature allows non-portable tests to be
ignored in environments where they don't make sense.
+@anchor{Simple tests and color-tests}
@vindex AM_COLOR_TESTS
+@cindex Colorized testsuite output
If the Automake option @code{color-tests} is used (@pxref{Options})
and standard output is connected to a capable terminal, then the test
results and the summary are colored appropriately. The user can disable
@@ -8699,10 +8716,12 @@ TESTS_ENVIRONMENT = $(PERL) -Mstrict -w
TESTS = foo.pl bar.pl baz.pl
@end example
-Note that the @option{parallel-tests} driver provides a more elegant
-way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
-variable for the user to override (@pxref{Simple Tests using
-parallel-tests}).
+It's important to note that this last use of @code{TESTS_ENVIRONMENT}
+is invalid when the @option{parallel-tests} option is active; however,
+the @option{parallel-tests} driver provides a more elegant way to
+achieve the same effect, with the further benefit of freeing the
+@code{TESTS_ENVIRONMENT} variable for the user
+(@pxref{Simple Tests using parallel-tests}).
@cindex Tests, expected failure
@@ -8747,7 +8766,9 @@ for exceptional failures. Similar to the simple test driver,
the @code{check_*} variables are honored, and the environment variable
@env{srcdir} is set during test execution. Also, @code{TESTS_ENVIRONMENT}
is still honored, but is complemented by a new developer-reserved variable
-@code{AM_TESTS_ENVIRONMENT} (described below).
+@code{AM_TESTS_ENVIRONMENT} (described below), and its allowed uses are
+somewhat restricted (other features of the @option{parallel-tests} driver
+compensate for this, though).
This test driver is still experimental and may undergo changes in order
to satisfy additional portability requirements.
@@ -8820,28 +8841,47 @@ and @samp{./wrapper-script -d baz} to produce @file{foo.log},
but should be reserved for the user.
@vindex AM_TESTS_ENVIRONMENT
-The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
-code and set environment variables for the tests' runs. The user can
-still employ the @code{TESTS_ENVIRONMENT} variable to override settings
-from @code{AM_TESTS_ENVIRONMENT}; for that to work portably, however,
-the contents of a non-empty @code{AM_TESTS_ENVIRONMENT} @emph{must} be
-terminated by a semicolon. Here is an example of a slightly elaborate
-definition:
+The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
+be used to run initialization code and set environment variables for the
+test scripts. The former variable is developer-reserved, and can be
+defined in the @file{Makefile.am}, while the latter is reserved for the
+user, which can employ it to extend or override the settings in the
+former; for this to work portably, however, the contents of a non-empty
+@code{AM_TESTS_ENVIRONMENT} @emph{must} be terminated by a semicolon.
@example
AM_TESTS_ENVIRONMENT = \
## Some environment initializations are kept in a separate shell file
-## `tests-env.sh', which can make it easier to also run tests from the
-## command line.
+## `tests-env.sh', which can make it easier to also run tests from
+## the command line.
. $(srcdir)/tests-env.sh; \
-## On Solaris, prefer more POSIX-compliant versions of the standard tools
-## by default.
+## On Solaris, prefer more POSIX-compliant versions of the standard
+## tools by default.
if test -d /usr/xpg4/bin; then \
PATH=/usr/xpg4/bin:$$PATH; export PATH; \
fi;
@c $$ restore font-lock
@end example
+It's important to note that, differently from what we've seen for the
+serial testsuite driver (@pxref{Simple Tests using parallel-tests}), the
+@code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables
+@emph{cannot} be use to define a custom test driver; the
+@code{LOG_COMPILER} and @code{LOG_FLAGS} (or their extension-specific
+counterparts) should be used instead:
+
+@example
+## This is WRONG!
+AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
+@end example
+
+@example
+## Do this instead.
+AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
+LOG_COMPILER = $(PERL)
+AM_LOG_FLAGS = -Mstrict -w
+@end example
+
@trindex mostlyclean
@trindex check-html
@vindex RST2HTML
@@ -8984,6 +9024,306 @@ semantics of FreeBSD and OpenBSD @command{make} conflict with this).
In case of doubt you may want to require to use GNU @command{make},
or work around the issue with inference rules to generate the tests.
+@node Custom Test Drivers
+@section Custom Test Drivers
+
+@menu
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
+@end menu
+
+@node Overview of Custom Test Drivers Support
+@subsection Overview of Custom Test Drivers Support
+
+Starting form Automake version 1.12, the @option{parallel-tests} harness
+allows the package authors to use third-party custom test drivers, in case
+the
+@c FIXME: this should become "default ones" once we have TAP and Subunit
+default one is inadequate for their purposes.
+@c FIXME: add this once we have TAP and Subunit
+@c "or do not support their testing protocol of choice."
+
+A custom test driver is expected to properly run the test programs
+passed to it, including the command-line arguments passed to it,
+to analyze their execution and outcome, to create the @file{.log}
+files associated to these test runs, and to display the test results
+on the console. It is responsibility of the author of the test driver
+to ensure that it implements all the above steps meaningfully and
+correctly; Automake isn't and can't be of any help here. On the other
+hand, the Automake-provided code for testsuite summary generation offers
+support for test drivers allowing several test results per test script,
+if they take care to register such results properly (@pxref{Log files
+generation and test results recording}).
+
+The exact details of how test scripts' results are to be determined and
+analyzed is left to the individual drivers. Some drivers might only
+consider the test script exit status (this is done for example by the
+default test driver used by the @option{parallel-tests} harness, described
+in the previous section). Other drivers might implement more complex and
+advanced test protocols, which might require them to parse and interpreter
+the output emitted by the test script they're running (examples of such
+protocols are TAP and SubUnit).
+
+It's very important to note that, even when using custom test drivers,
+most of the @option{parallel-tests} infrastructure described in the
+previous section remains in place; this includes:
+
+@itemize
+@item
+concurrency through the use of @command{make}'s option @option{-j};
+@item
+per-test @file{.log} files, and generation of a summary @file{.log}
+file from them;
+@item
+@code{recheck} target and lazy reruns of tests;
+@item
+inter-test dependencies;
+@item
+support for @code{check_*} variables (@code{check_PROGRAMS},
+@code{check_LIBRARIES}, ...);
+@item
+use of @code{VERBOSE} environment variable to get verbose output on
+testsuite failures;
+@item
+definition and honoring of @code{TESTS_ENVIRONMENT} and
+@code{AM_TESTS_ENVIRONMENT} variables;
+@item
+definition of generic and extension-specific @code{LOG_COMPILER} and
+@code{LOG_FLAGS} variables.
+@end itemize
+
+@noindent
+On the other hand, the exact semantics of how (and if)
+@option{color-tests}, @code{XFAIL_TESTS}, and hard errors are supported
+and handled is left to the individual test drivers.
+
+@c TODO: We should really add a working example in the doc/ directory,
+@c TODO: and reference if from here.
+
+@node Declaring Custom Test Drivers
+@subsection Declaring Custom Test Drivers
+
+@vindex _LOG_DRIVER
+@vindex _LOG_DRIVER_FLAGS
+@vindex LOG_DRIVER
+@vindex LOG_DRIVER_FLAGS
+@vindex @var{ext}_LOG_DRIVER
+@vindex @var{ext}_LOG_DRIVER_FLAGS
+@vindex AM_@var{ext}_LOG_DRIVER_FLAGS
+@vindex AM_LOG_DRIVER_FLAGS
+Custom testsuite drivers are declared by defining the make variables
+@code{LOG_DRIVER} or @code{@var{ext}_LOG_DRIVER} (where @var{ext} must
+be declared in @code{TEST_EXTENSIONS}). They must be defined to
+programs or scripts that will be used to drive the execution, logging,
+and outcome report of the tests with corresponding extensions (or of
+those with no registered extension in the case of @code{LOG_DRIVER}).
+Clearly, multiple distinct test drivers can be declared in the same
+@file{Makefile.am}. Note moreover that the @code{LOG_DRIVER} variables
+are @emph{not} a substitute for the @code{LOG_COMPILER} variables: the
+two sets of variables can, and often do, usefully and legitimately
+coexist.
+
+@c TODO: We should really be able to point to a clarifying example here!
+
+The developer-reserved variable @code{AM_LOG_DRIVER_FLAGS} and the
+user-reserved variable @code{LOG_DRIVER_FLAGS} can be used to define
+flags that will be passed to each invocation of @code{LOG_DRIVER}
+(with the user-defined flags obviously taking precedence over the
+developer-reserved ones). Similarly, for each extension @var{ext}
+declared in @code{TEST_EXTENSIONS}, flags listed in
+@code{AM_@var{ext}_LOG_DRIVER_FLAGS} and
+@code{@var{ext}_LOG_DRIVER_FLAGS} will be passed to
+invocations of @code{@var{ext}_LOG_DRIVER}.
+
+@node API for Custom Test Drivers
+@subsection API for Custom Test Drivers
+
+Note that @emph{the APIs described here are still somewhat experimental},
+and might undergo changes and tightenings in the future, to accommodate
+for new features or to satisfy additional portability requirements.
+
+The main characteristic of these APIs is that they are designed to share
+as much infrastructure, semantics, and implementation details as possible
+with the @option{parallel-tests} harness and its default driver. So
+everything said in the previous section should apply here too, unless
+explicitly stated otherwise.
+
+@menu
+* Command-line arguments for test drivers::
+* Log files generation and test results recording::
+* Testsuite progress output::
+* HTML generation from testsuite logs::
+@end menu
+
+@node Command-line arguments for test drivers
+@subsubsection Command-line arguments for test drivers
+
+A custom driver can rely on various command-line options and arguments
+being passed to it automatically by the Automake's @option{parallel-tests}
+harness. It is @emph{mandatory} that it understands all of them (even
+if the exact interpretation of the associated semantics can legitimately
+change between a test driver and another, and even be a no-op in some
+drivers).
+
+@noindent
+Here is the list of options:
+
+@table @option
+@item --test-name=@var{NAME}
+The name of the test, with VPATH prefix (if any) removed. This can have a
+suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
+mostly meant to be used in console reports about testsuite advancements and
+results (@pxref{Testsuite progress output}).
+@item --log-file=@var{PATH}
+The log file the test driver must create. If it has a directory component
+(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
+such directory exist @emph{before} the test driver is called.
+@item --color-tests=@{yes|no@}
+Whether the console output should be colorized or not (@pxref{Simple
+tests and color-tests}, to learn when this option gets activated and
+when it doesn't).
+@item --expect-failure=@{yes|no@}
+Whether the tested program is expected to fail.
+@item --enable-hard-errors=@{yes|no@}
+Whether ``hard errors'' in the tested program should be treated differently
+from normal failures or not (the default should be @code{yes}). The exact
+meaning of ``hard error'' is highly dependent from the test protocols or
+conventions in use.
+@item --
+Explicitly terminate the list of options.
+@end table
+
+The first of the remaining arguments passed to the test driver is the
+program to be run, and the other arguments are command-line options
+and arguments for this program.
+
+Note that the exact semantics attached to the @option{--color-tests},
+@option{--expect-failure} and @option{--enable-hard-errors} options are
+left up to the individual test drivers. Still, having a behaviour
+compatible or at least similar to that provided by the default
+@option{parallel-tests} driver is advised, as that would offer a better
+consistency and a more pleasant user experience.
+
+@node Log files generation and test results recording
+@subsubsection Log files generation and test results recording
+
+The test driver must correctly generate the file specified by the
+@option{--log-file} option (of course even when the tested program
+fails or crashes). This file is quite free-form, but still it has
+to conform to the following conventions, in order to work with the
+@option{parallel-tests} harness and take advantage of its features.
+
+@itemize @bullet
+@item
+@cindex Global test script result
+@cindex Test result, global
+The @emph{global test result} for the whole test script must be placed
+in the first line of the @file{.log} file.
+@c The following semantics is checked by tests 'test-driver-recheck.test'
+@c and 'test-driver-global-log.test'. Please keep them in sync whenever
+@c the following is changed.
+If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
+@code{SKIP:}, the test script will be considered failed, and it will be
+re-run by @code{make recheck}. Also, if this line does not begin with
+either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file
+will be copied into the global @file{test-suite.log} (in order to help in
+debugging and bug-report analysis).
+
+@item
+@cindex Single test case result
+@cindex Test result, single
+@cindex @code{:test-result:} reStructuredText field
+@cindex reStructuredText field, @code{:test-result:}
+One of the main features of the new testsuite harness is the ability to
+support test protocols that allow a single test script to run more
+test cases, @emph{each with its distinct result}. In order for the
+testsuite summary to be correct in this case, the test driver must
+register @emph{each} such result in the generated @file{.log} file, using
+the @code{:test-result:} reStructuredText field. Otherwise, only the
+global test result will be considered (as it is the case of the default
+@option{parallel-tests} driver).
+
+@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
+The only recognized test results are currently @code{PASS}, @code{XFAIL},
+@code{SKIP}, @code{FAIL} and @code{XPASS}. These results, when declared
+with @code{:test-result:}, can be optionally followed by text holding
+the name and/or a brief description of the corresponding test; the
+@option{parallel-tests} harness will ignore such extra text when
+generating @file{test-suite.log} and preparing the testsuite summary.
+
+For example, if a @file{.log} file contains the following lines:
+
+@example
+:test-result: PASS server starts
+:test-result: PASS HTTP/1.1 request
+:test-result: FAIL HTTP/1.0 request
+:test-result: SKIP HTTPS request (TLS library wasn't available)
+:test-result: PASS server stops
+@end example
+
+@noindent
+it means that the corresponding test script contributes with five test
+results to the testsuite summary (three of these tests being successful,
+one failed, and one skipped).
+
+@c FIXME: currently, the @code{:test-result:} field is recognized anywhere
+@c in the @file{.log} file; this is possibly prone to generating spurious
+@c results, in case of verbose tests. We should allow a special
+@c @code{:test-result:} that stops any further parsing (maybe a line
+@c @code{:test-result:END} will do?).
+
+@end itemize
+
+Finally, note that the declaration of the global test result in the first
+line, apart from being needed for backward-compatibility with the default
+@option{parallel-tests} driver, can be useful also for test protocols
+that allow more test results per test script: using it, a custom test
+driver is allowed to decide what the global outcome of the test script
+is in case of conflicting test results within the script. For example,
+if a test script reports 8 successful test cases and 2 skipped test
+cases, some drivers might report that globally as a SKIP, while others
+as a PASS.
+
+@node Testsuite progress output
+@subsubsection Testsuite progress output
+
+A custom test driver also has the task of displaying, on the standard
+output, the test results as soon as they become available. Depending on
+the protocol in use, it can also display the reasons for failures and
+skips, and, more generally, any useful diagnostic output (but remember
+that each line on the screen is precious, so that cluttering the screen
+with overly verbose information is bad idea). The exact format of this
+progress output is left up to the test driver; in fact, a custom test
+driver might @emph{theoretically} even decide not to do any such report,
+leaving it all to the testsuite summary (that would be a very lousy idea,
+of course, and serves only to illustrate the flexibility that is
+granted here).
+
+Remember that consistency is good; so, if possible, try to be consistent
+with the output of the built-in Automake test drivers, providing a similar
+``look & feel''. In particular, the testsuite progress output should be
+colorized when the @option{--color-tests} is passed to the driver. On the
+other end, if you are using a known and widespread test protocol with
+well-established implementations, being consistent with those
+implementations' output might be a good idea too.
+
+@c TODO: Give an example, maybe inspired to py.test-style output.
+@c TODO: That is a good idea because it shows a test driver that allows
+@c TODO: for different levels of verbosity in the progress output (could
+@c TODO: be implemented either using a driver cmdline flag, or an
+@c TODO: environment variable, or both).
+
+@node HTML generation from testsuite logs
+@subsubsection HTML generation from testsuite logs
+
+If HTML testsuite output (with @code{check-html}) is to be supported,
+the generated @file{.log} files must contain syntactically valid
+reStructuredText (among the other things, this means that every
+@code{:test-result:} line must be followed by a blank line). If this
+is not the case, the HTML generation will not work, although all the
+other functionalities of the Automake testsuite harness should remain
+untouched, and continue to work correctly.
@node DejaGnu Tests
@section DejaGnu Tests
@@ -9240,8 +9580,10 @@ in the first few lines of the @file{NEWS} file.
@item @option{color-tests}
@cindex Option, @option{color-tests}
@opindex color-tests
-Cause output of the simple test suite (@pxref{Simple Tests}) to be
-colorized on capable terminals.
+Cause output of the simple and parallel test suites (see @ref{Simple Tests}
+and @ref{Simple Tests using parallel-tests}) and of properly-written custom
+test drivers (@pxref{Custom Test Drivers}) to be colorized on capable
+terminals.
@item @option{dejagnu}
@cindex Option, @option{dejagnu}
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 9c344f5..e1867fb 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -464,7 +464,7 @@ recheck recheck-html:
list=`for f in $$list; do \
test -f $$f || continue; \
if test -r $$f && read line < $$f; then \
- case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
diff --git a/lib/am/check.am b/lib/am/check.am
index 062c59c..b02d6c5 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -205,16 +205,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
-## Here we assume that the test driver writes a proper summary for the
-## test script on the first line. Requiring this is not a limitation,
-## but a feature, since this way a custom test driver is allowed to decide
-## what the outcome is in case of conflicting testcase results in a test
-## script. For example, if a test script reports 8 successful testcases
-## and 2 skipped testcases, some drivers might report that globally as a
-## SKIP, while others as a PASS.
-## FIXME: This should be documented in the automake manual. The above
-## FIXME: explanation is indeed more appropriate for the manual than for
-## FIXME: comments in code.
+## Here we assume that the test driver writes a proper "summarizing
+## result" for the test script on the first line.
for f in $$list; do \
test -r $$f && read line < $$f || line=; \
case $$line in \
@@ -317,7 +309,8 @@ recheck recheck-html:
## test script on the first line. See the comments in the rules of
## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
## advisable.
- case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
+## A test whose log is unreadable is to be considered failed.
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b49e754..0b83b0c 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -738,6 +738,10 @@ test-driver-custom-multitest-recheck.test \
test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
+test-driver-create-log-dir.test \
+test-driver-global-log.test \
+test-driver-recheck.test \
+test-driver-strip-vpath.test \
test-driver-fail.test \
parse.test \
percent.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 0219eea..193fc95 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -998,6 +998,10 @@ test-driver-custom-multitest-recheck.test \
test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
+test-driver-create-log-dir.test \
+test-driver-global-log.test \
+test-driver-recheck.test \
+test-driver-strip-vpath.test \
test-driver-fail.test \
parse.test \
percent.test \
@@ -1510,7 +1514,7 @@ recheck recheck-html:
list=`for f in $$list; do \
test -f $$f || continue; \
if test -r $$f && read line < $$f; then \
- case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
diff --git a/tests/test-driver-create-log-dir.test b/tests/test-driver-create-log-dir.test
new file mode 100755
index 0000000..5713b92
--- /dev/null
+++ b/tests/test-driver-create-log-dir.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: if a log file has a directory component (as in
+# e.g., `sub/foo.log'), the Automake test harness must ensure that
+# directory exists before calling any custom test driver.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+# The extra `.' components below are meant.
+TESTS = sub/foo sub/subsub/bar.test ././sub1/./baz
+$(TESTS):
+LOG_DRIVER = $(srcdir)/checkdir-driver
+TEST_LOG_DRIVER = $(LOG_DRIVER)
+EXTRA_DIST = checkdir-driver
+
+check-local: $(TEST_SUITE_LOG)
+ test -d sub
+ test -d sub1
+ test -d sub/subsub
+ test -f sub/foo.log
+ test -f sub/subsub/bar.log
+ test -f sub1/baz.log
+END
+
+cat > checkdir-driver <<'END'
+#! /bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "log: $log_file" # For debugging.
+case $log_file in */*);; *) exit 1;; esac
+dir=`expr "$log_file" : '\(.*\)/[^/]*'`
+echo "dir: $dir" # For debugging.
+test -d "$dir" || exit 1
+echo dummy > "$log_file"
+END
+chmod a+x checkdir-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+$MAKE check
+$MAKE distcheck
+
+:
diff --git a/tests/test-driver-global-log.test b/tests/test-driver-global-log.test
new file mode 100755
index 0000000..cdaaff6
--- /dev/null
+++ b/tests/test-driver-global-log.test
@@ -0,0 +1,110 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the output of testcases having a global outcome that is not
+# "PASS" or "XFAIL" is copied in the global 'test-suite.log'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+# The :test-result: fields should be irrelevant for the decision
+# of whether a test output is to be copied in the test-suite.log.
+cat > pass.test <<END
+PASS: pass.test
+:test-result: SKIP
+:test-result: FAIL
+:test-result: XFAIL
+:test-result: XPASS
+:test-result: ERROR
+not seen (pass)
+END
+
+cat > xfail.test <<'END'
+XFAIL: xfail.test
+not seen (xfail)
+END
+
+echo SKIP: skip.test > skip.test
+echo FAIL: fail.test > fail.test
+echo XPASS: xpass.test > xpass.test
+echo :test-result: PASS > fake-pass.test
+echo "$tab $tab$tab" > empty.test
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./dummy-driver
+TESTS = pass.test skip.test fail.test xfail.test xpass.test \
+ fake-pass.test empty.test
+END
+
+
+cat > dummy-driver <<'END'
+#!/bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+cp $1 $log_file
+END
+chmod a+x dummy-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check && { cat test-suite.log; Exit 1; }
+cat test-suite.log
+grep '^SKIP: skip\.test$' test-suite.log
+grep '^FAIL: fail.test$' test-suite.log
+grep '^XPASS: xpass.test$' test-suite.log
+grep '^:test-result: PASS$' test-suite.log
+grep "^$tab $tab$tab$" test-suite.log
+$EGREP 'not seen' test-suite.log && Exit 1
+test `grep -c ':test-result:' test-suite.log` -eq 1
+
+cat > pass.test <<END
+PASS: pass.test
+:test-result: SKIP
+:test-result: XFAIL
+not seen (pass)
+END
+
+cat > xfail.test <<END
+XFAIL: xfail.test
+:test-result: SKIP
+:test-result: PASS
+not seen (xfail)
+END
+
+TESTS='pass.test skip.test xfail.test' $MAKE -e check \
+ || { cat test-suite.log; Exit 1; }
+cat test-suite.log
+grep '^SKIP: skip\.test$' test-suite.log
+$EGREP ':test-result:|not seen' test-suite.log && Exit 1
+
+:
diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
new file mode 100755
index 0000000..2ae75b9
--- /dev/null
+++ b/tests/test-driver-recheck.test
@@ -0,0 +1,126 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test the "make recheck" semantics for custom test driver, as
+# documented in detail in the Automake manual.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+LOG_DRIVER = ./dummy-driver
+TEST_EXTENSIONS =
+TESTS =
+END
+
+rechecked=''
+notrechecked=''
+
+for R in PASS XFAIL SKIP; do
+ echo $R: > $R-1
+ echo " $R:" > $R-2
+ echo $R:foo > $R-3
+ echo "$tab $R: bar baz" > $R-4
+ # The :test-result: fields should be irrelevant for the decision of
+ # whether "make recheck" should or should not re-run a test.
+ unindent > $R-5 <<END
+ $R:
+ :test-result: FAIL
+ :test-result: XPASS
+ :test-result: ERROR
+END
+ notrechecked="$notrechecked $R-1 $R-2 $R-3 $R-4 $R-5"
+ echo $R > BAD-$R-1
+ echo $R. > BAD-$R-2
+ echo :$R: > BAD-$R-3
+ echo $R asd > BAD-$R-4
+ rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
+done
+
+for R in FAIL XPASS UNKNOWN; do
+ echo $R: > $R-1
+ echo $R:foo > $R-2
+ echo $R: bar baz > $R-3
+ echo $R > $R-4
+ echo $R asd > $R-5
+ # The :test-result: fields should be irrelevant for the decision of
+ # whether "make recheck" should or should not re-run a test.
+ unindent > $R-6 <<END
+ :test-result: PASS
+ :test-result: SKIP
+ :test-result: XFAIL
+END
+ (echo $R: && cat $R-6) > $R-7
+ rechecked="$rechecked $R-1 $R-2 $R-3 $R-4 $R-5 $R-6 $R-7"
+done
+
+: > EMPTY
+echo " $tab $tab" > WHITE
+rechecked="$rechecked EMPTY WHITE"
+
+tests="$rechecked $notrechecked"
+
+for t in $tests; do echo $t; done | sed 's/.*/TESTS += &/' >> Makefile.am
+
+cat Makefile.am # For debugging.
+
+cat > dummy-driver <<'END'
+#!/bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+: > $test_name.run
+cp $1 $log_file
+END
+chmod a+x dummy-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# The ':test-result:' fields should be ignored by "make recheck",
+# but should cause the testsuite report to detect errors.
+$MAKE check && Exit 1
+ls -l
+for t in $tests; do test -f $t.run; done
+rm -f *.run
+
+# But now the tests that actually get re-run have only ':test-result:'
+# fields indicating success, so "make recheck" must pass. Still, the
+# next "make recheck" call should still re-run the same set of tests.
+for iteration in 1 2; do
+ $MAKE recheck
+ ls -l
+ for t in $rechecked; do test -f $t.run; done
+ for t in $notrechecked; do test ! -r $t.run; done
+ rm -f *.run
+done
+
+:
diff --git a/tests/test-driver-strip-vpath.test b/tests/test-driver-strip-vpath.test
new file mode 100755
index 0000000..f1b316c
--- /dev/null
+++ b/tests/test-driver-strip-vpath.test
@@ -0,0 +1,96 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: check that the test name passed to the test
+# driver has any VPATH prefix stripped.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+ocwd=`pwd` || fatal_ "cannot get current working directory"
+
+mkdir src build
+mv install-sh missing configure.in src
+rm -f depcomp
+
+cd src
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+# The directories of the `bar.test' and `baz.test' tests are deliberately
+# called as the source directory, to verify that the VPATH-stripping code
+# doesn't get too easily confused.
+# The $(empty) are for eliciting VPATH rewrites on make implementations
+# that support it (e.g., Solaris make), to improve coverage.
+empty =
+TESTS = $(empty) foo.test src/bar.test ./src/baz.test $(empty)
+$(TESTS):
+TEST_LOG_DRIVER = $(srcdir)/checkstrip-driver
+EXTRA_DIST = checkstrip-driver
+END
+
+cat > checkstrip-driver <<'END'
+#! /bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "test name: $test_name" # For debugging.
+case $test_name in
+ foo.test|./foo.test|src/ba[rz].test|./src/ba[rz].test);;
+ *) exit 1;;
+esac
+echo dummy > "$log_file"
+END
+chmod a+x checkstrip-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+cd ..
+
+mkdir build1
+cd build1
+../src/configure
+# "$MAKE -n" is for debugging, should highlight any VPATH rewrite.
+$MAKE -n check
+$MAKE check
+cd ..
+
+mkdir build2
+cd build2
+"$ocwd"/src/configure
+# "$MAKE -n" is for debugging, should highlight any VPATH rewrite.
+$MAKE -n check
+$MAKE check
+cd ..
+
+cd src
+./configure
+$MAKE distcheck
+
+:
--
1.7.2.3
