* doc/automake.texi ("Using the TAP test protocol"): Divide this
section into ...
("Introduction to TAP", "Use TAP with the Automake test harness",
"Incompatibilities with other TAP parsers and drivers", "Links
and external resources"): ... these subsections, extend them by
adding more information and examples, and improve them by removing
incomplete and/or temporary wordings and TODO items.
("Script-based Testsuites", "Parallel Test Harness"): Add a couple
of anchors to improve the granularity of cross-references.
* tests/tap-doc2.test: New test, verifying the correctness of the
new examples given in the manual.
* tests/Makefile.am (tap_other_tests): Add the new test.
---
ChangeLog | 9 ++-
doc/automake.texi | 181 ++++++++++++++++++++++++++++++++++++--------------
tests/Makefile.am | 1 +
tests/Makefile.in | 1 +
tests/tap-doc2.test | 139 +++++++++++++++++++++++++++++++++++++++
5 files changed, 276 insertions(+), 55 deletions(-)
create mode 100755 tests/tap-doc2.test
diff --git a/ChangeLog b/ChangeLog
index d09b308..6d2a016 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -6,12 +6,13 @@
("Introduction to TAP", "Use TAP with the Automake test harness",
"Incompatibilities with other TAP parsers and drivers", "Links
and external resources"): ... these subsections, extend them by
- adding more information, and improve them by removing incomplete
- and/or temporary wordings and TODO items. Also ...
- ("Links and external resources"): ... add a link to the `tap4j'
- project.
+ adding more information and examples, and improve them by removing
+ incomplete and/or temporary wordings and TODO items.
("Script-based Testsuites", "Parallel Test Harness"): Add a couple
of anchors to improve the granularity of cross-references.
+ * tests/tap-doc2.test: New test, verifying the correctness of the
+ new examples given in the manual.
+ * tests/Makefile.am (tap_other_tests): Add the new test.
2011-08-01 Stefano Lattarini <stefano.lattar...@gmail.com>
diff --git a/doc/automake.texi b/doc/automake.texi
index 8ac3687..c1ee0a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -330,6 +330,13 @@ Simple Tests
* Serial Test Harness:: Older (and obsolescent) serial test harness
* Parallel Test Harness:: Generic concurrent test harness
+Using the TAP test protocol
+
+* Introduction to TAP::
+* Use TAP with the Automake test harness::
+* Incompatibilities with other TAP parsers and drivers::
+* Links and external resources::
+
Custom Test Drivers
* Overview of Custom Test Drivers Support::
@@ -9534,7 +9541,7 @@ harness should remain untouched, and continue to work
correctly.
@subsection Introduction to TAP
TAP, the Test Anything Protocol, is a simple text-based interface between
-a test harness and testing modules or programs. The tests (also called
+testing modules or programs and a test harness. The tests (also called
``TAP producers'' in this context) write test results in a simple format
on standard output; a test harness (also called ``TAP consumer'') will
parse and interpret these results, and properly present them to the user,
@@ -9543,72 +9550,144 @@ is accomplished can vary among different test
harnesses. The Automake
parallel harness will present the results on the console in the usual
fashion (@pxref{Testsuite progress on console}), and will use the
@file{.trs} files (@pxref{Basics of test metadata}) to store the test
-results and related metadata.
+results and related metadata. Apart from that, it will try to remain
+as much compatible as possible with pre-existing and widespread utilities,
+such as the @uref{http://search.cpan.org/~andya/Test-Harness/bin/prove,
+@command{prove} utility}, at least for the simpler usages.
TAP started its life as part of the test harness for Perl, but today
it has been (mostly) standardized, and has various independent
-implementations in different languages (among them, C, C++, Perl,
-Python, PHP, and Java).
+implementations in different languages; among them, C, C++, Perl,
+Python, PHP, and Java. For a semi-official specification of the
+TAP protocol, please refer to the documentation of
+@uref{http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
+ @samp{Test::Harness::TAP}}.
-The most relevant real-world usages of TAP are obviously in the testsuite
-of @command{perl} and of many perl modules; but also other important
+The most relevant real-world usages of TAP are obviously in the testsuites
+of @command{perl} and of many perl modules. Still, also other important
third-party packages, such as @uref{http://git-scm.com/, @command{git}},
-uses TAP in their testsuite.
-
-The ``core'' of the TAP protocol is probably best summarized by this
-brief excerpt taken from the TAP specification at
-@uref{http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
- @samp{Test::Harness::TAP}}:
-@quotation
-A harness must only read TAP output from standard output and not
-from standard error. Lines written to standard output matching
-@code{/^(not )?ok\b/} must be interpreted as test lines. All other
-lines must not be considered test output.
-@end quotation
-@noindent
-Please refer to the above link for the complete TAP specifications.
+use TAP in their testsuite.
@node Use TAP with the Automake test harness
@subsection Use TAP with the Automake test harness
Currently, the TAP driver that comes with Automake requires a perl
-interpreter to work, and requires the developer taking various steps to
-set it up. You'll have grab the @file{tap-driver} script from the
-Automake distribution by hand, copy in in your source tree, add code
-to @file{configure.ac} to search a perl interpreter and define the
-@code{$(PERL)} variable accordingly, and use the Automake support for
-third-party test drivers to.
-
-Apart from the options that must be accepted by all the Automake test
-drivers, the @file{tap-driver}
-Option @option{--ignore-exit} causes the driver to ignore the exit
-status of the test scripts; by default, the driver will report an
-error if the script exit with status != 0.
-@emph{TODO}: also add a @option{--no-ignore-exit} option, for
-completeness?
-
-By default, TAP diagnostic (i.e., lines beginning with the @samp{#}
-character) are copied only in the @file{.log} file. The option
-@option{--comments} causes the driver to display them in the testsuite
-progress output too (@emph{TODO}: give example). The option
-@option{--no-comments} restore the default behaviour.
+interpreter to work, and requires various by-hand steps on the
+developer's part (this should be fixed in future Automake versions).
+You'll have grab the @file{tap-driver} script from the Automake
+distribution by hand, copy it in your source tree, add code to
+@file{configure.ac} to search a perl interpreter and to define the
+@code{$(PERL)} variable accordingly, and use the Automake support
+for third-party test drivers to instruct the harness to use the
+@file{tap-driver} to run your TAP-producing tests. See the example
+below for clarification.
+
+Apart from the options common to all the Automake test drivers
+(@pxref{Command-line arguments for test drivers}), the @file{tap-driver}
+supports the following options, whose names are chosen for enhanced
+compatibility with the @command{prove} utility.
+@table @option
+@item --ignore-exit
+Causes the test driver to ignore the exit status of the test scripts;
+by default, the driver will report an error if the script exit with a
+non-zero status.
+@item --comments
+Instruct the test driver to display TAP diagnostic (i.e., lines beginning
+with the @samp{#} character) in the testsuite progress output too; by
+default, TAP diagnostic is only copied in the @file{.log} file.
+@item --no-comments
+Revert the effects of @option{--comments}.
+@item --merge
+Instruct the test driver to merge the test scripts' standard error into
+their standard output. This is necessary if you want to ensure that
+diagnostics from the test scripts are displayed in the correct order
+relative to test results; this can be of great help in debugging
+(especially if your test scripts are shell scripts run with shell
+tracing active). As a downside, this option might cause the test
+harness to get confused if anything that appears on standard error
+looks like a test result.
+@item --no-merge
+Revert the effects of @option{--merge}.
+@end table
@noindent
-Here is an example of how to setup be done
+Here is an example of how the TAP driver can be set up and used.
+
+@c Keep in sync with tap-doc2.test.
@example
-TEST_LOG_DRIVER = $(srcdir)/tap-driver
-TESTS = foo.test bar.test
-@end example
+% @kbd{cat configure.ac}
+AC_INIT([GNU Try Tap], [1.0], [bug-automake@@gnu.org])
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
+AC_CONFIG_FILES([Makefile])
+AC_REQUIRE_AUX_FILE([tap-driver])
+AC_PATH_PROG([PERL], [perl])
+test -n "$PERL" || AC_MSG_ERROR([perl not found])
+$PERL -MTAP::Parser -e 1 || AC_MSG_ERROR([TAP::Parser not found])
+AC_OUTPUT
+
+% @kbd{cat Makefile.am}
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/build-aux/tap-driver
+TESTS = foo.test bar.test baz.test
+EXTRA_DIST = $(TESTS)
-@b{TODO}: Give example of output from @command{prove} command (when
-used in ``verbose'' mode, with option @option{-v}), to which we'd like
-to remain somewhat compatible compatible, while also (and foremost)
-being compatible with the default @option{parallel-tests} driver.
+% @kbd{cat foo.test}
+#!/bin/sh
+echo 1..4 # Number of tests to be executed.
+echo 'ok 1 - Swallows fly'
+echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
+echo 'ok 3 - Pigs fly # SKIP not enough acid'
+echo '# I just love word plays ...'
+echo 'ok 4 - Flies fly too :-)'
+
+% @kbd{cat bar.test}
+#!/bin/sh
+echo 1..3
+echo 'not ok 1 - Bummer, this test has failed.'
+echo 'ok 2 - This passed though.'
+echo 'Bail out! Ennui kicking in, sorry...'
+echo 'ok 3 - This will not be seen.'
+
+% @kbd{cat baz.test}
+#!/bin/sh
+echo 1..1
+echo ok 1
+# Exit with error, even if all the test case has been successful.
+exit 7
-@b{TODO}: Example of output from out own TAP testsuite driver.
-Point out similarities with the output from @command{prove --verbose}
-and from the default @option{parallel-tests} driver.
+% @kbd{cp @var{PREFIX}/share/automake-@var{APIVERSION}/tap-driver .}
+% @kbd{autoreconf -vi && ./configure && make check}
+...
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+PASS: foo.test 4 - Flies fly too :-)
+FAIL: bar.test 1 - Bummer, this test has failed.
+PASS: bar.test 2 - This passed though.
+ERROR: bar.test - Bail out! Ennui kicking in, sorry...
+PASS: baz.test 1
+ERROR: baz.test - exited with status 7
+...
+Please report to bug-automake@@gnu.org
+...
+% @kbd{echo exit status: $?}
+exit status: 1
+
+@c Keep the "skewed" indentation below, it produces pretty PDF output.
+% @kbd{env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
+ TESTS='foo.test baz.test' make -e check}
+...
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+# foo.test: I just love word plays...
+PASS: foo.test 4 - Flies fly too :-)
+PASS: baz.test 1
+...
+% @kbd{echo exit status: $?}
+exit status: 0
+@end example
@node Incompatibilities with other TAP parsers and drivers
@subsection Incompatibilities with other TAP parsers and drivers
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b8126f2..6a45fcf 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1184,6 +1184,7 @@ tap-bad-prog.test \
tap-bad-prog2.test \
tap-basic.test \
tap-doc.test \
+tap-doc2.test \
tap-empty.test \
tap-more.test \
tap-more2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 82e62a3..a085762 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1417,6 +1417,7 @@ tap-bad-prog.test \
tap-bad-prog2.test \
tap-basic.test \
tap-doc.test \
+tap-doc2.test \
tap-empty.test \
tap-more.test \
tap-more2.test \
diff --git a/tests/tap-doc2.test b/tests/tap-doc2.test
new file mode 100755
index 0000000..63bdc23
--- /dev/null
+++ b/tests/tap-doc2.test
@@ -0,0 +1,139 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that an example given in the documentation really works.
+# See section "Using the TAP test protocol", subsection "Use TAP
+# with the Automake test harness".
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+# Ensure we are run from the right directory.
+# (The last thing we want is to delete some random user files.)
+test -f ../defs-static && test -f ../defs || Exit 99
+rm -f *
+
+cat > Makefile.am <<'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/build-aux/tap-driver
+TESTS = foo.test bar.test baz.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > configure.ac <<'END'
+AC_INIT([GNU Try Tap], [1.0], [bug-autom...@gnu.org])
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
+AC_CONFIG_FILES([Makefile])
+AC_REQUIRE_AUX_FILE([tap-driver])
+AC_PATH_PROG([PERL], [perl])
+test -n "$PERL" || AC_MSG_ERROR([perl not found])
+$PERL -MTAP::Parser -e 1 || AC_MSG_ERROR([TAP::Parser not found])
+AC_OUTPUT
+END
+
+cat > foo.test <<'END'
+#!/bin/sh
+echo 1..4 # Number of tests to be executed.
+echo 'ok 1 - Swallows fly'
+echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
+echo 'ok 3 - Pigs fly # SKIP not enough acid'
+echo '# I just love word plays...'
+echo 'ok 4 - Flies fly too :-)'
+END
+
+cat > bar.test <<'END'
+#!/bin/sh
+echo 1..3
+echo 'not ok 1 - Bummer, this test has failed.'
+echo 'ok 2 - This passed though.'
+echo 'Bail out! Ennui kicking in, sorry...'
+echo 'ok 3 - This will not be seen.'
+END
+
+cat > baz.test <<'END'
+#!/bin/sh
+echo 1..1
+echo ok 1
+# Exit with error, even if all the test case has been successful.
+exit 7
+END
+
+chmod a+x *.test
+
+mkdir build-aux
+cp "$top_testsrcdir"/lib/tap-driver build-aux
+
+(export AUTOMAKE ACLOCAL AUTOCONF && $AUTORECONF -vi) || Exit 1
+
+./configure --help # Sanity check.
+./configure || skip_ "configure failed"
+
+case $MAKE in *\ -j*) skip_ "can't work easily with concurrent make";; esac
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+cat > exp <<'END'
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+PASS: foo.test 4 - Flies fly too :-)
+FAIL: bar.test 1 - Bummer, this test has failed.
+PASS: bar.test 2 - This passed though.
+ERROR: bar.test - Bail out! Ennui kicking in, sorry...
+PASS: baz.test 1
+ERROR: baz.test - exited with status 7
+END
+
+sed -n '/^PASS: foo\.test/,/^ERROR: baz\.test/p' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+grep '^Please report to bug-automake@gnu\.org$' stdout
+
+env \
+ TESTS='foo.test baz.test' \
+ TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
+ $MAKE -e check >stdout || { cat stdout; Exit 1; }
+
+cat > exp <<'END'
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+# foo.test: I just love word plays...
+PASS: foo.test 4 - Flies fly too :-)
+PASS: baz.test 1
+END
+
+sed -n '/^PASS: foo\.test/,/^PASS: baz\.test/p' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Sanity check the distribution.
+cat > bar.test <<'END'
+#!/bin/sh
+echo 1..1
+echo ok 1
+END
+echo AM_TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile.in
+./config.status Makefile
+$MAKE distcheck
+
+:
--
1.7.2.3
