Hi Richard - here's my tinkering with the description in the manual (pushed). Let me know if you see any problems.
Meanwhile, I guess we can close this. If anything more is pending from your side, let me know or feel free to re-open. Thanks again. -k ----------------------------------------------------------------------------- doc: move most --stderr-prefix doc to a new subsubsection. Still in https://bugs.gnu.org/72536. * doc/automake.texi (TAP setup and examples): (TAP stderr examples), TAP stderr prefixing): split TAP examples into new subsubsections. Add discussion of race condition and other reordering from the open bug. * NEWS: mention --stderr-prefix. diff --git a/NEWS b/NEWS index 34876f3db..26b433314 100644 --- a/NEWS +++ b/NEWS @@ -6,6 +6,9 @@ New in 1.x: * New features added + - New option --stderr-prefix for tap-driver.sh, to prefix each line of + stderr from a test script with a given string. (bug#72536) + - Support for Algol 68 added, based on the GNU Algol 68 compiler. (bug#75807) - New option dist-bzip3 for bzip3 compression of distributions. (bug#73795) @@ -18,6 +21,9 @@ New in 1.x: - Avoid Perl 5.41.8+ precedence warning for use of !!. (https://lists.gnu.org/archive/html/automake/2025-01/msg00000.html) + - a Perl path containing whitespace now emits a warning instead of + an error, so ./configure PERL='/usr/bin/env perl' can work. (bug#74453) + - The py-compile script once again does nothing (successfully) if the PYTHON environment variable is set to ":", or anything that isn't a Python interpreter (according to $PYTHON -V). Exception: if PYTHON diff --git a/doc/automake.texi b/doc/automake.texi index 9fa94689f..bb66aea5c 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -10516,6 +10516,7 @@ with the @samp{#} character) in the testsuite progress output too; by default, TAP diagnostics are only copied to the @file{.log} file. @item --no-comments Revert the effects of @option{--comments}. + @item --merge Instruct the test driver to merge the test scripts' standard error into their standard output. This is necessary if you want to ensure that @@ -10526,8 +10527,10 @@ tracing active). As a downside, this option might cause the test harness to get confused if anything that appears on standard error looks like a test result. Use @option{--stderr-prefix} to prefix each line of standard error to avoid this problem. + @item --no-merge Revert the effects of @option{--merge}. + @item --stderr-prefix @var{STRING} Prefix each line of the test script's standard error with @code{@var{STRING}}. Defaults to the empty string. This option makes @@ -10549,39 +10552,13 @@ literal characters @samp{@@%:@@} in the prefix (@samp{@@%@@&t@@:@@}, for example). @end table -For example, in shell, @samp{--stderr-prefix "@@%:@@[stderr] "} will -prefix each line of the test script's standard error with -@samp{#[stderr] }. - -When prefixing standard error lines, the test program's standard error -lines can be processed out of order relative to the test program's -standard output lines, even when @option{--merge} is used. This can be -due to many factors, including: - -@itemize @bullet -@item -Output buffering in the test program: The test program might be -accumulating output data and writing it to the operating system in -batches, not as soon as it is available. You can try -@uref{man:setvbuf(3),,setvbuf(3)} to change the buffering mode in the -test program, or you can add regular flushes. OS-specific utilities to -control buffering may be available, such as -@uref{man:stdbuf(1),,stdbuf(1)} on GNU/Linux systems. +For example, the option @samp{--stderr-prefix "@@%:@@[stderr] "} +passed on the command line will prefix each line of the test script's +standard error with @samp{#[stderr] }. -@item -Concurrent processing of standard output and standard error: Due to a -limitation of the shell command language, the TAP driver processes -standard output in a different thread or process than standard error. -Thus, processing order is influenced by how the operating system -schedules the two threads or processes for execution. - -@item -Line-based processing: When prefixing standard error, the TAP driver -waits until a complete line is read from the test program before -processing the line. This reduces the chances that characters from a -standard output line are mixed with characters from a standard error -line. -@end itemize +For more details on @option{--stderr-prefix} processing, especially +the occasional output reordering due to a race condition, @pxref{TAP +prefixing stderr}. @item --diagnostic-string @var{STRING} Change the string that introduces TAP diagnostics from the default value @@ -10596,7 +10573,18 @@ the ``official'' TAP protocol does not allow for such a customization; so don't use it if you can avoid it. @end table -@noindent +@menu +* TAP setup and examples:: Setup and examples of TAP usage. +* TAP stderr examples:: Examples of stderr issues and processing. +* TAP prefixing stderr:: Details about @option{--stderr-prefix} processing. +@end menu + +@node TAP setup and examples +@subsubsection TAP setup and examples + +@cindex TAP setup, example of +@cindex setting up TAP, example + Here is an example of how the TAP driver can be set up and used. @c Keep in sync with tap-doc2.sh @@ -10670,7 +10658,19 @@ PASS: baz.test 1 ... % @kbd{echo exit status: $?} exit status: 0 +@end example +@node TAP stderr examples +@subsubsection TAP stderr examples + +@cindex TAP stderr example +@cindex stderr example in TAP + +Here is a sequence of examples showing the issue with merging stdout +and stderr, and the resolution with @option{--stderr-prefix}. Read +the comments for specifics. + +@example % @kbd{cat stderr-demo.test} #!/bin/sh echo 'ok 1' @@ -10721,9 +10721,69 @@ PASS: stderr-demo.test 1 ok 2 PASS: stderr-demo.test 2 1..2 - @end example +@node TAP prefixing stderr +@subsubsection TAP prefixing stderr + +@cindex TAP prefixing stderr +@cindex prefixing stderr in TAP +@cindex stderr prefixing in TAP +@cindex race condition in stderr prefixing + +@c see https://bugs.gnu.org/72536 for this feature and discussion. + +When using the above @option{--stderr-prefix} TAP option, the test +program's standard error lines might occasionally be processed out of +order relative to the test program's standard output lines, +particularly when @option{--merge} is also used. This can be due to +many factors, including: + +@itemize @bullet +@item +Output buffering in the test program: The test program might be +accumulating output data and writing it to the operating system in +batches, not as soon as it is available. You can try setvbuf(3) to +change the buffering mode in the test program, or you can add regular +flushes. OS-specific utilities to control buffering may be available, +such as stdbuf(1) on GNU/Linux systems. + +@item +Concurrent processing of standard output and standard error: Due to a +limitation of the shell command language, the TAP driver processes +standard output in a different thread or process than standard error, +causing a race condition. Thus, processing order is influenced by how +the operating system schedules the two threads or processes for +execution. + +@item +Line-based processing: When prefixing standard error, the TAP driver +waits until a complete line is read from the test program before +processing the line. This reduces the chances that characters from a +standard output line are mixed with characters from a standard error +line. +@end itemize + +Thus, stderr lines might appear out of order relative to stdout lines, +even when @option{--merge} is passed. All stderr lines are still +processed in order relative to other stderr lines, and the same goes +for the stdout lines relative to other stdout lines. So the +reordering shouldn't break a package's TAP tests, since there's no +reason to output TAP commands to stderr; even more so when stderr +lines are prefixed. + +When working on a particular test, if you find the reordering is +causing trouble, you can omit @option{--stderr-prefix}, or run the +test directly (not via TAP) to get a more accurate understanding of +output timing. + +Overall, though, the order of processing (and logging) should not +differ from the order of the test program's write(2) calls by much. +Out-of-order processing only happens if the test program writes to +stdout then stderr but the kernel wakes up the TAP driver's +thread/process that is reading stderr before it wakes up the TAP +driver's thread/process that is reading stdout, or vice versa. + @node Incompatibility with other TAP parsers and drivers @subsection Incompatibility with other TAP parsers and drivers compile finished at Sun Feb 23 09:55:43 2025
