Package: devscripts
Version: 2.23.6
Severity: wishlist
Tags: patch
Hey.
Shamlessly copy&pasting my text from:
https://salsa.debian.org/debian/devscripts/-/merge_requests/377
This is some major overhaul of annotate-output.
The core parts are IMO:
- getting rid of any shell variables set by the script itself, which
may override variables that the caller wants to export to the
invoked program (a355a37f1e285673d0fcb54fdff87893646848b9,
383a82917fe180cc3a45dec520afa139b552e093 and
ecea512825e964d073c74f6c9961ced44d5d3809)
- greatly improving performance when no date conversion specifiers are
used (c236a7ecb4c178d72a4f477be451948831684063)
Per default the behaviour (in terms of output format) should not be
changed, at least the stuff that might get parsed by someone.
Please notice that I did some ugly reference of another commit ID (that
of a355a37f1e285673d0fcb54fdff87893646848b9) in some commits. So if you
don’t apply that on top of the current master or change commits, please
adapt the commit messages, or let me do it.
Other than that, please review and merge[0] :-)
Thanks, Chris.
[0] Or even better, let someone skilled in C and especially IO streams
(the latter being a weakness of mine), re-implemented it properly.
PS: I'm attaching the patch series here, too, though I guess I'd prefer
to discuss things on salsa/Gitlab.
>From 4091b4a3c12d8ef98fd949d09ab53d9dfaaf6efa Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 05:23:36 +0100
Subject: [PATCH 01/23] annotate-output: fixed typo
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 84025f58..da308252 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -70,7 +70,7 @@ echo Started "$@" | handler $formatter I
# The following block redirects FD 2 (stderr) to FD 1 (stdout) which is then
# processed by the stderr handler. It redirects FD 1 (stdout) to FD 4 such
-# that it can later be move to FD 1 (stdout) and handled by the stdout handler.
+# that it can later be moved to FD 1 (stdout) and handled by the stdout
handler.
# The exit status of the program gets written to FD 2 (stderr) which is then
# captured to produce the correct exit status as the last step of the pipe.
# Both the stdout and stderr handler output to FD 3 such that after exiting
--
2.43.0
From a355a37f1e285673d0fcb54fdff87893646848b9 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 05:42:22 +0100
Subject: [PATCH 02/23] annotate-output: prevent `PROGNAME`-variable from
leaking to command
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
annotate-output is a wrapper program that runs other programs. Such programs may
in turn make use of arbitrary environment variables.
Every variable that is set within the script, may have been already in the
environment of the shell (because it was already exported when `annotate-output`
itself was called).
The script would then overwrite the variable and export it further to the actual
program, which would not get the value as expected by the user.
A simple example (before this commit) would have been:
```
$ export PROGNAME=foo; annotate-output env | grep -E ^.{12}PROGNAME=
15:50:19 O: PROGNAME=annotate-output
```
Unfortunately, the problem cannot be fully solved in the Shell Command Language,
amongst others because:
- Shell always unconditionally set some own variables - which again my have
already been marked as exported.
- Environment variables need to to be valid shell variables (e.g. `*=value`) is
in principle a valid environment variable, but not a shell variable) and it’s
unspecified whether such are still passed on to commands (called from the
shell) or not.
A re-implementation in e.g. C would be needed to overcome this limitation (side
note: even Python cannot be made 100% transparent of the environment, see
https://github.com/python/cpython/issues/93446).
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index da308252..25307a17 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -14,8 +14,6 @@
set -eu
-PROGNAME=${0##*/}
-
handler() {
while IFS= read -r line; do
printf "%s %s: %s\n" "$($1)" "$2" "$line"
@@ -27,7 +25,7 @@ handler() {
usage() {
echo \
-"Usage: $PROGNAME [options] program [args ...]
+"Usage: ${0##*/} [options] program [args ...]
Run program and annotate STDOUT/STDERR with a timestamp.
Options:
--
2.43.0
>From 554612e126d5fa6bafb3af2fad50d8e442dde777 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 06:26:48 +0100
Subject: [PATCH 03/23] annotate-output: define `handler`-function in a
definition function
This prepares for subsequent commits and merely moves the definition of the
`handler`-function into a definition function, which is then called.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 25307a17..80951668 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -14,14 +14,17 @@
set -eu
-handler() {
- while IFS= read -r line; do
- printf "%s %s: %s\n" "$($1)" "$2" "$line"
- done
- if [ -n "$line" ]; then
- printf "%s %s: %s" "$($1)" "$2" "$line"
- fi
+define_handler() {
+ handler() {
+ while IFS= read -r line; do
+ printf "%s %s: %s\n" "$($1)" "$2" "$line"
+ done
+ if [ -n "$line" ]; then
+ printf "%s %s: %s" "$($1)" "$2" "$line"
+ fi
+ }
}
+define_handler
usage() {
echo \
--
2.43.0
From c236a7ecb4c178d72a4f477be451948831684063 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 06:42:13 +0100
Subject: [PATCH 04/23] annotate-output: split up `handler`-function for date
and plain cases
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This splits up the single `handler`-function into two, one for the case when a
date conversion specifier is used and one for the case when not.
It uses the previously introduced definition function approach, to get the
`handler`-function defined as either of its two implementations – but always
with the same function name.
The respective definition function is called, depending on the format string.
The benefit of all this are performance improvements.
On my system, with a date conversion specifier (the default one):
```
$ time annotate-output.sh tree /usr/share/doc
```
takes previously: against now:
real 1m14,336s real 1m5,457s
user 1m2,140s user 0m52,115s
sys 0m17,953s sys 0m17,058s
The improvements in this case come likely from the no longer needed call to the
`datefmt`-function and perhaps a bit from not needing to expand `$1`.
~10s is admittedly not much, but the gain gets far bigger when not having a
date conversion specifier:
```
$ time annotate-output.sh + tree /usr/share/doc
```
takes previously: against now:
real 0m19,067s real 0m2,418s
user 0m11,738s user 0m0,910s
sys 0m9,784s sys 0m2,585s
The improvements in this case come likely from the no longer needed subshell
(from the command substitution) and perhaps a bit from the saved `printf`.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 80951668..384a998e 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -14,17 +14,27 @@
set -eu
-define_handler() {
+define_handler_with_date_conversion_specifiers() {
handler() {
while IFS= read -r line; do
- printf "%s %s: %s\n" "$($1)" "$2" "$line"
+ printf "%s %s: %s\n" "$(date "$FMT")" "$2" "$line"
done
if [ -n "$line" ]; then
- printf "%s %s: %s" "$($1)" "$2" "$line"
+ printf "%s %s: %s" "$(date "$FMT")" "$2" "$line"
+ fi
+ }
+}
+
+define_handler_with_plain_prefix() {
+ handler() {
+ while IFS= read -r line; do
+ printf "%s %s: %s\n" "$FMT" "$2" "$line"
+ done
+ if [ -n "$line" ]; then
+ printf "%s %s: %s" "$FMT" "$2" "$line"
fi
}
}
-define_handler
usage() {
echo \
@@ -63,8 +73,8 @@ plainfmt() { printf "%s" "$FMT"; }
# shellcheck disable=SC2317
datefmt() { date "$FMT"; }
case "$FMT" in
- *%*) formatter=datefmt;;
- *) formatter=plainfmt; FMT="${FMT#+}";;
+ *%*) formatter=datefmt; define_handler_with_date_conversion_specifiers;;
+ *) formatter=plainfmt; FMT="${FMT#+}";
define_handler_with_plain_prefix;;
esac
echo Started "$@" | handler $formatter I
--
2.43.0
>From 6ebc513c07b842b501912d47f943dd23577c1a7b Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 07:09:00 +0100
Subject: [PATCH 05/23] annotate-output: removed no longer needed code
These remnants became obsolete with the previous commit.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 24 ++++++++++--------------
1 file changed, 10 insertions(+), 14 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 384a998e..4657f050 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -17,10 +17,10 @@ set -eu
define_handler_with_date_conversion_specifiers() {
handler() {
while IFS= read -r line; do
- printf "%s %s: %s\n" "$(date "$FMT")" "$2" "$line"
+ printf "%s %s: %s\n" "$(date "$FMT")" "$1" "$line"
done
if [ -n "$line" ]; then
- printf "%s %s: %s" "$(date "$FMT")" "$2" "$line"
+ printf "%s %s: %s" "$(date "$FMT")" "$1" "$line"
fi
}
}
@@ -28,10 +28,10 @@ define_handler_with_date_conversion_specifiers() {
define_handler_with_plain_prefix() {
handler() {
while IFS= read -r line; do
- printf "%s %s: %s\n" "$FMT" "$2" "$line"
+ printf "%s %s: %s\n" "$FMT" "$1" "$line"
done
if [ -n "$line" ]; then
- printf "%s %s: %s" "$FMT" "$2" "$line"
+ printf "%s %s: %s" "$FMT" "$1" "$line"
fi
}
}
@@ -68,16 +68,12 @@ if [ $# -lt 1 ]; then
exit 1
fi
-# shellcheck disable=SC2317
-plainfmt() { printf "%s" "$FMT"; }
-# shellcheck disable=SC2317
-datefmt() { date "$FMT"; }
case "$FMT" in
- *%*) formatter=datefmt; define_handler_with_date_conversion_specifiers;;
- *) formatter=plainfmt; FMT="${FMT#+}";
define_handler_with_plain_prefix;;
+ *%*) define_handler_with_date_conversion_specifiers;;
+ *) FMT="${FMT#+}"; define_handler_with_plain_prefix;;
esac
-echo Started "$@" | handler $formatter I
+echo Started "$@" | handler I
# The following block redirects FD 2 (stderr) to FD 1 (stdout) which is then
# processed by the stderr handler. It redirects FD 1 (stdout) to FD 4 such
@@ -93,11 +89,11 @@ err=0
{
{
"$@" 2>&1 1>&4 3>&- 4>&-; echo $? >&2;
- } | handler $formatter E >&3;
- } 4>&1 | handler $formatter O >&3;
+ } | handler E >&3;
+ } 4>&1 | handler O >&3;
} 2>&1;
} | { read -r xs; exit "$xs"; };
} 3>&1 || err=$?
-echo "Finished with exitcode $err" | handler $formatter I
+echo "Finished with exitcode $err" | handler I
exit $err
--
2.43.0
>From 812c944548600dd90918ea0d1a7b07d3e9c0d7a3 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 07:17:42 +0100
Subject: [PATCH 06/23] annotate-output: removed static separator between
date/prefix and stream indicator
This prepare to allow for removing the separator (a space) between the
date/prefix and stream indicator in a future commit.
It should not have any effect on the output format as it is.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 4657f050..2f1a929b 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -17,10 +17,10 @@ set -eu
define_handler_with_date_conversion_specifiers() {
handler() {
while IFS= read -r line; do
- printf "%s %s: %s\n" "$(date "$FMT")" "$1" "$line"
+ printf "%s%s: %s\n" "$(date "$FMT")" "$1" "$line"
done
if [ -n "$line" ]; then
- printf "%s %s: %s" "$(date "$FMT")" "$1" "$line"
+ printf "%s%s: %s" "$(date "$FMT")" "$1" "$line"
fi
}
}
@@ -28,10 +28,10 @@ define_handler_with_date_conversion_specifiers() {
define_handler_with_plain_prefix() {
handler() {
while IFS= read -r line; do
- printf "%s %s: %s\n" "$FMT" "$1" "$line"
+ printf "%s%s: %s\n" "$FMT" "$1" "$line"
done
if [ -n "$line" ]; then
- printf "%s %s: %s" "$FMT" "$1" "$line"
+ printf "%s%s: %s" "$FMT" "$1" "$line"
fi
}
}
@@ -46,11 +46,11 @@ usage() {
-h, --help - Show this message"
}
-FMT="+%H:%M:%S"
+FMT="+%H:%M:%S "
while [ -n "${1-}" ]; do
case "$1" in
+*)
- FMT="$1"
+ FMT="$1 "
shift
;;
-h|-help|--help)
--
2.43.0
From 383a82917fe180cc3a45dec520afa139b552e093 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 07:36:19 +0100
Subject: [PATCH 07/23] annotate-output: prevent `err`-variable from leaking to
command
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Analogous to commit a355a37f1e285673d0fcb54fdff87893646848b9 (“annotate-output:
annotate-output: prevent `PROGNAME`-variable from leaking to command”).
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 2f1a929b..aabd7443 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -82,7 +82,6 @@ echo Started "$@" | handler I
# captured to produce the correct exit status as the last step of the pipe.
# Both the stdout and stderr handler output to FD 3 such that after exiting
# with the correct exit code, FD 3 can be redirected to FD 1 (stdout).
-err=0
{
{
{
@@ -93,7 +92,5 @@ err=0
} 4>&1 | handler O >&3;
} 2>&1;
} | { read -r xs; exit "$xs"; };
-} 3>&1 || err=$?
-
-echo "Finished with exitcode $err" | handler I
-exit $err
+} 3>&1 && { echo "Finished with exitcode 0" | handler I; exit 0;
} \
+ || { err=$?; echo "Finished with exitcode $err" | handler I; exit $err;
}
--
2.43.0
From 680aba4dfee43a6943540eb5cf23c7fb7ce29d62 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 16:09:19 +0100
Subject: [PATCH 08/23] annotate-output: double quoted parameter expansions
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The script does (and should) not set `IFS` to a sane value, thus in principle it
may contain digits and thereby affect parameter expansions of positional and
special parameters.
While this is in principle not necessary for the assignments, it’s done there
nevertheless for the case that the code would be reused in some places where
field splitting does happen.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index aabd7443..4e8d3530 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -63,7 +63,7 @@ while [ -n "${1-}" ]; do
esac
done
-if [ $# -lt 1 ]; then
+if [ "$#" -lt 1 ]; then
usage
exit 1
fi
@@ -87,10 +87,10 @@ echo Started "$@" | handler I
{
{
{
- "$@" 2>&1 1>&4 3>&- 4>&-; echo $? >&2;
+ "$@" 2>&1 1>&4 3>&- 4>&-; echo "$?" >&2;
} | handler E >&3;
} 4>&1 | handler O >&3;
} 2>&1;
} | { read -r xs; exit "$xs"; };
} 3>&1 && { echo "Finished with exitcode 0" | handler I; exit 0;
} \
- || { err=$?; echo "Finished with exitcode $err" | handler I; exit $err;
}
+ || { err="$?"; echo "Finished with exitcode $err" | handler I; exit
"$err"; }
--
2.43.0
From 1873f414263f9934af7e99a59c41d8d9b6c4e643 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 16:58:27 +0100
Subject: [PATCH 09/23] annotate-output: use exit status `127` if no program
was specified
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
If after option parsing no arguments remain, no program to be executed was
specified.
POSIX.1-2017 reserves the exit status `127` for cases when a command to be
executed was not found in several places of the standard:
- For the Shell Command Language in chapters “2.8.2 Exit Status for Commands”
and “2.9.1 Simple Commands”.
- For the `exec`-special-built-in-utility in chapter “2.14. Special Built-In
Utilities”.
- In general, in the rationale of the `exit`-special-built-in-utility in “2.14.
Special Built-In Utilities”
- For the `nice`-, `nohup`-, `sh`, and `time`-utilities in chapter “4.
Utilities”.
(all in “Volume 3, Shell and Utilities”).
This seems to also be the best fit, for the case when no program was specified.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 4e8d3530..8dc0a0ee 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -65,7 +65,7 @@ done
if [ "$#" -lt 1 ]; then
usage
- exit 1
+ exit 127
fi
case "$FMT" in
--
2.43.0
>From d8b6aefb34ef0e4180daa0d5bfac2d272c6a2a21 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 17:15:54 +0100
Subject: [PATCH 10/23] annotate-output: documented exit statuses
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.1 | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/scripts/annotate-output.1 b/scripts/annotate-output.1
index 2352221e..3e43f24d 100644
--- a/scripts/annotate-output.1
+++ b/scripts/annotate-output.1
@@ -15,7 +15,21 @@ Controls the timestamp format, as per \fBdate\fR(1).
Defaults to
"%H:%M:%S".
.TP
\fB\-h\fR, \fB\-\-help\fR
-Display a help message and exit successfully.
+Display a help message.
+
+.SH EXIT STATUS
+If \fIprogram\fR is invoked, the exit status of \fBannotate\-output\fR
+shall be the exit status of \fIprogram\fR; otherwise,
+\fBannotate\-output\fR shall exit with one of the following values:
+.TP
+0
+\fB\-h\fR or \fB\-\-help\fR was used.
+.TP
+126
+\fIprogram\fR was found but could not be invoked.
+.TP
+127
+\fIprogram\fR could not be found or was not specified.
.SH EXAMPLE
--
2.43.0
From 1979fc4c378237056810b5f04a9baf831dc75537 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 17:17:15 +0100
Subject: [PATCH 11/23] annotate-output: removed undocumented `-help`-option
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Long option names with only one leading `-` are rather uncommon and cause
problems should further short options be added.
Since the option is anyway not documented, neither in the manpage nor in the
program’s usage output, it’s simply removed.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 8dc0a0ee..3e750962 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -53,7 +53,7 @@ while [ -n "${1-}" ]; do
FMT="$1 "
shift
;;
- -h|-help|--help)
+ -h|--help)
usage
exit 0
;;
--
2.43.0
From 750a43142b717e106a9b768f996b839b77d8c61b Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 18:03:37 +0100
Subject: [PATCH 12/23] annotate-output: improved the manpage
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This prepares for changes from follow-up commits by improving the description of
the program output and the format string, in particular by adding missing
information, like the documentation of the `I`-stream indicator and the exact
formatting of the whole lines.
It further applies upper-case spelling of `STDOUT` and `STDERR` like it’s
already used in the program’s usage.
Further it more clearly differentiates between “this” program (that is
`annotate-output`) and the program that is executed by it.
It also indicates that multiple options can in principle be given.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.1 | 40 ++++++++++++++++++++++++---------------
1 file changed, 25 insertions(+), 15 deletions(-)
diff --git a/scripts/annotate-output.1 b/scripts/annotate-output.1
index 3e43f24d..61905d16 100644
--- a/scripts/annotate-output.1
+++ b/scripts/annotate-output.1
@@ -2,17 +2,27 @@
.SH NAME
annotate-output \- annotate program output with time and stream
.SH SYNOPSIS
-\fBannotate\-output\fR [\fIoptions\fR] \fIprogram\fR [\fIargs\fR ...]
+\fBannotate\-output\fR [\fIoptions\fR ...] \fIprogram\fR [\fIargs\fR ...]
.SH DESCRIPTION
-\fBannotate\-output\fR will execute the specified program, while
-prepending every line with the current time and O for stdout and E for
-stderr.
+\fBannotate\-output\fR executes \fIprogram\fR with \fIargs\fR as arguments
+and prepends printed lines with a format string followed by an indicator
+for the stream on which the line was printed followed by a colon and a
+single space.
+.br
+The stream indicators are \fBI\fR for information from
+\fBannotate\-output\fR as well as \fBO\fR for STDOUT and \fBE\fR for STDERR
+from \fIprogram\fR.
+
.SH OPTIONS
.TP
\fB+FORMAT\fR
-Controls the timestamp format, as per \fBdate\fR(1). Defaults to
-"%H:%M:%S".
+A format string that may use the conversion specifiers from the
+\fBdate\fR(1)-utility. The printed string is separated from the following
+stream indicator by a single space. May be overridden by later options that
+specify the format string.
+.br
+Defaults to "%H:%M:%S".
.TP
\fB\-h\fR, \fB\-\-help\fR
Display a help message.
@@ -44,26 +54,26 @@ $ annotate-output make
.fi
.SH BUGS
-Since stdout and stderr are processed in parallel, it can happen that
-some lines received on stdout will show up before later-printed stderr
+Since STDOUT and STDERR are processed in parallel, it can happen that
+some lines received on STDOUT will show up before later-printed STDERR
lines (and vice-versa).
This is unfortunately very hard to fix with the current annotation
strategy. A fix would involve switching to PTRACE'ing the process.
-Giving nice a (much) higher priority over the executed program could
+Giving nice a (much) higher priority over \fIprogram\fR could
however cause this behaviour to show up less frequently.
-The program does not work as well when the output is not linewise. In
-particular, when an interactive program asks for input, the question
-might not be shown until after you have answered it. This will give
-the impression that the annotated program has hung, while it has not.
+\fBannotate\-output\fR does not work as well when the output is not
+linewise. In particular, when an interactive \fIprogram\fR asks for input,
+the question might not be shown until after you have answered it. This
+will give the impression that \fIprogram\fR has hung, while it has not.
.SH "SEE ALSO"
\fBdate\fR(1)
.SH SUPPORT
-This program is community-supported (meaning: you'll need to fix it
-yourself). Patches are however appreciated, as is any feedback
+\fBannotate\-output\fR is community-supported (meaning: you'll need to fix
+it yourself). Patches are however appreciated, as is any feedback
(positive or negative).
.SH AUTHOR
--
2.43.0
>From 2134bf4d10a54f413df64e9ef96bf4235fece2cf Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 18:05:58 +0100
Subject: [PATCH 13/23] annotate-output: improved program usage output
Align it with the changes to the manpage in the previous commit.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 3e750962..9bf3d4a7 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -38,12 +38,17 @@ define_handler_with_plain_prefix() {
usage() {
echo \
-"Usage: ${0##*/} [options] program [args ...]
- Run program and annotate STDOUT/STDERR with a timestamp.
+"Usage: ${0##*/} [OPTIONS ...] PROGRAM [ARGS ...]
+ Executes PROGRAM with ARGS as arguments and prepends printed lines with a
+ format string, a stream indicator and \`: \`.
Options:
- +FORMAT - Controls the timestamp format as per date(1)
- -h, --help - Show this message"
+ +FORMAT - A format string that may use the conversion specifiers from the
+ \`date\`(1)-utility.
+ The printed string is separated from the following stream
+ indicator by a single space.
+ Defaults to \`%H:%M:%S\`.
+ -h, --help - Display this help message."
}
FMT="+%H:%M:%S "
--
2.43.0
From a8ef760415e0f02a1956f838381f648688ba01a3 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 18:12:22 +0100
Subject: [PATCH 14/23] annotate-output: print concrete error message rather
than just the usage
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
If the actual problem is known, it’s better to print a concrete error message
than the rather useless program usage, which doesn’t tell the user what went
wrong.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 9bf3d4a7..1d5f93ea 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -69,7 +69,7 @@ while [ -n "${1-}" ]; do
done
if [ "$#" -lt 1 ]; then
- usage
+ printf '%s: No program to be executed was specified.\n' "${0##*/}" >&2
exit 127
fi
--
2.43.0
From 85edfa0af69d74fd91dca85cc834afcc62236c0f Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 19:05:45 +0100
Subject: [PATCH 15/23] annotate-output: added option that allows date format
string with no separator
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This option mainly allows to print the date format string as `+FORMAT` does, but
without the following space that separates it from the stream indicator, which
may be of use if one wants to print no date at all (what currently causes a
useless leading space).
It’s added as a new option, to leave `+FORMAT` backwards compatible.
The name `--raw-date-format` was chosen, rather than for example just
`--raw-format` as future versions of the program may want to allow for more
generic format strings (for example such, where the stream indicator or even the
line is also a conversion specifier).
If no format is specified, the program exits with `125`.
This is used for the same purpose by several GNU Core Utilities (for example
`nice`, `nohup`, `stdbuf` and `timeout`).
POSIX.1-2017 reserves in principle only `126` and `127` as “special” exit
statuses and both do not really fit (though it does indeed specify the `127`
exit status for the `nohup` utility to mean both, the utility itself having
failed or the utility invoked by it having not been found).
It does however specify the exit statuses `1` up to `125` for the `nice` and
`time` utilities to mean an error in the respective utilities themselves, which
overlaps with exit statuses that may be returned by the utilities invoked by
them.
Since `annotate-output` is conceptually similar to `time` (annotating output) it
was chosen that `125` is a proper choice, despite not being 100% POSIX.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.1 | 9 +++++++++
scripts/annotate-output.sh | 24 ++++++++++++++++++------
2 files changed, 27 insertions(+), 6 deletions(-)
diff --git a/scripts/annotate-output.1 b/scripts/annotate-output.1
index 61905d16..9f6e5263 100644
--- a/scripts/annotate-output.1
+++ b/scripts/annotate-output.1
@@ -24,6 +24,12 @@ specify the format string.
.br
Defaults to "%H:%M:%S".
.TP
+\fB--raw-date-format\fR \fIFORMAT\fR
+A format string that may use the conversion specifiers from the
+\fBdate\fR(1)-utility. There is no separator between the printed string and
+the following stream indicator. May be overridden by later options that
+specify the format string.
+.TP
\fB\-h\fR, \fB\-\-help\fR
Display a help message.
@@ -35,6 +41,9 @@ shall be the exit status of \fIprogram\fR; otherwise,
0
\fB\-h\fR or \fB\-\-help\fR was used.
.TP
+125
+An error occurred in \fBannotate\-output\fR.
+.TP
126
\fIprogram\fR was found but could not be invoked.
.TP
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 1d5f93ea..331e7f12 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -43,12 +43,16 @@ usage() {
format string, a stream indicator and \`: \`.
Options:
- +FORMAT - A format string that may use the conversion specifiers from the
- \`date\`(1)-utility.
- The printed string is separated from the following stream
- indicator by a single space.
- Defaults to \`%H:%M:%S\`.
- -h, --help - Display this help message."
+ +FORMAT - A format string that may use the conversion
+ specifiers from the \`date\`(1)-utility.
+ The printed string is separated from the
following
+ stream indicator by a single space.
+ Defaults to \`%H:%M:%S\`.
+ --raw-date-format FORMAT - A format string that may use the conversion
+ specifiers from the \`date\`(1)-utility.
+ The printed string is not separated from the
+ following stream indicator.
+ -h, --help - Display this help message."
}
FMT="+%H:%M:%S "
@@ -58,6 +62,14 @@ while [ -n "${1-}" ]; do
FMT="$1 "
shift
;;
+ --raw-date-format)
+ if [ "$#" -lt 2 ]; then
+ printf '%s: The `--raw-date-format`-option requires an
argument.\n' "${0##*/}" >&2
+ exit 125
+ fi
+ FMT="+$2"
+ shift 2
+ ;;
-h|--help)
usage
exit 0
--
2.43.0
>From 701ac60ed0fd78d2fc75e1d17592a6668310c48a Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 19:26:29 +0100
Subject: [PATCH 16/23] annotate-output: support `--` to end option parsing
Previously, it would not have been possible to execute a program that has the
name of an option recognised by `annotate-output`.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.1 | 5 ++++-
scripts/annotate-output.sh | 6 +++++-
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/scripts/annotate-output.1 b/scripts/annotate-output.1
index 9f6e5263..aa0b7b4c 100644
--- a/scripts/annotate-output.1
+++ b/scripts/annotate-output.1
@@ -2,7 +2,7 @@
.SH NAME
annotate-output \- annotate program output with time and stream
.SH SYNOPSIS
-\fBannotate\-output\fR [\fIoptions\fR ...] \fIprogram\fR [\fIargs\fR ...]
+\fBannotate\-output\fR [\fIoptions\fR ...] [--] \fIprogram\fR [\fIargs\fR ...]
.SH DESCRIPTION
\fBannotate\-output\fR executes \fIprogram\fR with \fIargs\fR as arguments
and prepends printed lines with a format string followed by an indicator
@@ -30,6 +30,9 @@ A format string that may use the conversion specifiers from
the
the following stream indicator. May be overridden by later options that
specify the format string.
.TP
+\fB--\fR
+Ends option parsing (unless it is itself an argument to an option).
+.TP
\fB\-h\fR, \fB\-\-help\fR
Display a help message.
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 331e7f12..6e09daba 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -38,7 +38,7 @@ define_handler_with_plain_prefix() {
usage() {
echo \
-"Usage: ${0##*/} [OPTIONS ...] PROGRAM [ARGS ...]
+"Usage: ${0##*/} [OPTIONS ...] [--] PROGRAM [ARGS ...]
Executes PROGRAM with ARGS as arguments and prepends printed lines with a
format string, a stream indicator and \`: \`.
@@ -74,6 +74,10 @@ while [ -n "${1-}" ]; do
usage
exit 0
;;
+ --)
+ shift
+ break
+ ;;
*)
break
;;
--
2.43.0
From f378a6ff2d94e53e1d26408763981188dd2fc99e Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 19:40:24 +0100
Subject: [PATCH 17/23] annotate-output: switched from `echo` to `printf`
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
`echo` cannot portably be used, unless `-n` is used and escape sequences are
omitted.
Especially the `$0` and `$@` could have included escape sequences.
Since `printf` is anyway also built-in in all reasonable shells, switched
everywhere to that.
It does not matter, that `$?` and `$err` are used directly in the format string,
as these can only ever contain digits which will never get special meaning in
there.
Strictly speaking, it’s not necessary to set `IFS=`, but it costs nothing and
makes it more clear how `"$*"` shall be expanded.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 6e09daba..a69a1b4a 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -37,22 +37,23 @@ define_handler_with_plain_prefix() {
}
usage() {
- echo \
-"Usage: ${0##*/} [OPTIONS ...] [--] PROGRAM [ARGS ...]
+ printf \
+'Usage: %s [OPTIONS ...] [--] PROGRAM [ARGS ...]
Executes PROGRAM with ARGS as arguments and prepends printed lines with a
- format string, a stream indicator and \`: \`.
+ format string, a stream indicator and `: `.
Options:
+FORMAT - A format string that may use the conversion
- specifiers from the \`date\`(1)-utility.
+ specifiers from the `date`(1)-utility.
The printed string is separated from the
following
stream indicator by a single space.
- Defaults to \`%H:%M:%S\`.
+ Defaults to `%%H:%%M:%%S`.
--raw-date-format FORMAT - A format string that may use the conversion
- specifiers from the \`date\`(1)-utility.
+ specifiers from the `date`(1)-utility.
The printed string is not separated from the
following stream indicator.
- -h, --help - Display this help message."
+ -h, --help - Display this help message.
+' "${0##*/}"
}
FMT="+%H:%M:%S "
@@ -94,7 +95,7 @@ case "$FMT" in
*) FMT="${FMT#+}"; define_handler_with_plain_prefix;;
esac
-echo Started "$@" | handler I
+IFS= printf 'Started %s\n' "$*" | handler I
# The following block redirects FD 2 (stderr) to FD 1 (stdout) which is then
# processed by the stderr handler. It redirects FD 1 (stdout) to FD 4 such
@@ -108,10 +109,10 @@ echo Started "$@" | handler I
{
{
{
- "$@" 2>&1 1>&4 3>&- 4>&-; echo "$?" >&2;
+ "$@" 2>&1 1>&4 3>&- 4>&-; printf "$?\n" >&2;
} | handler E >&3;
} 4>&1 | handler O >&3;
} 2>&1;
} | { read -r xs; exit "$xs"; };
-} 3>&1 && { echo "Finished with exitcode 0" | handler I; exit 0;
} \
- || { err="$?"; echo "Finished with exitcode $err" | handler I; exit
"$err"; }
+} 3>&1 && { printf 'Finished with exitcode 0\n' | handler I; exit
0; } \
+ || { err="$?"; printf "Finished with exitcode $err\n" | handler I; exit
"$err"; }
--
2.43.0
>From ef4223b2b7c21a2d93c3189ef631ebd52914b25c Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 19:59:52 +0100
Subject: [PATCH 18/23] annotate-output: sanitise `IFS` when reading the exit
status
Again, strictly speaking not necessary, but costs nothing and indicates that
whatever `IFS` might ever be changed to in the future within the script, the
read must not do field splitting.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index a69a1b4a..3d589693 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -113,6 +113,6 @@ IFS= printf 'Started %s\n' "$*" | handler I
} | handler E >&3;
} 4>&1 | handler O >&3;
} 2>&1;
- } | { read -r xs; exit "$xs"; };
+ } | { IFS= read -r xs; exit "$xs"; };
} 3>&1 && { printf 'Finished with exitcode 0\n' | handler I; exit
0; } \
|| { err="$?"; printf "Finished with exitcode $err\n" | handler I; exit
"$err"; }
--
2.43.0
From ecea512825e964d073c74f6c9961ced44d5d3809 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Sun, 10 Dec 2023 22:50:40 +0100
Subject: [PATCH 19/23] annotate-output: prevent `FMT`-variable from leaking to
command
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Analogous to commit a355a37f1e285673d0fcb54fdff87893646848b9 (“annotate-output:
annotate-output: prevent `PROGNAME`-variable from leaking to command”).
This removes the need for the `FMT`-variable altogether, by “hardcoding” it into
the new generically (via `eval`) performed definitions of the
`handler`-function.
First, the function is always defined once to get the default of `+%H:%M:%S `
set. Then, depending on the program options, the function may be re-defined
numerous times (this was also already the case, when there were not even the
`define_handler_*`-functions and is also documented meanwhile.
Using `eval` is always tricky:
- The leading space in `eval " handler() {` is not strictly necessary here, but
nevertheless good practise, as `eval` is not required to support `--` and
POSIX recommends a leading space instead.
While we don't evaluate a string with a leading `-`, this might in principle
change in the future.
- As this commit wants to “hardcode” the user-supplied format string into the
function it must be escaped, otherwise unusual format strings like:
```
$ annotate-output '+foo"; ls; printf "bar' date
```
(or smilar, depending on the actual non-escaping implementation)
would lead to code-injection (above, the `ls` would be executed).
Properly escaping a string for literal re-use as shell input is difficult, but
some implementations of `printf` have added the – as of now still – non-POISX
`%q` conversion specifier which performs the escaping.
I myself has requested this to be standardised but was too late for the
upcoming new revision of POSIX, though the working group agreed to at least
reserve `%q` and `%Q` for that purpose (see
https://austingroupbugs.net/view.php?id=1771).
So it’s actually not so unlikely, that this will become true POSIX sooner or
later.
Anyway, meanwhile, `printf` from the GNU Core Utilities can be used.
It’s anyway executed only at most twice per invocation and since the
`coreutils` is an essential package in Debian, no new dependency is intrduced
by that.
With this commit, there should be no more variables set by our script that may
be exported into the invoked program.
Only those environment variables for which the shell (usually `dash`) sets its
own variables unconditionally or that are lost because they’re not valid shell
variables, may not be passed on properly from a caller to the invoked program.
This would however require a re-implementation in a language like C.
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 43 ++++++++++++++++++++++----------------
1 file changed, 25 insertions(+), 18 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 3d589693..936ee0e9 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -14,26 +14,31 @@
set -eu
+# TODO: Switch from using `/usr/bin/printf` to the (likely built-in) `printf`
+# once POSIX has standardised `%q` for that
+# (see https://austingroupbugs.net/view.php?id=1771) and `dash`
+# implemented it.
+
define_handler_with_date_conversion_specifiers() {
- handler() {
+ eval " handler() {
while IFS= read -r line; do
- printf "%s%s: %s\n" "$(date "$FMT")" "$1" "$line"
+ printf \"%s%s: %s\\n\" \"\$(date $(/usr/bin/printf '%q'
"$1") )\" \"\$1\" \"\$line\"
done
- if [ -n "$line" ]; then
- printf "%s%s: %s" "$(date "$FMT")" "$1" "$line"
+ if [ -n \"\$line\" ]; then
+ printf \"%s%s: %s\" \"\$(date $(/usr/bin/printf '%q'
"$1") )\" \"\$1\" \"\$line\"
fi
- }
+ }"
}
define_handler_with_plain_prefix() {
- handler() {
+ eval " handler() {
while IFS= read -r line; do
- printf "%s%s: %s\n" "$FMT" "$1" "$line"
+ printf \"%s%s: %s\\n\" $(/usr/bin/printf '%q' "$1")
\"\$1\" \"\$line\"
done
- if [ -n "$line" ]; then
- printf "%s%s: %s" "$FMT" "$1" "$line"
+ if [ -n \"\$line\" ]; then
+ printf \"%s%s: %s\" $(/usr/bin/printf '%q' "$1")
\"\$1\" \"\$line\"
fi
- }
+ }"
}
usage() {
@@ -56,11 +61,15 @@ usage() {
' "${0##*/}"
}
-FMT="+%H:%M:%S "
+define_handler_with_date_conversion_specifiers "+%H:%M:%S "
while [ -n "${1-}" ]; do
case "$1" in
+ +*%*)
+ define_handler_with_date_conversion_specifiers "$1 "
+ shift
+ ;;
+*)
- FMT="$1 "
+ define_handler_with_plain_prefix "${1#+} "
shift
;;
--raw-date-format)
@@ -68,7 +77,10 @@ while [ -n "${1-}" ]; do
printf '%s: The `--raw-date-format`-option requires an
argument.\n' "${0##*/}" >&2
exit 125
fi
- FMT="+$2"
+ case "$2" in
+ *%*) define_handler_with_date_conversion_specifiers
"+$2";;
+ *) define_handler_with_plain_prefix "${2#+}";;
+ esac
shift 2
;;
-h|--help)
@@ -90,11 +102,6 @@ if [ "$#" -lt 1 ]; then
exit 127
fi
-case "$FMT" in
- *%*) define_handler_with_date_conversion_specifiers;;
- *) FMT="${FMT#+}"; define_handler_with_plain_prefix;;
-esac
-
IFS= printf 'Started %s\n' "$*" | handler I
# The following block redirects FD 2 (stderr) to FD 1 (stdout) which is then
--
2.43.0
>From 503beebec99e1a04f86f9541d404ec7401901bcb Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Mon, 11 Dec 2023 01:00:48 +0100
Subject: [PATCH 20/23] annotate-output: unify spelling of `STDOUT` and
`STDERR`
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 936ee0e9..512096b4 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -104,13 +104,13 @@ fi
IFS= printf 'Started %s\n' "$*" | handler I
-# The following block redirects FD 2 (stderr) to FD 1 (stdout) which is then
-# processed by the stderr handler. It redirects FD 1 (stdout) to FD 4 such
-# that it can later be moved to FD 1 (stdout) and handled by the stdout
handler.
-# The exit status of the program gets written to FD 2 (stderr) which is then
+# The following block redirects FD 2 (STDERR) to FD 1 (STDOUT) which is then
+# processed by the STDERR handler. It redirects FD 1 (STDOUT) to FD 4 such
+# that it can later be moved to FD 1 (STDOUT) and handled by the STDOUT
handler.
+# The exit status of the program gets written to FD 2 (STDERR) which is then
# captured to produce the correct exit status as the last step of the pipe.
-# Both the stdout and stderr handler output to FD 3 such that after exiting
-# with the correct exit code, FD 3 can be redirected to FD 1 (stdout).
+# Both the STDOUT and STDERR handler output to FD 3 such that after exiting
+# with the correct exit code, FD 3 can be redirected to FD 1 (STDOUT).
{
{
{
--
2.43.0
>From cc7ba17c187e5c0698542e4b8403d55ec5ac63ee Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Mon, 11 Dec 2023 01:10:12 +0100
Subject: [PATCH 21/23] annotate-output: reformat program usage
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.sh | 29 ++++++++++++++++-------------
1 file changed, 16 insertions(+), 13 deletions(-)
diff --git a/scripts/annotate-output.sh b/scripts/annotate-output.sh
index 512096b4..6a2ba6bc 100755
--- a/scripts/annotate-output.sh
+++ b/scripts/annotate-output.sh
@@ -44,20 +44,23 @@ define_handler_with_plain_prefix() {
usage() {
printf \
'Usage: %s [OPTIONS ...] [--] PROGRAM [ARGS ...]
- Executes PROGRAM with ARGS as arguments and prepends printed lines with a
- format string, a stream indicator and `: `.
+Executes PROGRAM with ARGS as arguments and prepends printed lines with a
format
+string, a stream indicator and `: `.
- Options:
- +FORMAT - A format string that may use the conversion
- specifiers from the `date`(1)-utility.
- The printed string is separated from the
following
- stream indicator by a single space.
- Defaults to `%%H:%%M:%%S`.
- --raw-date-format FORMAT - A format string that may use the conversion
- specifiers from the `date`(1)-utility.
- The printed string is not separated from the
- following stream indicator.
- -h, --help - Display this help message.
+Options:
+ +FORMAT
+ A format string that may use the conversion specifiers from the `date`(1)-
+ utility.
+ The printed string is separated from the following stream indicator by a
+ single space.
+ Defaults to `%%H:%%M:%%S`.
+--raw-date-format FORMAT
+ A format string that may use the conversion specifiers from the `date`(1)-
+ utility.
+ The printed string is not separated from the following stream indicator.
+ -h
+--help
+ Display this help message.
' "${0##*/}"
}
--
2.43.0
From 9282a3281cd76f9d1d001393657ddb1c5213b9a8 Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Mon, 11 Dec 2023 01:26:13 +0100
Subject: [PATCH 22/23] annotate-output: improve documentation of bugs/caveats
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Describe more clearly which output (to STDOUT and STDERR) is expected from the
invoked program.
Especially notice that NUL bytes always lead to undefined behaviour as POSIX
shells cannot read them (and it’s undefined what they do, when they encounter
one).
Other bytes may lead to undefined behaviour, depending on the locale that’s
being used. For `dash` this is however not a problem, as that doesn’t support
locales other than the `C`/`POSIX` locale in which all bytes are valid
characters (other than NUL).
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.1 | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
diff --git a/scripts/annotate-output.1 b/scripts/annotate-output.1
index aa0b7b4c..3e14f00c 100644
--- a/scripts/annotate-output.1
+++ b/scripts/annotate-output.1
@@ -65,20 +65,27 @@ $ annotate-output make
21:43:19 I: Finished with exitcode 2
.fi
-.SH BUGS
+.SH CAVEATS AND BUGS
Since STDOUT and STDERR are processed in parallel, it can happen that
some lines received on STDOUT will show up before later-printed STDERR
lines (and vice-versa).
-
+.br
This is unfortunately very hard to fix with the current annotation
strategy. A fix would involve switching to PTRACE'ing the process.
Giving nice a (much) higher priority over \fIprogram\fR could
however cause this behaviour to show up less frequently.
-\fBannotate\-output\fR does not work as well when the output is not
-linewise. In particular, when an interactive \fIprogram\fR asks for input,
-the question might not be shown until after you have answered it. This
-will give the impression that \fIprogram\fR has hung, while it has not.
+\fBannotate\-output\fR expects \fIprogram\fR to output (text) lines (as
+specified by POSIX) to STDOUT and STDERR.
+.br
+In particular, it leads to undefined behaviour when lines are printed that
+contain NUL bytes. It further may lead to undefined behaviour when lines
+are printed that contain bytes that do not form valid characters in the
+current locale.
+
+When an interactive \fIprogram\fR asks for input, the question might not be
+shown until after you have answered it. This will give the impression that
+\fIprogram\fR has hung, while it has not.
.SH "SEE ALSO"
\fBdate\fR(1)
--
2.43.0
>From b09e185cefebc5c4242fccdff98bb678d5f94bed Mon Sep 17 00:00:00 2001
From: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
Date: Mon, 11 Dec 2023 01:50:11 +0100
Subject: [PATCH 23/23] annotate-output: describe caveats on environment
variables
Signed-off-by: Christoph Anton Mitterer <m...@christoph.anton.mitterer.name>
---
scripts/annotate-output.1 | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/scripts/annotate-output.1 b/scripts/annotate-output.1
index 3e14f00c..a64db409 100644
--- a/scripts/annotate-output.1
+++ b/scripts/annotate-output.1
@@ -87,6 +87,23 @@ When an interactive \fIprogram\fR asks for input, the
question might not be
shown until after you have answered it. This will give the impression that
\fIprogram\fR has hung, while it has not.
+\fBannotate\-output\fR is implemented as a script in the Shell Command
+Language. Shells typically set various (shell) variables when started and
+may set the `export` attribute on some of them. They further initialise
+(shell) variables from their own environment (as set by the caller of the
+shell respectively the caller of \fBannotate\-output\fR) and set the
+`export` attribute on them.
+.br
+It follows from this, that when the caller of \fBannotate\-output\fR wants
+to set the environment (variables) of \fIprogram\fR, they may get
+overridden or additional ones may get added by the shell.
+.br
+Further, environment variables are in principle allowed to have names (for
+example `.`) that are not valid shell variable names. POSIX does not
+specify whether or not such environment variables are exported to programs
+invoked from the shell. No assumptions can thus be made on whether such
+environment variables will be exported correctly or at all to \fIprogram\fR.
+
.SH "SEE ALSO"
\fBdate\fR(1)
--
2.43.0
