Unexpected behaviour when using process substitution with stdout and stderr
GNU bash, version 5.1.8(1)-release (x86_64-pc-linux-gnu)
I have attempted to use process substitution in order to feed the
output of a command into two filters, one for handling stdout and the
other for stderr.
Prior to this I was using POSIX sh and named pipes to achieve this but
decided to try bash with its >() and <() notation.
Perhaps I am misunderstanding how this works because I found them to be
unusable in this context.
What appears to be happening is that the output from standard error is
being mixed into the function handling standard out, even more
surprisingly that xtrace output is also being consumed and filtered as
well.
I don't quite know how to describe what's happening so I have provided a
small demonstration/reproducer instead:
#!/bin/bash --
generate() {
for ((i=0; i<10; ++i)); do
if ((RANDOM % 2)); then
printf 'generate to stdout\n'
else
printf 'generate to stderr\n' >&2
fi
done
}
stdout() {
local line
while read -r line; do
printf 'from stdout: %s\n' "$line"
done
}
stderr() {
local line
while read -r line; do
printf 'from stderr: %s\n' "$line"
done
}
# Using process substitution.
unexpected() {
# This is particularly dangerous when the script is executed under `bash -x'
# as the xtrace output is read as standard input to the `stdout' function.
generate > >(stdout) 2> >(stderr)
wait
# Example output:
# from stdout: generate to stdout
# from stdout: generate to stdout
# from stdout: from stderr: generate to stderr
# from stdout: from stderr: generate to stderr
# from stdout: from stderr: generate to stderr
# from stdout: generate to stdout
# from stdout: from stderr: generate to stderr
# from stdout: generate to stdout
# from stdout: from stderr: generate to stderr
# from stdout: from stderr: generate to stderr
}
# Using named pipes.
expected() {
mkfifo a b
trap 'rm a b' EXIT
generate > a 2> b &
stdout < a &
stderr < b &
wait
# Example output:
# from stdout: generate to stdout
# from stderr: generate to stderr
# from stdout: generate to stdout
# from stderr: generate to stderr
# from stdout: generate to stdout
# from stdout: generate to stdout
# from stderr: generate to stderr
# from stdout: generate to stdout
# from stdout: generate to stdout
# from stderr: generate to stderr
}
Re: Unexpected behaviour when using process substitution with stdout and stderr
you printf basically to stdout from stderr, ..maybe its that
if you'd change your stderr printf to >&2 maybe works, ..just saying, maybe
or change the order to 2> and then >
..maybe..
peace
On Sun, Jul 11, 2021, 12:09 earnestly wrote:
> GNU bash, version 5.1.8(1)-release (x86_64-pc-linux-gnu)
>
> I have attempted to use process substitution in order to feed the
> output of a command into two filters, one for handling stdout and the
> other for stderr.
>
> Prior to this I was using POSIX sh and named pipes to achieve this but
> decided to try bash with its >() and <() notation.
>
> Perhaps I am misunderstanding how this works because I found them to be
> unusable in this context.
>
> What appears to be happening is that the output from standard error is
> being mixed into the function handling standard out, even more
> surprisingly that xtrace output is also being consumed and filtered as
> well.
>
> I don't quite know how to describe what's happening so I have provided a
> small demonstration/reproducer instead:
>
> #!/bin/bash --
>
> generate() {
> for ((i=0; i<10; ++i)); do
> if ((RANDOM % 2)); then
> printf 'generate to stdout\n'
> else
> printf 'generate to stderr\n' >&2
> fi
> done
> }
>
> stdout() {
> local line
>
> while read -r line; do
> printf 'from stdout: %s\n' "$line"
> done
> }
>
> stderr() {
> local line
>
> while read -r line; do
> printf 'from stderr: %s\n' "$line"
> done
> }
>
> # Using process substitution.
> unexpected() {
> # This is particularly dangerous when the script is executed under
> `bash -x'
> # as the xtrace output is read as standard input to the `stdout'
> function.
> generate > >(stdout) 2> >(stderr)
> wait
>
> # Example output:
> # from stdout: generate to stdout
> # from stdout: generate to stdout
> # from stdout: from stderr: generate to stderr
> # from stdout: from stderr: generate to stderr
> # from stdout: from stderr: generate to stderr
> # from stdout: generate to stdout
> # from stdout: from stderr: generate to stderr
> # from stdout: generate to stdout
> # from stdout: from stderr: generate to stderr
> # from stdout: from stderr: generate to stderr
> }
>
> # Using named pipes.
> expected() {
> mkfifo a b
> trap 'rm a b' EXIT
>
> generate > a 2> b &
>
> stdout < a &
> stderr < b &
> wait
>
> # Example output:
> # from stdout: generate to stdout
> # from stderr: generate to stderr
> # from stdout: generate to stdout
> # from stderr: generate to stderr
> # from stdout: generate to stdout
> # from stdout: generate to stdout
> # from stderr: generate to stderr
> # from stdout: generate to stdout
> # from stdout: generate to stdout
> # from stderr: generate to stderr
> }
>
>
Re: Unexpected behaviour when using process substitution with stdout and stderr
On Sun, Jul 11, 2021 at 11:09:10AM +0100, earnestly wrote:
> What appears to be happening is that the output from standard error is
> being mixed into the function handling standard out, even more
> surprisingly that xtrace output is also being consumed and filtered as
> well.
First, xtrace (set -x) output is written to stderr by default. You can
change this by setting the BASH_XTRACEFD variable. But in the absence
of that change, it goes to stderr, along with any other content that's
written to stderr, either by bash itself, or by commands that bash calls.
> stdout() {
> local line
>
> while read -r line; do
> printf 'from stdout: %s\n' "$line"
> done
> }
>
> stderr() {
> local line
>
> while read -r line; do
> printf 'from stderr: %s\n' "$line"
> done
> }
> generate > >(stdout) 2> >(stderr)
> wait
You're sending the stderr of generate to a subshell where the stderr()
function is executed. But the stderr() function writes to stdout.
Therefore, all of the output from generate is ultimately ending up going
to the script's stdout, with potentially weird interleaving depending
on the timing of the two background subshells.
If you want to keep them separate, you either need to change the stderr()
function to write to stderr, or change the >(stderr) call to >(stderr >&2).
Re: Unexpected behaviour when using process substitution with stdout and stderr
On Jul 11 2021, Greg Wooledge wrote: > You're sending the stderr of generate to a subshell where the stderr() > function is executed. But the stderr() function writes to stdout. > Therefore, all of the output from generate is ultimately ending up going > to the script's stdout, with potentially weird interleaving depending > on the timing of the two background subshells. There is no interleaving. The output of stderr is connected to the current disposition of fd 1 which is redirected to the >(stdout) pipe. You can either switch the order of redirections, or save the previous disposition of fd 1 in another fd and redirect the output of stderr there. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Re: Bash-5.1-rc3 available
On 7/10/21 6:03 PM, Alex fxmbsw7 Ratchev wrote: is this --branch bash-5.1-testing stilk actual, or should i use devel No. It's for pre-release versions of bash-5.1. If you want to keep up with development, use the devel branch. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Bash-5.1-rc3 available
thank you On Sun, Jul 11, 2021, 20:51 Chet Ramey wrote: > On 7/10/21 6:03 PM, Alex fxmbsw7 Ratchev wrote: > > is this --branch bash-5.1-testing stilk actual, or should i use devel > > No. It's for pre-release versions of bash-5.1. If you want to keep up with > development, use the devel branch. > > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ >
