'foo > >(bar)' doesn't set $! for external foo not invoked via 'command'

[Rusty Bird]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -O2 -g -pipe -Wall -Werror=format-security 
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions 
-fstack-protector-strong -grecord-gcc-switches 
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic 
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection 
-Wno-parentheses -Wno-format-security
uname output: Linux [...] 4.19.132-1.pvops.qubes.x86_64 #1 SMP Tue Jul 14 
03:42:21 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu

Bash Version: 5.0
Patch Level: 17
Release Status: release

Description:
$! isn't set to the PID of 'bar' for

foo > >(bar)

if 'foo' is an external program (as opposed to a builtin or
function) - unless it's invoked via 'command'.

Repeat-By:
type date  # "date is /usr/bin/date"
set -u
echo $!  # "bash: $!: unbound variable"

date > >(sleep 20)
echo $!  # "bash: $!: unbound variable"

command date > >(sleep 21)
echo $!  # "123"

date_wrapper() { date; }
date_wrapper > >(sleep 22)
echo $!  # "234"


For context - I'm filtering a program's stdout and stderr
(separately), requiring successful exit statuses for the program and
both filters:

set -u -o pipefail
{ program 2> >(stderr_filter >&2) && wait $!; } | stdout_filter && ...

This only works if 'program' is changed to 'command program'.

Rusty


signature.asc
Description: PGP signature

Re: 'foo > >(bar)' doesn't set $! for external foo not invoked via 'command'

[Rusty Bird]

Oğuz:
> > For context - I'm filtering a program's stdout and stderr
> > (separately), requiring successful exit statuses for the program and
> > both filters:
> >
> > set -u -o pipefail
> > { program 2> >(stderr_filter >&2) && wait $!; } | stdout_filter &&
> > ...

> Not sure if process substitution is really necessary here,
> 
> set -u -o pipefail
> { program 2>&3 | stdout_filter;  } 3>&1 | stderr_filter && ...
> 
> does the same thing.

That one filters program's stdout twice - first through stdout_filter
and then through stderr_filter - with program's stdout and stderr both
finally arriving at stdout. But tweaked like this, it seems to cover
all the bases:

set -u -o pipefail
{
program 2>&1  >&"$out" {out}>&- |
stderr_filter >&2  {out}>&-
} {out}>&1 | stdout_filter

And it even waits for stderr_filter if program failed. My original
snippet neglected that case, otherwise it would have looked more like

   set -u -o pipefail
   (
   trap 'wait $! || exit $?' EXIT
   command program 2> >(stderr_filter >&2)
   ) | stdout_filter

which isn't exactly pretty either, even if the bug(?) requiring
'command' is fixed.

Rusty


signature.asc
Description: PGP signature

Re: 'foo > >(bar)' doesn't set $! for external foo not invoked via 'command'

[Rusty Bird]

Oğuz:
> `>&"$out" is very ugly though, it'd be nice if `{var}' thing worked at
> the RHS of redirection operator, like `>&{var}`, which, on bash 5.0.11,
> ignores `&' -another bug?- and redirects stdout to a file named `{var}'.

Probably more backwards compatibility than a bug - '>&word' (with word
neither a number nor a dash) is an alternative form of '&>word':

https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Redirecting-Standard-Output-and-Standard-Error

Rusty


signature.asc
Description: PGP signature

Re: 'foo > >(bar)' doesn't set $! for external foo not invoked via 'command'

[Rusty Bird]

Chet Ramey:
> >> Description:
> >> $! isn't set to the PID of 'bar' for
> >>
> >> foo > >(bar)
> >>
> >> if 'foo' is an external program (as opposed to a builtin or
> >> function) - unless it's invoked via 'command'.
> 
> Redirections are performed in the subshell created to execute `foo', so the
> process substitution is invoked there and can't affect the parent's $!.

I guess my question is: Would it not make sense to always perform the
redirection before the subshell is created?

OTOH, looks like it's trivial to work around the issue by using
'{ foo; } > >(bar)'.

Rusty


signature.asc
Description: PGP signature

Re: use of set -e inside parenthesis and conditionnal second command

[Rusty Bird]

Greg Wooledge:
> The behavior of set -e is extremely surprising, and people should
> stop expecting it to make intuitive sense.  The best way to avoid being
> surprised by it is to stop using it altogether.

Has it ever been considered to add something like 'shopt -s
composable_compound'? Roughly meaning "execution and exit status of a
compound command are not changed by surrounding context (in the sense
of logical operators, or being the condition of an if statement etc.)"

That, together with 'set -e -o pipefail; shopt -s inherit_errexit'
seems like it would go a long way towards easier error handling.

Rusty


signature.asc
Description: PGP signature