Re: anonymous pipes in recursive function calls
On Sat, Jun 29, 2024 at 6:39 AM Zachary Santer wrote:
> command this-file |
> while IFS='' read -r -d '' path; do
> cd -- "${starting_PWD}/${path}"
> if [[ -r this-file ]]; then
> recursive_function "${entry_path}/${path}"
> fi
> # fill arrays
> # there is another anonymous pipe here, as well
> done
> #
> if (( PIPESTATUS[0] != 0 )); then
> printf '%s\n' "command failed" >&2
> error='true'
> fi
> cd -- "${starting_PWD}"
> }
You can avoid pipe recursions by storing the output first in an array.
There's also no need to use an error flag variable. Just make sure
return calls are chained.
-
local temp
command this-file | readarray -d '' temp # -d is a Bash 5.0 feature.
if (( PIPESTATUS[0] != 0 )); then
printf '%s\n' "command failed" >&2
return 1
fi
for path in "${temp[@]}"; do
cd -- "${starting_PWD}/${path}" || return
if [[ -r this-file ]]; then
recursive_function "${entry_path}/${path}" || return
fi
...
done
cd -- "${starting_PWD}"
-
You can also just use a 'die' function if cleanups aren't needed.
Anything is valid in a contained code.
--
konsolebox
waiting for process substitutions
>From the manual:
wait [-fn] [-p varname] [id ...]
Wait for each specified child process and return its termination
status. Each id may be a process ID or a job specification; if a job
spec is given, all processes in that job's pipeline are waited for. If
id is not given, wait waits for all running background jobs and the
last-executed process substitution, if its process id is the same as
$!, and the return status is zero. [...]
Why do this? Why not just wait for all process substitutions?
If I do something like
command-1 | tee >( command-2 ) >( command-3 ) >( command-4 )
with 'shopt -s lastpipe', how do I get the pids for the three process
substitutions that all just got forked at the same time? Only one is
going to show up in ${!}. I can't explicitly wait for the pids of all
three of them if I can't get them all in the first place.
Zack
Re: waiting for process substitutions
On Sat, Jun 29, 2024 at 3:47 PM Zachary Santer wrote: > Why not just wait for all process substitutions? What if I substitute thousands of processes and never call wait?
Re: anonymous pipes in recursive function calls
On Sat, Jun 29, 2024 at 4:40 AM konsolebox wrote:
>
> You can avoid pipe recursions by storing the output first in an array.
So is this a known issue?
> There's also no need to use an error flag variable. Just make sure
> return calls are chained.
The intention here is to report as many error conditions as possible
before exiting.
> -
> local temp
> command this-file | readarray -d '' temp # -d is a Bash 5.0 feature.
>
> if (( PIPESTATUS[0] != 0 )); then
> printf '%s\n' "command failed" >&2
> return 1
> fi
>
> for path in "${temp[@]}"; do
> cd -- "${starting_PWD}/${path}" || return
> if [[ -r this-file ]]; then
> recursive_function "${entry_path}/${path}" || return
> fi
> ...
> done
>
> cd -- "${starting_PWD}"
> -
Filling an array of paths and then looping over it in a subsequent for
loop wouldn't actually be unreasonable here, considering how many
paths there are. It just feels like bad practice. There's always the
possibility of busting out the named pipes again. Bash 4.2 couldn't
wait for process substitutions.
set -o nounset -o noglob +o braceexpand
shopt -s lastpipe
main () {
tempdlir="$( mktemp --tmpdir --directory -- script- )" || exit 1
local -i pipeindex=0
# initialize arrays
recursive_function .
# loop through arrays
}
recursive_function () {
local entry_path="${1}"
local named_pipe="${tempdir}/${pipeindex}.pipe"
(( ++pipeindex ))
mkfifo -- "${named_pipe}"
local starting_PWD="${PWD}"
command this-file > "${pipe}" &
local command_pid="${?}"
while IFS='' read -r -d '' path; do
cd -- "${starting_PWD}/${path}" || exit 1
if [[ -r this-file ]]; then
recursive_function "${entry_path}/${path}"
fi
# fill arrays
# there is another anonymous pipe here, as well
done < "${named_pipe}"
if ! wait -- "${command_pid}"; then
printf '%s\n' "command failed" >&2
error='true'
fi
cd -- "${starting_PWD}" || exit 1
}
main "${@}"
Yeah, honestly, filling an array of paths and then looping over it in
a subsequent for loop wouldn't be the end of the world. Either way,
it'll show if the issue is actually the anonymous pipe in a recursive
function call.
Thanks, konsolebox.
Re: waiting for process substitutions
On Sat, Jun 29, 2024 at 9:02 AM Oğuz wrote: > > What if I substitute thousands of processes and never call wait? Couldn't you do the exact same thing with regular background processes forked with &? That's on you.
Re: feature suggestion: ability to expand a set of elements of an array or characters of a scalar, given their indices
On Fri, 28 Jun 2024, 18:31 Oğuz, wrote: > On Friday, June 28, 2024, Martin D Kealey wrote: > >> modern Perl scripts >> > > No such thing. > For the purpose of this argument, "modern" means anything written in the last 25 years, targeting Perl 5 rather than Perl 4. Perl is a dead language, > Whether you think Perl is dead, or indeed whether Perl is actually dead, doesn't affect the validity of my point: it's a historical precedent demonstrating that it's possible and practical to get rid of insane behaviour from a language. and for good reason. > If "good reasons" were actually sufficient to kill off a language, the shell would have died before 2000, and PHP would have been stillborn. Even the things that Perl did wrong could help guide us in better directions. -Martin PS: Some folk think Perl is "hard" and/or "ugly" because it doesn't look like languages they're used to. Guess what: that applies to all languages. Try reading MATLAB or LISP or Prolog or PostScript or YACC. Or Thai or Cherokee or Tok Pisin. Some folk hate Perl's sigles because they hate punctuation generally. (But then I don't know why they would tolerate the shell, much less like it.) Some folk hate that Perl has more than one way to do any given task. Some folk love Perl for exactly that reason. Mostly, younger folk have been told that "Perl is dead and for good reason", so they avoid it without even trying to make their own assessment. Perl has moved on a long way since 1995. By contrast the Shell has stagnated, yet it has moved just enough not to have the benefit of stability. >
Re: Proposal for a New Bash Option: failfast for Immediate Pipeline Failure
I see, thank you then. It was nice to hear from you. Regards Mateusz On Fri, Jun 28, 2024 at 7:26 PM Chet Ramey wrote: > On 6/24/24 10:21 AM, ama bamo wrote: > > > To address these issues, I propose the introduction of a new option, > > failfast, which would immediately terminate the pipeline if any command > in > > the pipeline fails. This would streamline error handling and provide more > > predictable script execution, aligning with user expectations in many > > common use cases. > > I don't see general value in killing pipeline processes (using SIGKILL, > possibly after trying SIGTERM first), as soon as one of them fails. > > Chet > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ > >
Re: waiting for process substitutions
On Saturday, June 29, 2024, Zachary Santer wrote: > > Couldn't you do the exact same thing with regular background processes > forked with &? > But the shell notifies me when they terminate and reap them. -- Oğuz
Re: waiting for process substitutions
On Sat, Jun 29, 2024 at 11:30 AM Oğuz wrote:
>
> On Saturday, June 29, 2024, Zachary Santer wrote:
>>
>> Couldn't you do the exact same thing with regular background processes
>> forked with &?
>
>
> But the shell notifies me when they terminate and reap them.
Is that relevant?
$ { sleep 5; printf 'Words\n'; } & wait -- "${!}"; printf '%s\n' "${?}"
[6] 2401
Words
[6] Done{ sleep 5; printf 'Words\n'; }
0
$ { sleep 5; printf 'Words\n'; } &
[6] 2403
$ Words
[6] Done{ sleep 5; printf 'Words\n'; }
$ wait -- "${!}"; printf '%s\n' "${?}"
0
$ { sleep 5; printf 'Words\n'; } & pid="${!}"
[6] 2405
$ Words
[6] Done{ sleep 5; printf 'Words\n'; }
$ wait -- "${pid}"; printf '%s\n' "${?}"
0
$ wait -- "${pid}"; printf '%s\n' "${?}"
0
$ { sleep 5; printf 'Words\n'; } > >( sleep 10; IFS='' read -r; printf
'|%s|\n' "${REPLY}"; ); pid="${!}"
$ |Words|
$ wait -- "${pid}"; printf '%s\n' "${?}"
0
$ wait -- "${pid}"; printf '%s\n' "${?}"
0
With job control enabled, I can evidently wait on the same pid over
and over again with no ill effect, even if I've already been notified
that said job has terminated.
With a command like this
diff --unified -- <( command-1 ) <( command-2 )
it's safe to assume that the commands in both command substitutions
have terminated when the call to diff has terminated. In that case, if
the shell notifies you that both command substitutions have
terminated, that's redundant and annoying.
On the other hand, I'm pretty sure
command-1 | tee >( command-2 ) >( command-3 ) >( command-4 )
will terminate as soon as command-1 and tee have terminated, but the
command substitutions could still be running. If you want to run
commands like this on the command line, it still might be useful to
know when the command substitutions have terminated.
I just tend to write a script by the time things get this complicated.
waiting for process substitutions
On Sat, Jun 29, 2024 at 8:08 PM Zachary Santer wrote: > Is that relevant? I think so. There is a limit to the number of jobs Bash can remember, once it's exceeded the oldest job is overwritten. Do we really want process substitutions to count against that limit? Or did you mean something else? -- Oğuz
Re: anonymous pipes in recursive function calls
On Sat, Jun 29, 2024 at 10:23 PM Zachary Santer wrote: > > On Sat, Jun 29, 2024 at 4:40 AM konsolebox wrote: > > > > You can avoid pipe recursions by storing the output first in an array. > > So is this a known issue? I don't know. I haven't really checked your issues. I just think avoiding too many opened pipes at the same time should be intuitive and it's likely going to fix whatever issue there is. > > There's also no need to use an error flag variable. Just make sure > > return calls are chained. > > The intention here is to report as many error conditions as possible > before exiting. You can print an error message before calling return. Are you planning to run more commands even if an error has already happened? > Filling an array of paths and then looping over it in a subsequent for > loop wouldn't actually be unreasonable here, considering how many > paths there are. It just feels like bad practice. Why do you think it's a bad practice? > There's always the > possibility of busting out the named pipes again. Bash 4.2 couldn't > wait for process substitutions. The solution I suggested doesn't use process substitution so I'm not sure what you mean. -- konsolebox
Re: anonymous pipes in recursive function calls
On Sat, Jun 29, 2024 at 2:29 PM konsolebox wrote: > > On Sat, Jun 29, 2024 at 10:23 PM Zachary Santer wrote: > > > > The intention here is to report as many error conditions as possible > > before exiting. > > You can print an error message before calling return. Are you > planning to run more commands even if an error has already happened? "Error conditions" in the sense of things that would cause problems later - minimizing the likelihood that something will only come up after the script has already made some updates. If the script reports multiple issues before any updates are made, the user can go and resolve all that stuff before trying to run the script again. If the script were to leave the user in a partially-updated state, the user's left having to do the rest of the work manually. > > Filling an array of paths and then looping over it in a subsequent for > > loop wouldn't actually be unreasonable here, considering how many > > paths there are. It just feels like bad practice. > > Why do you think it's a bad practice? Some of the power of bash is the ease with which it enables parallelism. Collecting all the output from a command or block of code before it gets passed to the next is throwing away that benefit. Additionally, you might find yourself with so much output that it's too much to expand all at once in the for loop. Don't know what that limit is, but you're at the very least using more memory than you would be otherwise. > > There's always the > > possibility of busting out the named pipes again. Bash 4.2 couldn't > > wait for process substitutions. > > The solution I suggested doesn't use process substitution so I'm not > sure what you mean. In my earlier email, I mentioned potentially trying while [...] done < <( command this-file ) to see if it resolved the issue. If I do so, however, I have no way to determine the exit status of 'command' in the parent shell, at least in bash 4.2. I have somewhat of a repeat-by now, but I want to see if it causes the same hiccup in bash 4.2 as the original script did, because it doesn't do it in bash 5.2. We shall see.
Re: waiting for process substitutions
On Sat, Jun 29, 2024 at 2:07 PM Oğuz wrote:
>
> There is a limit to the number of jobs Bash can remember, once it's exceeded
> the oldest job is overwritten. Do we really want process substitutions to
> count against that limit?
They might already. Now I'm wondering if the documentation just needed updating.
I'm afraid to report this as a bug, because it feels like something
that running bash in MSYS2 on Windows could be responsible for, but
here goes.
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: msys
Compiler: gcc
Compilation CFLAGS: -march=nocona -msahf -mtune=generic -O2 -pipe
-D_STATIC_BUILD
uname output: MINGW64_NT-10.0-19045 Zack2021HPPavilion 3.5.3.x86_64
2024-06-03 06:22 UTC x86_64 Msys
Machine Type: x86_64-pc-msys
Bash Version: 5.2
Patch Level: 26
Release Status: release
Description:
So bash can wait on process substitutions.
1) When all child processes are process substitutions:
a. wait without arguments actually appears to wait for all of them,
not just the last-executed one, contradicting the man page.
b. A subsequent call to wait listing all child process pids
immediately terminates successfully.
c. If calling wait -n in the middle of all this, whether listing only
un-waited-on child process pids or all child process pids, it lists
all argument pids as "no such job" and terminates with code 127. This
is probably incorrect behavior.
2) When a standard background process is added:
a. wait without arguments waits for all child processes.
b. A subsequent call to wait listing all child process pids lists all
argument pids as not children of the shell and terminates with code
127. This seems incorrect, or at least the change in behavior from 1b.
is unexpected.
c. If calling wait -n in the middle of all this, we see that it only
lists the pids from process substitutions as "no such job".
Repeat-By:
./procsub-wait false false
./procsub-wait false true
./procsub-wait true false
./procsub-wait true true
procsub-wait and the results of running it on the listed system are attached.
zsant@Zack2021HPPavilion MINGW64 ~/random
$ ./procsub-wait false false
+ add_background_process=false
+ try_wait_n=false
+ pid=()
+ declare -a pid
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 4
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 6
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 8
+ : /dev/fd/63
+ pid+=(${!})
+ [[ false == \t\r\u\e ]]
++ sleep 10
+ : /dev/fd/63
+ pid+=(${!})
+ SECONDS=0
+ declare -p pid
declare -a pid=([0]="4004" [1]="4005" [2]="4006" [3]="4007" [4]="4008")
+ wait -- 4004
++ sleep 2
+ : 'termination status 0 at 4 seconds'
+ [[ false == \t\r\u\e ]]
+ wait
+ : 'termination status 0 at 10 seconds'
+ wait -- 4004 4005 4006 4007 4008
+ : 'termination status 0 at 10 seconds'
zsant@Zack2021HPPavilion MINGW64 ~/random
$ ./procsub-wait false true
+ add_background_process=false
+ try_wait_n=true
+ pid=()
+ declare -a pid
+ : /dev/fd/63
++ sleep 4
+ pid+=(${!})
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 6
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 8
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 10
+ [[ false == \t\r\u\e ]]
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 2
+ SECONDS=0
+ declare -p pid
declare -a pid=([0]="4010" [1]="4011" [2]="4012" [3]="4013" [4]="4014")
+ wait -- 4010
+ : 'termination status 0 at 4 seconds'
+ [[ true == \t\r\u\e ]]
+ wait -n -p completed_pid -- 4010 4011 4012 4013 4014
./procsub-wait: line 22: wait: 4010: no such job
./procsub-wait: line 22: wait: 4011: no such job
./procsub-wait: line 22: wait: 4012: no such job
./procsub-wait: line 22: wait: 4013: no such job
./procsub-wait: line 22: wait: 4014: no such job
+ : 'termination status 127 at 4 seconds'
+ declare -p completed_pid
./procsub-wait: line 24: declare: completed_pid: not found
+ wait
+ : 'termination status 0 at 10 seconds'
+ wait -- 4010 4011 4012 4013 4014
+ : 'termination status 0 at 10 seconds'
zsant@Zack2021HPPavilion MINGW64 ~/random
$ ./procsub-wait true false
+ add_background_process=true
+ try_wait_n=false
+ pid=()
+ declare -a pid
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 4
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 6
+ : /dev/fd/63
+ pid+=(${!})
++ sleep 8
+ : /dev/fd/63
+ pid+=(${!})
+ [[ true == \t\r\u\e ]]
++ sleep 10
+ pid+=(${!})
+ sleep 1
+ : /dev/fd/63
+ pid+=(${!})
+ SECONDS=0
+ declare -p pid
declare -a pid=([0]="4016" [1]="4017" [2]="4018" [3]="4019" [4]="4020"
[5]="4021")
+ wait -- 4016
++ sleep 2
+ : 'termination status 0 at 4 seconds'
+ [[ false == \t\r\u\e ]]
+ wait
+ : 'termination status 0 at 10 seconds'
+ wait -- 4016 4017 4018 4019 4020 4021
./procsub-wait: line 28: wait: pid 4016 is not a child of this shell
./procsub-wait: line 28: wait: pid 4017 is not a child of this shell
./procsub-wait: line 28: wait: pid 4018 is not a child of this shell
./procsub-wait: line 28: wait: pid 4019 is not a child of this shell
./procsub-wait: line 28: wait: pid 4020 is not a child of this shell
./procsub-wait: line 28: wait: pid 4021 is not a child of this shell
+ : 'termination status 127 at 10 seconds'
zs
