Re: kill $! won't kill the last command in asynchronous list

[Chet Ramey]

On 4/1/20 12:19 AM, Oğuz wrote:
> $ bash -c '{ sleep 15; } & pstree -p $!; kill $!; echo $?'
> bash(6955)───sleep(6959)
> 0
> $ pkill -e sleep
> sleep killed (pid 6959)
> 
> As seen above, kill $! doesn't kill `sleep` if it's enclosed in curly
> braces. Dropping curly braces, or execing sleep (e.g `{ exec sleep 15; }`)
> fixes this problem, but I didn't understand why.

The braces enclose a group command, and that group command is what's
started in the background. That process is effectively a copy of the shell
that runs the command list contained in the group command (there can be
more than one command, though your example doesn't include that).

Since that shell is what's forked to execute the command list, it's the
process whose pid ends up in $! and the one that receives the SIGTERM.
Unless the shell kills the process it's waiting for when it receives the
SIGTERM, or you send the signal to the entire process group, sleep won't
see it.

There are various race condition here. Shells always ignore SIGTERM. One of
the first things a subshell does after the shell forks it is reset the
SIGTERM handler to SIG_DFL. If the background process doesn't run until the
rest of the parent's command list executes, the parent will send a SIGTERM
to a process that's still ignoring it. It's also possible that the SIGTERM
arrives before the sleep is executed and kills the shell, so the sleep
never runs at all.

Under most circumstances, but not all, I would expect the shell to be
killed and the sleep to survive.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: kill $! won't kill the last command in asynchronous list

[Oğuz]

Thanks for the reply.

1 Nisan 2020 Çarşamba tarihinde Chet Ramey  yazdı:

> On 4/1/20 12:19 AM, Oğuz wrote:
> > $ bash -c '{ sleep 15; } & pstree -p $!; kill $!; echo $?'
> > bash(6955)───sleep(6959)
> > 0
> > $ pkill -e sleep
> > sleep killed (pid 6959)
> >
> > As seen above, kill $! doesn't kill `sleep` if it's enclosed in curly
> > braces. Dropping curly braces, or execing sleep (e.g `{ exec sleep 15;
> }`)
> > fixes this problem, but I didn't understand why.
>
> The braces enclose a group command, and that group command is what's
> started in the background. That process is effectively a copy of the shell
> that runs the command list contained in the group command (there can be
> more than one command, though your example doesn't include that).
>
> Since that shell is what's forked to execute the command list, it's the
> process whose pid ends up in $! and the one that receives the SIGTERM.
> Unless the shell kills the process it's waiting for when it receives the
> SIGTERM, or you send the signal to the entire process group, sleep won't
> see it.
>
>
You are right, I think I was just confused. But this is exactly where I'm
stuck now;
1) I couldn't figure out how to get shell to kill the process it's waiting
for,
2) kill -- -$! says no such process group is found; I couldn't figure out a
reliable way to kill $! and its children. Sending signal to -$$ works most
of the time, but $$ is not guaranteed to be the process group ID if I'm not
mistaken.

Should I ask this on a Q&A forum?

There are various race condition here. Shells always ignore SIGTERM. One of
> the first things a subshell does after the shell forks it is reset the
> SIGTERM handler to SIG_DFL. If the background process doesn't run until the
> rest of the parent's command list executes, the parent will send a SIGTERM
> to a process that's still ignoring it. It's also possible that the SIGTERM
> arrives before the sleep is executed and kills the shell, so the sleep
> never runs at all.
>
> Under most circumstances, but not all, I would expect the shell to be
> killed and the sleep to survive.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>


-- 
Oğuz

Command substitution before while loop

[Krystian Wojtas via Bug reports for the GNU Bourne Again SHell]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-musl
Compiler: gcc
Compilation CFLAGS: -g -O2 -Wno-parentheses -Wno-format-security
uname output: Linux bc55e8b47896 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri 
Feb 28 13:42:26 UTC 2020 x86_64 Linux
Machine Type: x86_64-pc-linux-musl

Bash Version: 5.0
Patch Level: 16
Release Status: release

Description:
Making comments in pipes using command substitution is very helpful. 
But there is syntax error it it is done just before while loop. Could it be 
considered as bug?

Repeat-By:
Script is attached. Ouput
bash-5.0# ./comments.sh
message
message
./comments.sh: line 20: syntax error near unexpected token `do'
./comments.sh: line 20: `while read variable; do'

Best regards,
Krystian Wojtas

Sent with [ProtonMail](https://protonmail.com) Secure Email.

comments.sh
Description: application/shellscript

errexit and local variable set by failing command

[Krystian Wojtas via Bug reports for the GNU Bourne Again SHell]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-musl
Compiler: gcc
Compilation CFLAGS: -g -O2 -Wno-parentheses -Wno-format-security
uname output: Linux bc55e8b47896 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri 
Feb 28 13:42:26 UTC 2020 x86_64 Linux
Machine Type: x86_64-pc-linux-musl

Bash Version: 5.0
Patch Level: 16
Release Status: release

Description:
Script has set errexit flag. It runs function. Function declares local 
variable and set it to output of run subcommand. Subcommand fails. Script does 
not care and continue execution.

When same function sets global variable instead of local one, the 
script is stopped by errexit flag as expected.

Repeat-By:
Scripts are attached. Ouputs
bash-5.0# ./script-global-variable-ok.sh
This message should be printed
bash-5.0# ./script-local-variable-bug.sh
This message should be printed
Script should stop and this message should not be printed

Best regards,
Krystian Wojtas

Sent with [ProtonMail](https://protonmail.com) Secure Email.

script-global-variable-ok.sh
Description: application/shellscript


script-local-variable-bug.sh
Description: application/shellscript

Re: errexit and local variable set by failing command

[Greg Wooledge]

On Wed, Apr 01, 2020 at 04:54:42PM +, Krystian Wojtas via Bug reports for 
the GNU Bourne Again SHell wrote:
> Script has set errexit flag. It runs function. Function declares 
> local variable and set it to output of run subcommand. Subcommand fails. 
> Script does not care and continue execution.

https://mywiki.wooledge.org/BashPitfalls#pf27

Re: errexit and local variable set by failing command

[Krystian Wojtas]

Aha, already known. Thank you for link. I'll get familiar with other cases.

I just figured out that it works as expected if local variable is firstly 
declared and then set.


Pozdrawiam,
Krystian Wojtas

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
środa, kwiecień 1, 2020 7:17 PM, Greg Wooledge  napisał(a):

> On Wed, Apr 01, 2020 at 04:54:42PM +, Krystian Wojtas via Bug reports for 
> the GNU Bourne Again SHell wrote:
>
> > Script has set errexit flag. It runs function. Function declares 
> > local variable and set it to output of run subcommand. Subcommand fails. 
> > Script does not care and continue execution.
> >
>
> https://mywiki.wooledge.org/BashPitfalls#pf27




script-local-variable-bug-workaround.sh
Description: application/shellscript

Re: Command substitution before while loop

[Andreas Schwab]

On Apr 01 2020, Krystian Wojtas via Bug reports for the GNU Bourne Again SHell 
wrote:

> Making comments in pipes using command substitution is very helpful. 
> But there is syntax error it it is done just before while loop. Could it be 
> considered as bug?

Not a bug.  Reserved words are only recognized at the start of a
command, before any expansions.  For the same reason a compound command
cannot be preceded by redirections.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Command substitution before while loop

[Chet Ramey]

On 4/1/20 1:47 PM, Andreas Schwab wrote:
> On Apr 01 2020, Krystian Wojtas via Bug reports for the GNU Bourne Again 
> SHell wrote:
> 
>> Making comments in pipes using command substitution is very helpful. 
>> But there is syntax error it it is done just before while loop. Could it be 
>> considered as bug?
> 
> Not a bug.  Reserved words are only recognized at the start of a
> command, before any expansions.  For the same reason a compound command
> cannot be preceded by redirections.

Yes, and also please note that the historical `` form of command
substitution isn't guaranteed to be able to parse arbitrary commands.
Use $(...) for that.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Command substitution before while loop

[Krystian Wojtas]

I see. Thank you for your answers.


Best regards,
Krystian Wojtas

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
środa, kwiecień 1, 2020 8:20 PM, Chet Ramey  napisał(a):

> On 4/1/20 1:47 PM, Andreas Schwab wrote:
>
> > On Apr 01 2020, Krystian Wojtas via Bug reports for the GNU Bourne Again 
> > SHell wrote:
> >
> > > Making comments in pipes using command substitution is very 
> > > helpful. But there is syntax error it it is done just before while loop. 
> > > Could it be considered as bug?
> > >
> >
> > Not a bug. Reserved words are only recognized at the start of a
> > command, before any expansions. For the same reason a compound command
> > cannot be preceded by redirections.
>
> Yes, and also please note that the historical `` form of command
> substitution isn't guaranteed to be able to parse arbitrary commands.
> Use $(...) for that.
>
> Chet
>
> 
>
> `The lyf so short, the craft so long to lerne.'' - Chaucer`Ars longa, vita 
> brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/

Re: kill $! won't kill the last command in asynchronous list

[Chet Ramey]

On 4/1/20 12:16 PM, Oğuz wrote:
> Thanks for the reply.
> 
> 1 Nisan 2020 Çarşamba tarihinde Chet Ramey  > yazdı:
> 
> On 4/1/20 12:19 AM, Oğuz wrote:
> > $ bash -c '{ sleep 15; } & pstree -p $!; kill $!; echo $?'
> > bash(6955)───sleep(6959)
> > 0
> > $ pkill -e sleep
> > sleep killed (pid 6959)
> >
> > As seen above, kill $! doesn't kill `sleep` if it's enclosed in curly
> > braces. Dropping curly braces, or execing sleep (e.g `{ exec sleep
> 15; }`)
> > fixes this problem, but I didn't understand why.
> 
> The braces enclose a group command, and that group command is what's
> started in the background. That process is effectively a copy of the shell
> that runs the command list contained in the group command (there can be
> more than one command, though your example doesn't include that).
> 
> Since that shell is what's forked to execute the command list, it's the
> process whose pid ends up in $! and the one that receives the SIGTERM.
> Unless the shell kills the process it's waiting for when it receives the
> SIGTERM, or you send the signal to the entire process group, sleep won't
> see it.
> 
> 
> You are right, I think I was just confused. But this is exactly where I'm
> stuck now;
> 1) I couldn't figure out how to get shell to kill the process it's waiting 
> for,

I think the only reliable way to do it is to use the process group.

> 2) kill -- -$! says no such process group is found; 

You have to start the job with job control enabled if you want the
background process to be its own process group leader, and at that point
the shell started to run the group command will not have job control
enabled and will run all processes in the same process group (its own),'
so you could in theory use $!.

If you start it from a shell with job control enabled, the bash running the
-c command will be in its own process group and be the process group
leader, and not have job control enabled.

> I couldn't figure out a
> reliable way to kill $! and its children. Sending signal to -$$ works most
> of the time, but $$ is not guaranteed to be the process group ID if I'm not
> mistaken.

See above about job control shells. The -c command shell is not
interactive, and will not have job control enabled by default, so all
processes in that command will be in the same process group, so if you can
figure out what it is, you should be fine. If you don't start that from a
shell with job control enabled, the process group will be difficult to
find, but $$ is your best bet and should work if you're starting it from
a job control shell.


> Should I ask this on a Q&A forum?

Different perspectives are always good. There are a lot of creative people
out there.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: errexit and local variable set by failing command

[Chet Ramey]

On 4/1/20 1:21 PM, Krystian Wojtas wrote:
> Aha, already known. Thank you for link. I'll get familiar with other cases.
> 
> I just figured out that it works as expected if local variable is firstly 
> declared and then set.

That's because the exit status of a command consisting only of assignment
statements is special-cased to be the exit status of the last command
substitution to occur while processing the assignments.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: kill $! won't kill the last command in asynchronous list

[Oğuz]

> You have to start the job with job control enabled if you want the
> background process to be its own process group leader, and at that point
> the shell started to run the group command will not have job control
> enabled and will run all processes in the same process group (its own),'
> so you could in theory use $!.
>
> If you start it from a shell with job control enabled, the bash running the
> -c command will be in its own process group and be the process group
> leader, and not have job control enabled.



> See above about job control shells. The -c command shell is not
> interactive, and will not have job control enabled by default, so all
> processes in that command will be in the same process group, so if you can
> figure out what it is, you should be fine. If you don't start that from a
> shell with job control enabled, the process group will be difficult to
> find, but $$ is your best bet and should work if you're starting it from
> a job control shell.
>

Yes, -m is exactly what I'm looking for. I should have read more about job
control and processes / process groups before posting the first mail.
Thanks for bearing with my ignorance


-- 
Oğuz

Obscure issue with process substitution since bash-5.0.016 upgrade

[Joan Bruguera Micó]

I'm having a problem with a script that used to work, which I reduced
to the following test case:

#!/bin/bash

function docat() {
for x in 1 2 3 ; do true; done
cat "$@"
}

for x in $(seq 25); do
docat <(echo a) <(echo a) <(echo a) <(echo a) <(echo a) <(echo a)
done

Expected behaviour: A lot of lines with an "a" are printed

Actual behaviour: Some lines with an "a" are printed but a lot of
prints fail with an error:

[...]
a
cat: /dev/fd/63: No such file or directory
cat: /dev/fd/62: No such file or directory
cat: /dev/fd/61: No such file or directory
cat: /dev/fd/60: No such file or directory
a
[...]

Now if you remove the apparently useless for loop in the first line of
docat, this works as I would expect. It also works on zsh.

After some head banging I found that what makes the difference is the
latest bash update, bash-5.0.016. On bash-5.0.015 this script works,
so the the patch that makes the difference is this one:
https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-016

Was the script technically wrong but worked due to some bash
implementation detail, or is this could actually a problem with bash?

I found a very similar report in
https://lists.gnu.org/archive/html/bug-bash/2020-03/msg00062.html ,
but I'm not sure if it's the same underlying issue, since that
particular case is said to be introduced between 4.4-5.0, and this one
is between 5.0.015-5.0.016, and no fix is yet available to test.

Thanks for your help,
- Joan Bruguera

PS: This is the first time I post to the mailing list, so preemptive
apologies for any mistake I could have made.

Re: Obscure issue with process substitution since bash-5.0.016 upgrade

[Joan Bruguera Micó]

Apologies, since after reading the other bug report more thoroughly, I
understood that a fix for that issue has actually already been commited to
the devel branch at this point, and indeed, I can also not reproduce my
issue on the latest snapshot. So it looks like this has already been
addressed.

Therefore, ignore my previous email and consider this solved.

Regards,
- Joan Bruguera



On Thu, 2 Apr 2020 at 04:15, Joan Bruguera Micó 
wrote:

> I'm having a problem with a script that used to work, which I reduced
> to the following test case:
>
> #!/bin/bash
>
> function docat() {
> for x in 1 2 3 ; do true; done
> cat "$@"
> }
>
> for x in $(seq 25); do
> docat <(echo a) <(echo a) <(echo a) <(echo a) <(echo a) <(echo a)
> done
>
> Expected behaviour: A lot of lines with an "a" are printed
>
> Actual behaviour: Some lines with an "a" are printed but a lot of
> prints fail with an error:
>
> [...]
> a
> cat: /dev/fd/63: No such file or directory
> cat: /dev/fd/62: No such file or directory
> cat: /dev/fd/61: No such file or directory
> cat: /dev/fd/60: No such file or directory
> a
> [...]
>
> Now if you remove the apparently useless for loop in the first line of
> docat, this works as I would expect. It also works on zsh.
>
> After some head banging I found that what makes the difference is the
> latest bash update, bash-5.0.016. On bash-5.0.015 this script works,
> so the the patch that makes the difference is this one:
> https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-016
>
> Was the script technically wrong but worked due to some bash
> implementation detail, or is this could actually a problem with bash?
>
> I found a very similar report in
> https://lists.gnu.org/archive/html/bug-bash/2020-03/msg00062.html ,
> but I'm not sure if it's the same underlying issue, since that
> particular case is said to be introduced between 4.4-5.0, and this one
> is between 5.0.015-5.0.016, and no fix is yet available to test.
>
> Thanks for your help,
> - Joan Bruguera
>
> PS: This is the first time I post to the mailing list, so preemptive
> apologies for any mistake I could have made.
>


-- 

*Joan Bruguera Micó* - PGP key id: 88A7 A061 6B47 0CE1 EB4E  D431 8744 44D1
21CE B72E