Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Chet Ramey]

On 4/30/24 3:59 AM, Koichi Murase wrote:

2024年4月30日(火) 5:07 Chet Ramey :

OK, let's explore this again.


I haven't yet been convinced by the previous discussion [1,2] about
the reporting of the foreground dead jobs in the trap handler, but
this time, the situation is slightly different. The `jobs' builtin
reports the foreground dead jobs even in a normal context of the
commands executed through the command line when the new function
substitutions are used.


Yes, and I said I could work around this case.



[1] https://lists.gnu.org/archive/html/bug-bash/2022-10/threads.html#00054
[2] https://lists.gnu.org/archive/html/bug-bash/2022-11/threads.html#00049
(continued. I haven't received a reply)


To what?


POSIX says `jobs' should report on

"the status [...] not been reported by the shell"

So the question is what exactly constitutes a foreground job whose status
has "not been reported."


I'm not sure about the answer, but kre's opinion seems reasonable to
me. 


OK. The standard does say "reporting the exit status with the special
parameter '?'" in

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_02



POSIX Draft (202x/D4.1) XCU 2.11
mentions the reporting of the ``background'' jobs ``immediately prior
to writing the next prompt'', but the reporting of the foreground jobs
does not seem to be described.


Correct. Is setting $? sufficient, or does the standard mean the reporting
on foreground jobs that are killed by a signal -- one that shells don't
ignore -- before the next command, which is what all shells do?

For instance, if you run

sleep 20; jobs

and hit the `sleep' with SIGUSR1 from another process, all shells will
print some message about the process exiting due to SIGUSR1 before `jobs'
is executed, and `jobs' prints nothing, because the reporting has already
happened.

So it's not exactly the jobs builtin, nor precisely jobs list management.
It's more about which terminated foreground processes get a message
displayed -- all of them set $? -- and which ones are silent, and whether
that counts as `reporting'.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Oğuz]

On Tuesday, April 30, 2024, Chet Ramey  wrote:
>
> comsubs). I can work around this case, but I'm still interested in what
> people think the general rule should be.
>

I don't think anyone would expect to run `jobs' and see anything but
asynchronous and stopped jobs.


-- 
Oğuz

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Chet Ramey]

On 5/1/24 2:39 PM, Oğuz wrote:

On Tuesday, April 30, 2024, Chet Ramey  wrote:


comsubs). I can work around this case, but I'm still interested in what
people think the general rule should be.



I don't think anyone would expect to run `jobs' and see anything but
asynchronous and stopped jobs.


Sure. Then you wonder why POSIX bothered to include "and all jobs whose
status has changed and have not been reported by the shell" in the
standard.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Oğuz]

On Wed, May 1, 2024 at 10:36 PM Chet Ramey  wrote:
> Sure. Then you wonder why POSIX bothered to include "and all jobs whose
> status has changed and have not been reported by the shell" in the
> standard.

No, I think it's a botched paraphrase. Issue 7 says "By default, the
jobs utility shall display the status of all stopped jobs, running
background jobs and all jobs whose status has changed and have not
been reported by the shell." and the description for set -m agrees;
"all jobs whose status has changed and have not been reported by the
shell" are background jobs terminated since the last prompt.

[bug #65670] Bash is not memory safe.

[anonymous]

URL:
  

 Summary: Bash is not memory safe.
   Group: The GNU Bourne-Again SHell
   Submitter: None
   Submitted: Wed 01 May 2024 11:39:28 PM UTC
Category: None
Severity: 3 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any


___

Follow-up Comments:


---
Date: Wed 01 May 2024 11:39:28 PM UTC By: Anonymous
$ bash --version | head -n1
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)

$ valgrind --leak-check=full \
 --track-origins=yes \
 --verbose \
 --log-file=valgrind-out-bash.txt \
 /bin/bash



==2762== Memcheck, a memory error detector
==2762== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==2762== Using Valgrind-3.19.0-8d3c8034b8-20220411 and LibVEX; rerun with -h
for copyright info
==2762== Command: /bin/bash
==2762== Parent PID: 2251
==2762== 
--2762-- 
--2762-- Valgrind options:
--2762----leak-check=full
--2762----track-origins=yes
--2762----verbose
--2762----log-file=valgrind-out-bash.txt
--2762-- Contents of /proc/version:
--2762--   Linux version 6.1.0-20-amd64 (debian-ker...@lists.debian.org)
(gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1
SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11)
--2762-- 
--2762-- Arch and hwcaps: AMD64, LittleEndian,
amd64-cx16-lzcnt-rdtscp-sse3-ssse3-avx-avx2-bmi-f16c-rdrand-rdseed
--2762-- Page sizes: currently 4096, max supported 4096
--2762-- Valgrind library directory: /usr/libexec/valgrind
--2762-- Reading syms from /bin/bash
--2762--object doesn't have a symbol table
--2762-- Reading syms from /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
--2762--   Considering
/usr/lib/debug/.build-id/80/8ccde0c755608358b0915a6615c19401fe6bf6.debug ..
--2762--   .. build-id is valid
--2762-- Reading syms from /usr/libexec/valgrind/memcheck-amd64-linux
--2762--   Considering
/usr/lib/debug/.build-id/82/26c2aa6b808ebd5a6fafb694a7fb3287f33590.debug ..
--2762--   .. build-id is valid
--2762--object doesn't have a dynamic symbol table
--2762-- Scheduler: using generic scheduler lock implementation.
--2762-- Reading suppressions file: /usr/libexec/valgrind/default.supp
==2762== embedded gdbserver: reading from
/tmp/vgdb-pipe-from-vgdb-to-2762-by-user-on-???
==2762== embedded gdbserver: writing to  
/tmp/vgdb-pipe-to-vgdb-from-2762-by-user-on-???
==2762== embedded gdbserver: shared mem  
/tmp/vgdb-pipe-shared-mem-vgdb-2762-by-user-on-???
==2762== 
==2762== TO CONTROL THIS PROCESS USING vgdb (which you probably
==2762== don't want to do, unless you know exactly what you're doing,
==2762== or are doing some strange experiment):
==2762==   /usr/bin/vgdb --pid=2762 ...command...
==2762== 
==2762== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==2762==   /path/to/gdb /bin/bash
==2762== and then give GDB the following command
==2762==   target remote | /usr/bin/vgdb --pid=2762
==2762== --pid is optional if only one valgrind process is running
==2762== 
--2762-- REDIR: 0x40238e0 (ld-linux-x86-64.so.2:strlen) redirected to
0x580bb0e2 (vgPlain_amd64_linux_REDIR_FOR_strlen)
--2762-- REDIR: 0x40220c0 (ld-linux-x86-64.so.2:index) redirected to
0x580bb0fc (vgPlain_amd64_linux_REDIR_FOR_index)
--2762-- Reading syms from
/usr/libexec/valgrind/vgpreload_core-amd64-linux.so
--2762--   Considering
/usr/lib/debug/.build-id/ad/f1388be4d8781737b0c83fe111a5a9c6e930aa.debug ..
--2762--   .. build-id is valid
--2762-- Reading syms from
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so
--2762--   Considering
/usr/lib/debug/.build-id/d8/ec66cffcb23a75c3f15940674d6028709121f8.debug ..
--2762--   .. build-id is valid
==2762== WARNING: new redirection conflicts with existing -- ignoring it
--2762-- old: 0x040238e0 (strlen  ) R-> (.0) 0x580bb0e2
vgPlain_amd64_linux_REDIR_FOR_strlen
--2762-- new: 0x040238e0 (strlen  ) R-> (2007.0) 0x048468a0
strlen
--2762-- REDIR: 0x40222e0 (ld-linux-x86-64.so.2:strcmp) redirected to
0x4847780 (strcmp)
--2762-- REDIR: 0x4021550 (ld-linux-x86-64.so.2:mempcpy) redirected to
0x484b1a0 (mempcpy)
--2762-- Reading syms from /usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0
--2762--object doesn't have a symbol table
--2762-- Reading syms from /lib/x86_64-linux-gnu/libtinfo.so.6.4
--2762--object doesn't have a symbol table
--2762-- Reading syms from /lib/x86_64-linux-gnu/libc.so.6
--2762--   Considering
/usr/lib/debug/.build-id/ee/3145ecaaff87a133daea77fbc3eecd458fa0d1.debug ..
--2762--   .. build-id is valid
==2762== WARNING: new redirection conflicts with existing -- ignoring it
--2762-- old: 0x0493b540 (memalign) R-> (1011.0) 0x04845bc0
me

(Just making sure this went through.) Bug: Bash is not memory safe.

[Joshua Brown]

$ bash --version | head -n1
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)

$ valgrind --leak-check=full \
 --track-origins=yes \
 --verbose \
 --log-file=valgrind-out-bash.txt \
 /bin/bash



==2762== Memcheck, a memory error detector
==2762== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==2762== Using Valgrind-3.19.0-8d3c8034b8-20220411 and LibVEX; rerun
with -h for copyright info ==2762== Command: /bin/bash
==2762== Parent PID: 2251
==2762== 
--2762-- 
--2762-- Valgrind options:
--2762----leak-check=full
--2762----track-origins=yes
--2762----verbose
--2762----log-file=valgrind-out-bash.txt
--2762-- Contents of /proc/version:
--2762--   Linux version 6.1.0-20-amd64
(debian-ker...@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU
ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian
6.1.85-1 (2024-04-11) --2762-- --2762-- Arch and hwcaps: AMD64,
LittleEndian,
amd64-cx16-lzcnt-rdtscp-sse3-ssse3-avx-avx2-bmi-f16c-rdrand-rdseed
--2762-- Page sizes: currently 4096, max supported 4096 --2762--
Valgrind library directory: /usr/libexec/valgrind --2762-- Reading syms
from /bin/bash --2762--object doesn't have a symbol table --2762--
Reading syms from /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --2762--
Considering
/usr/lib/debug/.build-id/80/8ccde0c755608358b0915a6615c19401fe6bf6.debug
.. --2762--   .. build-id is valid --2762-- Reading syms from
/usr/libexec/valgrind/memcheck-amd64-linux --2762--   Considering
/usr/lib/debug/.build-id/82/26c2aa6b808ebd5a6fafb694a7fb3287f33590.debug
.. --2762--   .. build-id is valid --2762--object doesn't have a
dynamic symbol table --2762-- Scheduler: using generic scheduler lock
implementation. --2762-- Reading suppressions file:
/usr/libexec/valgrind/default.supp ==2762== embedded gdbserver: reading
from /tmp/vgdb-pipe-from-vgdb-to-2762-by-user-on-??? ==2762== embedded
gdbserver: writing to   /tmp/vgdb-pipe-to-vgdb-from-2762-by-user-on-???
==2762== embedded gdbserver: shared mem
/tmp/vgdb-pipe-shared-mem-vgdb-2762-by-user-on-??? ==2762== ==2762== TO
CONTROL THIS PROCESS USING vgdb (which you probably ==2762== don't want
to do, unless you know exactly what you're doing, ==2762== or are doing
some strange experiment): ==2762==   /usr/bin/vgdb --pid=2762
...command... ==2762== ==2762== TO DEBUG THIS PROCESS USING GDB: start
GDB like this ==2762==   /path/to/gdb /bin/bash ==2762== and then give
GDB the following command ==2762==   target remote | /usr/bin/vgdb
--pid=2762 ==2762== --pid is optional if only one valgrind process is
running ==2762== 
--2762-- REDIR: 0x40238e0 (ld-linux-x86-64.so.2:strlen) redirected to
0x580bb0e2 (vgPlain_amd64_linux_REDIR_FOR_strlen) --2762-- REDIR:
0x40220c0 (ld-linux-x86-64.so.2:index) redirected to 0x580bb0fc
(vgPlain_amd64_linux_REDIR_FOR_index) --2762-- Reading syms from
/usr/libexec/valgrind/vgpreload_core-amd64-linux.so --2762--
Considering
/usr/lib/debug/.build-id/ad/f1388be4d8781737b0c83fe111a5a9c6e930aa.debug
.. --2762--   .. build-id is valid --2762-- Reading syms from
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so --2762--
Considering
/usr/lib/debug/.build-id/d8/ec66cffcb23a75c3f15940674d6028709121f8.debug
.. --2762--   .. build-id is valid ==2762== WARNING: new redirection
conflicts with existing -- ignoring it --2762-- old: 0x040238e0
(strlen  ) R-> (.0) 0x580bb0e2
vgPlain_amd64_linux_REDIR_FOR_strlen --2762-- new: 0x040238e0
(strlen  ) R-> (2007.0) 0x048468a0 strlen --2762-- REDIR:
0x40222e0 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4847780
(strcmp) --2762-- REDIR: 0x4021550 (ld-linux-x86-64.so.2:mempcpy)
redirected to 0x484b1a0 (mempcpy) --2762-- Reading syms from
/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0 --2762--object doesn't
have a symbol table --2762-- Reading syms from
/lib/x86_64-linux-gnu/libtinfo.so.6.4 --2762--object doesn't have a
symbol table --2762-- Reading syms from /lib/x86_64-linux-gnu/libc.so.6
--2762--   Considering
/usr/lib/debug/.build-id/ee/3145ecaaff87a133daea77fbc3eecd458fa0d1.debug
.. --2762--   .. build-id is valid ==2762== WARNING: new redirection
conflicts with existing -- ignoring it --2762-- old: 0x0493b540
(memalign) R-> (1011.0) 0x04845bc0 memalign --2762--
new: 0x0493b540 (memalign) R-> (1017.0) 0x04845b90
aligned_alloc ==2762== WARNING: new redirection conflicts with existing
-- ignoring it --2762-- old: 0x0493b540 (memalign) R->
(1011.0) 0x04845bc0 memalign --2762-- new: 0x0493b540 (memalign
   ) R-> (1017.0) 0x04845b60 aligned_alloc --2762-- Reading syms
from /lib/x86_64-linux-gnu/libdl.so.2 --2762--   Considering
/usr/lib/debug/.build-id/53/eaa845e9ca621f159b0622daae7387cdea1e97.debug
.. --2762--   .. build-id is valid --2762-- Reading syms from
/lib/x86_64-linux-gnu/libpthread.so.0 --2762--   Considering
/usr/lib/debug/.build-id/26/820458adaf5d95718fb502d170fe374ae3ee70.debug
.. --2762-

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

2024年5月1日(水) 22:58 Chet Ramey :
> Yes, and I said I could work around this case.

Thank you for your consideration.

> > [1] https://lists.gnu.org/archive/html/bug-bash/2022-10/threads.html#00054
> > [2] https://lists.gnu.org/archive/html/bug-bash/2022-11/threads.html#00049
> > (continued. I haven't received a reply)
>
> To what?

To the email [2]. In particular, to the question in the second paragraph [2].

> > I'm not sure about the answer, but kre's opinion seems reasonable to
> > me.
>
> OK. The standard does say "reporting the exit status with the special
> parameter '?'" in
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_02

I think the description of the standard is still unclear, or the
standard doesn't try to specify this detail. I'm not sure whether we
can relate "the exit status reported by $?" and "the job status that
has been reported'' just from the description of the standard.
However, when we try to define "the job status that has been reported"
consistently with both the standard description and the actual
behaviors of shells, I think it is *one* reasonable interpretation to
regard the job status is reported when the exit status is stored in
$?.

> Correct. Is setting $? sufficient, or does the standard mean the reporting
> on foreground jobs that are killed by a signal -- one that shells don't
> ignore -- before the next command, which is what all shells do?
>
> For instance, if you run
>
> sleep 20; jobs

As far as I test it, yash and zsh don't print the information of the
signaled foreground jobs in this case, though ash/ksh families and
bash/posh print it as you tell. If we interpret $? can be used to
``report'' the foreground job status, the standard technically
wouldn't require printing the job information to stdout in this case,
but it doesn't prohibit printing either.

> and hit the `sleep' with SIGUSR1 from another process, all shells will
> print some message about the process exiting due to SIGUSR1 before `jobs'
> is executed, and `jobs' prints nothing, because the reporting has already
> happened.

In that theory, what happens when `sleep 20' is successfully
terminated without any signals? If the `reporting' must be the actual
text printed to stdout, « sleep 20; jobs » should print the job
information of the foreground terminated job `sleep 20', but not a
single does that.



To summarize my current thinking, I feel the standard didn't intend to
specify the detailed behavior for the foreground jobs. I don't expect
the `jobs' builtin to print any foreground jobs for consistency with
the behavior in the existing shells (except for Bash
trap/bindx/PROMPT_COMMAND handlers and 5.3 funsubs). An
``interpretation'' of the POSIX standard consistent with the existing
shell behavior would be to count $? as `reporting'.

With this interpretation, the `jobs' builtin will never print
foreground jobs because, at the point when the `jobs' builtin is
running in the foreground, any other foreground jobs should have been
terminated (while setting $?) or turned into background jobs as no
more than one jobs can run in the foreground at the same time. In a
case, the `jobs' builtin is inside a background job, the foreground
jobs that started after forking the background jobs are not visible to
the `jobs' builtin anyway.

--
Koichi