Re: ‘command … &’ creates subshell
Date:Wed, 01 Jul 2020 00:43:14 +0300
From:Dmitry Alexandrov
Message-ID:
| > If you need to ensure a disk executable is used,
| Of course. Why "command" otherwise?
That doesn't actually work, "command" can run built-ins, there is
actually no method (not even via use of "env") which guarantees
execution of an executable from an external file, other than by
using the complete path name (containing at least one '/') of the
desired file.
command avoids executing functions. That's its purpose (so you
can write
cd() {
test "$1" = "-b" && set -- whatever
command cd "$@"
}
(probably not quite that) without getting recursive "cd' invocation.
It also changes the rules for what happens with errors encountered
with special builtins (but unless in posix mode, bash tends to ignore
those anyway).
When given no options (just args) or with just -p (and args), there's no
reason at all for "command" to ever invoke a subshell for itself, all it
does is alter the search strategy for the command name that follows.
I'd actually call bash's behaviour a bug, and if not strictly that,
then certainly "extraordinarily non-intuitive" - rather than just an
"untaken (yet) opportunity for optimisation".
kre
Re: ‘command … &’ creates subshell
On 7/1/20 6:24 AM, Robert Elz wrote: > Date:Wed, 01 Jul 2020 00:43:14 +0300 > From:Dmitry Alexandrov > Message-ID: > > | > If you need to ensure a disk executable is used, > | Of course. Why "command" otherwise? > > That doesn't actually work, "command" can run built-ins, there is > actually no method (not even via use of "env") which guarantees > execution of an executable from an external file, other than by > using the complete path name (containing at least one '/') of the > desired file. Indeed -- that's why I specifically used the bashism $(type -P ...) as type -P forces the printing of an external file executable. Regarding use of env, I presume you're referring to the busybox behavior here, where busybox's builtin env applet will still execute other builtin applets. That's the only case I'm aware of that will not guarantee execution of an executable from an external file -- so shouldn't env be safe to use as such, if you know: a) your interpreter is bash rather than /bin/sh, b) the external disk executable "env" is not a symlink to busybox -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature
"shell-expand-line" wrongly(?) expanded $'foo' to $foo
See the following example (tested with bash 5.0.7): # bind -q shell-expand-line shell-expand-line can be invoked via "\e\C-e". # echo $PWD # press ESC C-e # echo /root # echo $( pwd ) # press ESC C-e # echo /root # echo $'foo'# press ESC C-e # echo $foo Is this a bug? -clark
Re: approach to async-signal safety in bash
On 6/30/20 4:28 PM, Godmar Back wrote:
> Hi,
>
> I'm trying to understand what approach bash takes to async-signal safety in
> its design.
I'm always interested in this kind of analysis, since signal handling
conflicts can be a ripe source of deadlocks -- think glibc and its dozens
of internal locks.
> Looking at the bash 5.0 code, I see some comments in the code about
> strategies to protect the jobs array and other data structures from
> arriving SIGCHLD signals, but I have questions about, for instance, these:
>
> - printable_job_status uses a 'static' variable "temp". However,
> printable_job_status is called during the execution of the builtin command
> "jobs" and here (I believe at least) without blocking or queuing SIGCHLD.
I don't see how. It either calls list_{all,running,stopped}_jobs, which end
up calling map_over_jobs and blocking SIGCHLD there, or jobs_builtin blocks
SIGCHLD itself before calling list_one_job.
> Therefore, if set -b is set, it could be reentered if a child process exits
> at that time. This could clobber 'temp'.
So this and the next one cover set -b, which by design has to notify
asynchronously, out of the signal handler. Maybe it would be better to run
it only if the shell is reading command input at the time the signal
arrives.
>
> - If set -b is set, calls to notify_job_status from the SIGCHLD handler may
> invoke printf or fprintf(stderr,), which is also called on builtin paths,
> for instance, when executing just "set" and listing all variables/functions.
>
> - The SIGALRM handler in eval.c calls printf and fflush(stdout), even
> though SIGALRM doesn't appear to be blocked elsewhere where printf() is
> called.
This is another case that I find unlikely to cause problems, since the code
path this call surrounds doesn't include other calls to printf. The signal
handler is only installed while the shell is reading a command.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Re: ‘command … &’ creates subshell
Date:Wed, 1 Jul 2020 07:54:05 -0400
From:Eli Schwartz
Message-ID: <511886e8-2262-39ee-2a01-7d284e981...@archlinux.org>
| Indeed -- that's why I specifically used the bashism $(type -P ...) as
| type -P forces the printing of an external file executable.
Yes, saw that, that method would work, I was commenting only upon
the use of "command" with the intent of executing a file.
| Regarding use of env, I presume you're referring to the busybox behavior
| here,
Or anything similar. env is an entirely reasonable command to build in,
it is doing very shell like operations, so any shell (including bash as
a loadable builtin I'd assume, but I don't do that kind of think to bash
so I don't know for sure) could build in env, and as soon as it is built
in it can run other built in commands (nothing in its spec forbids any of
that).
| so shouldn't env be safe to use as such, if you know: [...]
If you know all that, then you also know what is builtin, and that
if you say just "ls" or "awk" the external command will be run, as
there is no other possibility ("will be" assuming they're found in
the current PATH of course). Similarly you probably know that
/bin/cat works.
The whole point of trying to find a technique like was hoped for
using command, and others have used env for, and still others,
(exec command) (none of which are guaranteed to work) is so that
the script will work with different shells and in different
environments.
kre
Re: ‘command … &’ creates subshell
Robert Elz wrote: > | > If you need to ensure a disk executable is used, > | Of course. Why "command" otherwise? > > That doesn't actually work, "command" can run built-ins Ah, yes, thanks for correction. > there is actually no method ‹…› which guarantees execution of an executable > from an external file, other than by using the complete path name Well, I am not sure whether itʼs guaranteed, but the ‘exec’ as above does execute an external command in GNU Bash. signature.asc Description: PGP signature
Re: ‘command … &’ creates subshell
On 7/1/20 11:29 AM, Dmitry Alexandrov wrote: > Robert Elz wrote: >> | > If you need to ensure a disk executable is used, >> | Of course. Why "command" otherwise? >> >> That doesn't actually work, "command" can run built-ins > > Ah, yes, thanks for correction. > >> there is actually no method ‹…› which guarantees execution of an executable >> from an external file, other than by using the complete path name > > Well, I am not sure whether itʼs guaranteed, but the ‘exec’ as above does > execute an external command in GNU Bash. And in busybox, just like "env" it runs busybox builtin applets. -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature
Re: ‘command … &’ creates subshell
Robert Elz wrote: > The whole point of trying to find a technique like was hoped for using > command, and others have used env for, and still others, (exec command) (none > of which are guaranteed to work) is so that the script will work with > different shells and in different environments. One could argue, that an ability to ignore builtin utilities actually adds nothing to applicability of a script to different (unknown a priory) environments. On the contrary — it can rather break things. signature.asc Description: PGP signature
Re: Undocumented feature: Unnamed fifo '<(:)'
Thanks Pierre,
On Sun, Jun 28, 2020 at 10:48:42PM +0200, Pierre Gaston wrote:
> Maybe "coproc" is already the feature you need (limited to only 1 though)?
Correct: I missed this!
This work fine:
coproc stdbuf -o0 date -f - +%s 2>&1
DATEIN=${COPROC[1]} DATEOUT=$COPROC
echo >&$DATEIN 2009-02-13 23:31:30 UTC;read -u $DATEOUT out;declare -p out
declare -- out="1234567890"
echo foo bar >&$DATEIN ;read -u $DATEOUT out;declare -p out
declare -- out="date: invalid date 'foo bar'"
Unfortunely, coproc don't have option for separated error output...
If I want to handle errors, I could do this by using ``<(:)'' again:
exec 8<> <(:)
coproc stdbuf -o0 date -f - +%s 2>&8
DATEIN=${COPROC[1]} DATEOUT=$COPROC DATEERR=8
bound=wrong_date_$(uuidgen)
date_to_epoch() {
local _out
echo ${@:2}$'\n'$bound 1>&$DATEIN
read -u $DATEERR _out
if [ -z "${_out//*$bound*}" ]; then
read -u $DATEOUT $1
else
printf -v $1 %s "$_out"
read -u $DATEERR _out
return -1
fi
}
Usage: date_to_epoch
Then
if date_to_epoch result 2009-02-13 23:31:30 UTC ;then
echo $result ; else echo ERROR: $result ;fi
1234567890
if date_to_epoch result bad entry ;then
echo $result ; else echo ERROR: $result ;fi
ERROR: date: invalid date 'bad entry'
Again, I use this for not only with `date` and `bc`, but with `mysql`, `ftp`
or even `ssh` too.
--
Félix Hauri -- http://www.f-hauri.ch
Re: Undocumented feature: Unnamed fifo '<(:)'
On Wed, Jul 01, 2020 at 07:21:04PM +0200, felix wrote: > Again, I use this for not only with `date` and `bc`, but with `mysql`, `ftp` > or even `ssh` too. You're like the poster child for Expect.
Re: ‘command … &’ creates subshell
Date:Wed, 01 Jul 2020 18:29:46 +0300 From:Dmitry Alexandrov Message-ID: | Well, I am not sure whether it's guaranteed, but the "exec" | as above does execute an external command in GNU Bash. Yes, I know, but the way exec works in bash is broken (well, non-standard, not POSIX compliant). POSIX says: If command is specified, exec shall not return to the shell; but in bash... bash jinx$ echo $$ 23361 jinx$ exec : bash: exec: :: not found jinx$ echo $$ 23361 jinx$ exit exit "command was specified" yet exec returned to the shell. That should never happen (that it was unable to find the ':' builtin command to execute is another problem - but that one is shared by several shells). Turning on posix mode makes no difference: bash -o posix jinx$ echo $$ 18742 jinx$ exec : bash: exec: :: not found jinx$ echo $$ 18742 jinx$ exit exit These use 5.0.17(1)-release bash. What is required of exec (with a "command" arg) is that the current shell is replaced by the command, and when that exits, the process that used to be the shell exits. When not interactive bash seems to do this properly, but not when interactive. The spec makes no variation for that case, There's nothing in the spec that limits what kind of command can be exec'd (except to parse, it has to be a simple command, nothing else fits into the grammar) though many shells treat the exec builtin as simply a shorthand for a call to one of the exec*() system calls, and consequently only work properly if there's a command in the filesystem to run. kre
Re: ‘command … &’ creates subshell
On Thu, Jul 02, 2020 at 12:46:33AM +0700, Robert Elz wrote: > POSIX says: > > If command is specified, exec shall not return to the shell; > > but in bash... > > bash > jinx$ echo $$ > 23361 > jinx$ exec : > bash: exec: :: not found > jinx$ echo $$ > 23361 You're in an interactive shell. Bash chose this behavior to make your life better. unicorn:~$ cat foo #!/bin/bash exec : echo "I am still here" unicorn:~$ ./foo ./foo: line 2: exec: :: not found unicorn:~$
Re: ‘command … &’ creates subshell
On Jul 02 2020, Robert Elz wrote: > POSIX says: > > If command is specified, exec shall not return to the shell; But it also says: If command is not found, the exit status shall be 127. If command is found, but it is not an executable utility, the exit status shall be 126. And that's what bash does. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Re: ‘command … &’ creates subshell
Date:Wed, 01 Jul 2020 18:45:07 +0300 From:Dmitry Alexandrov Message-ID: | One could argue, that an ability to ignore builtin utilities actually adds | nothing to applicability of a script to different (unknown a priory) | environments. One can indeed (sometimes when someone a point like that, I reply that one can argue almost anything, it is not whether one can argue it or not that matters, but whether one wins the argument - that is succeeds in convincing people that one is correct) - but I won't do that here, as I don't disagree. The lack of an easy method to force execution of a command from the filesystem is very likely because there is generally no reason to do that. That is, no-one ever really wanted it, so no method was invented. However, for whatever reason, people keep asking how. Either by directly asking the question, or by picking one of the popular (but incorrect) methods, and either claiming it works, or complaining when it doesn't. As best I can tell, other than doing a search of PATH, and then using the full path name (which is what $(type -P cmd) does in bash, more succinctly than shells that don't have this -P option to type), the most reliable, that is, which is 99.9% unlikely to accidentally invoke a builtin, is gdb cmd run but putting that in a script would be absurd. kre
Re: approach to async-signal safety in bash
On Wed, Jul 1, 2020 at 10:58 AM Chet Ramey wrote:
>
> > Looking at the bash 5.0 code, I see some comments in the code about
> > strategies to protect the jobs array and other data structures from
> > arriving SIGCHLD signals, but I have questions about, for instance,
> these:
> >
> > - printable_job_status uses a 'static' variable "temp". However,
> > printable_job_status is called during the execution of the builtin
> command
> > "jobs" and here (I believe at least) without blocking or queuing SIGCHLD.
>
> I don't see how. It either calls list_{all,running,stopped}_jobs, which end
> up calling map_over_jobs and blocking SIGCHLD there, or jobs_builtin blocks
> SIGCHLD itself before calling list_one_job.
>
>
I think you are correct here. I set a breakpoint in there, but only checked
queue_sigchld
and not the actual SigBlk mask in /proc/*/status, which shows that SIGCHLD
was indeed blocked
on this path.
I did a little experiment, changing find_process to assert that SIGCHLD is
blocked (like the accompanying comment demands).
This fails on one of the provided unit tests for me, for instance:
[gback@oak bash-5.0]$ gdb ./bash
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-11.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Re: ‘command … &’ creates subshell
On 7/1/20 2:01 PM, Robert Elz wrote: > The lack of an easy method to force execution of a command from the > filesystem is very likely because there is generally no reason to do > that. That is, no-one ever really wanted it, so no method was invented. > > However, for whatever reason, people keep asking how. Either by directly > asking the question, or by picking one of the popular (but incorrect) > methods, and either claiming it works, or complaining when it doesn't. > > As best I can tell, other than doing a search of PATH, and then using the > full path name (which is what $(type -P cmd) does in bash, more succinctly > than shells that don't have this -P option to type), [...] One reason might be because the busybox applets for programs which are *normally* external disk utilities, can be lacking in functionality, often by a lot. In some cases, *hilariously* so (look at busybox ps, it's basically a long list of FIXMEs for POSIX requirements). I will confess however that I usually solve this by not considering busybox to be a suitable /bin/sh. The more general solution, possibly, would be e.g. invoking gawk, not awk, if you know you need extended functionality. awk might be a builtin (is, with busybox), but gawk probably isn't! -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature
Re: approach to async-signal safety in bash
On 7/1/20 2:18 PM, Godmar Back wrote: > I did a little experiment, changing find_process to assert that SIGCHLD is > blocked (like the accompanying comment demands). > This fails on one of the provided unit tests for me, for instance: Thanks. I think it's possible for a SIGCHLD to arrive during the window that find_process is executing when waitchld is not called from a SIGCHLD signal handler. I'll take a look at fixing that. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: "shell-expand-line" wrongly(?) expanded $'foo' to $foo
On 7/1/20 8:48 AM, Clark Wang wrote: > See the following example (tested with bash 5.0.7): > > > # bind -q shell-expand-line > shell-expand-line can be invoked via "\e\C-e". > > # echo $PWD # press ESC C-e > # echo /root > > # echo $( pwd ) # press ESC C-e > # echo /root > > # echo $'foo'# press ESC C-e > # echo $foo > > > Is this a bug? I suppose it is. $'...' isn't a word expansion, it's a form of quoting that gets translated and transformed into '...' by the parser. When the word expansion code sees $'...', unexpectedly, it just performs the usual quote removal. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: ‘command … &’ creates subshell
Date:Wed, 01 Jul 2020 20:00:23 +0200 From:Andreas Schwab Message-ID: <871rlvcdzc@igel.home> | But it also says: | | If command is not found, the exit status shall be 127. If command is | found, but it is not an executable utility, the exit status shall be | 126. If the command is built-in, it can hardly not be found, so the first half of that doesn't apply. All that's left is "executable utility". That isn't actually defined anywhere, but "utility" is, and while it excludes the special built-ins, everything else that can be run as a command counts, including those that only make sense when built in (like say, command). bash jinx$ echo $$ 10628 jinx$ exec command echo hello bash: exec: command: not found jinx$ echo $$ 10628 jinx$ exit NetBSD (deliberately, as it is stupid) ignores the requirement that all non-special builtns be also provided as commands in the filesystem, if we had a /usr/bin/command (or something) I assume bash would have run it (even though XCU 2.9.1.1 1.d says that "command" is one of the magic builtins which should be run as the built-in without bothering with a PATH search first). Having said all this, do be aware that this is all relevant to the current POSIX standard, the next version [probably] changes the wording to: If exec is specified with a utility operand, the shell shall execute a non-built-in utility as described in Section 2.9.1.6 with utility as the command name [...] It will also require interactive top level shells not to exit on exec error, [None of this is set in stone yet, and changes might still be made]. The first of those changes matches what most shells do (as I said earlier), the second will require changes in most shells, which currently do exit as the current standard requires., kre
Re: "shell-expand-line" wrongly(?) expanded $'foo' to $foo
Clark Wang wrote: > # echo $'foo'# press ESC C-e > # echo $foo + the same with $"foo". signature.asc Description: PGP signature
