Re: [@]@A weird behaviour when IFS does not contain space
On Thu, Jul 04, 2024 at 09:08:21PM -0400, Dale R. Worley wrote:
> Emanuele Torre writes:
> > [...]
> > Today, I have noticed that if IFS is set to a value that does not
> > include space, [@]@A will expand to a single value
> > [...]
> > As an aside, [*]@A always expands to the declare command joined by
> > space, even if the first character of IFS is not space; I think that is
> > a bit confusing, and surprising, but maybe that is done intentionally:
> > "intended and undocumented"(?).
>
> IMHO, the second observation is what should happen: The construct
> "${a[*]@A}", like almost all variable expansions, produces a *character
> string*, and then the characters are parsed into words and interpreted.
> In this case, the string contains spaces between the characters that are
> generated for each array member. But if IFS doesn't contain a space,
> then that string of characters isn't split into multiple words.
>
> Although perhaps "${a[*]@A}" should operate like "${a[*]}" does
> (according to my very old man page):
>
>If
>the word is double-quoted, ${name[*]} expands to a single word with the
>value of each array member separated by the first character of the IFS
>special variable, [...]
>
> That is, the blocks of the result string should be separated by the
> first character of IFS.
>
I don't really understand what you are trying to say here; anyway what I
meant was just that ${a[*]@A} always expands to 'declare -a foo=(bar)'
even though "${a[@]@A}" expands to 'declare' '-a' 'foo=(bar)'; it does
not expand to 'declare-afoo=(bar)' (for IFS=) or
'declarez-azfoo=(bar)' (for IFS=z). I find that surprising.
> The first case is more complicated. The man page says for "${a[@]}":
>
>[...] ${name[@]} expands each element of name to a sep‐
>arate word.
>
> This is the one case where the results of a variable expansion can't be
> modeled simply as replacing the variable reference with a string of
> characters (which are then parsed). It suggests that "${a[@]@A}" should
> automagically generate a separate word for each element of the array a,
> regardless of the value of IFS.
>
> Dale
No, that is not how ${[@]@A} should work since you are supposed to run
it as a command ( "${foo[@]@A}" ). Anyway, [@] should not be
influenced by IFS. This is obviously a bug.
Whether the intended behaviour was:
* to make both "${a[*]@A}" and "${a[@]@A}" always expand to 'declare -a
foo=(bar)' regardless of IFS, and you are actually supposed to use the
result as eval -- "${a[@]@A}"
* to make "${a[@]@A}" expand to separate values as it is doing now (so
that "${a[@]@A}" works as a command)
* to make "${a[*]@A}" always expand as space concatenated results of
"${a[@]@A}"
* to make "${a[*]@A}" always expand as IFS concatenated results of
"${a[@]@A}"
I don't know; but definitely that IFS=z makes "${a[@]@A}" expand how
"${a[*]@A}" is currently expanding is not intended.
o/
emanuele6
Re: waiting for process substitutions
On 6/29/24 10:51 PM, Zachary Santer wrote: They might already. Now I'm wondering if the documentation just needed updating. It might, but maybe not how you think. See below. I'm afraid to report this as a bug, because it feels like something that running bash in MSYS2 on Windows could be responsible for, but here goes. No, it's not MSYS-specific. Bash Version: 5.2 Patch Level: 26 Release Status: release Description: So bash can wait on process substitutions. 1) When all child processes are process substitutions: a. wait without arguments actually appears to wait for all of them, not just the last-executed one, contradicting the man page. This is how it is in bash-5.2. It does contradict the man page, and I reverted it to match the man page in October, 2022, after https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00107.html The whole business is kind of a mess. Let's see if we can figure out the best behavior. Through bash-4.4, `wait' without arguments didn't wait for process substitutions at all. I changed bash in 12/2017 to wait for all procsubs in addition to explicitly waiting for the last process substitution if its pid was the same as $!, mostly the result of this discussion: https://lists.gnu.org/archive/html/bug-bash/2017-12/msg2.html and that change was in bash-5.0. The man page didn't change until 06/2019. There were some bug reports about this behavior, e.g., https://lists.gnu.org/archive/html/bug-bash/2019-06/msg0.html In bash-5.1 I added code to treat process substitutions more like jobs, even though they're not, in response to this use case: https://lists.gnu.org/archive/html/bug-bash/2019-09/msg00021.html so you were then able to wait for each process substitution individually, as long as you saved $! after they were created. `wait' without arguments would still wait for all process substitutions (procsub_waitall()), but the man page continued to guarantee only waiting for the last one. This was unchanged in bash-5.2. I changed the code to match what the man page specified in 10/2022, after https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00107.html and that is what is in bash-5.3-alpha. The other thing that wait without arguments does is clear the list of remembered asynchronous pids and statuses. But process substitutions don't count against this (they probably should), so you've waited for the process substitutions and remembered their statuses. Bash should probably clear the list of reaped process substitutions before wait without arguments returns. But it doesn't do that yet. b. A subsequent call to wait listing all child process pids immediately terminates successfully. See above. `wait' finds the pids and exit statuses in the list of reaped procsubs. This is probably wrong: they should probably be cleared when `wait' runs without arguments, c. If calling wait -n in the middle of all this, whether listing only un-waited-on child process pids or all child process pids, it lists all argument pids as "no such job" and terminates with code 127. This is probably incorrect behavior. We've discussed this before. `wait -n' waits for the next process to terminate; it doesn't look back at processes that have already terminated and been added to the list of saved exit statuses. There is code tagged for bash-5.4 that allows `wait -n' to look at these exited processes as long as it's given an explicit set of pid arguments. 2) When a standard background process is added: a. wait without arguments waits for all child processes. Same. b. A subsequent call to wait listing all child process pids lists all argument pids as not children of the shell and terminates with code 127. This seems incorrect, or at least the change in behavior from 1b. is unexpected. It's different. The reaped background process triggers the cleanup of all background processes, including cleaning out the list of reaped procsubs. This is what doesn't happen, but probably should, in case 1. c. If calling wait -n in the middle of all this, we see that it only lists the pids from process substitutions as "no such job". Same. So the upshot is that bash should probably manage process substitutions even more like other asynchronous processes in that the pid/status pair should be saved on the list of saved exit statuses for `wait' to find it, and clear that list when `wait' is called without arguments. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ OpenPGP_signature.asc Description: OpenPGP digital signature
Comments on bash 5.2's undocumented <((
Bash 5.2 apparently added <(< file) that expand to the path to a fifo (openable only for read on BSD) to which the contents of file are written to, without documenting it. It also added >(< file) which is rather weird and fun; it expands to the path to a fifo (openable only for write on BSD), but (< file) will still write the contents of file to stdout even though >() didn't redirect stdout, so the content of file are not written to the fifo, but to caller's stdout. Effectively : >( file bash-5.2$ : >(> file bash-5.2$ : >(
Re: waiting for process substitutions
On 7/2/24 9:59 PM, Zachary Santer wrote:
I *am* seeing a difference between having lastpipe enabled (and job
control off) or not when running your example in the interactive
shell, though:
SECONDS=0; echo $'foo\nbar' | tee >(echo first ; exit 1) >(wc ; sleep
10 ; echo wc) >(tail -n 1; echo tail); wait; printf '%s\n'
"SECONDS=${SECONDS}"
With lastpipe disabled, wait exits immediately.
There are no process substitutions or asynchronous processes for `wait'
to return, since the last pipeline element containing the procsubs is
executed in a child process.
With lastpipe enabled,
it does seem to wait for everything.
The last pipeline element is executed in the current environment, and
any side effects persist in the current environment, so you have process
substitutions for `wait' to return.
I didn't look at the script behavior, since you don't have any pipelines
where lastpipe might have an effect.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
OpenPGP_signature.asc
Description: OpenPGP digital signature
Re: waiting for process substitutions
On 7/3/24 8:40 PM, Zachary Santer wrote:
On Wed, Jul 3, 2024 at 11:21 AM Chet Ramey wrote:
Process substitutions are word expansions, with a scope of a single
command, and are not expected to survive their read/write file descriptors
becoming invalid. You shouldn't need to `wait' for them; they're not
true asynchronous processes.
They clearly are. The fact that it results from a word expansion is irrelevant.
They're similar, but they're not jobs. They run in the background, but you
can't use the same set of job control primitives to manipulate them.
Their scope is expected to be the lifetime of the command they're a part
of, not run in the background until they're wanted.
Consider my original example:
command-1 | tee >( command-2 ) >( command-3 ) >( command-4 )
Any nontrivial command is going to take more time to run than it took
to be fed its input.
In some cases, yes.
The idea that no process in a process
substitution will outlive its input stream precludes a reading process
substitution from being useful.
It depends on whether or not it can cope with its input (in this case)
file descriptor being invalidated. In some cases, yes, in some cases, no.
And nevermind
exec {fd}< <( command )
I shouldn't do this?
Sure, of course you can. You've committed to managing the file descriptor
yourself at this point, like any other file descriptor you open with exec.
To me, a process substitution is just a way to avoid the overhead of
creating named pipes.
They're not (often) named pipes. What they are is a way to expose an
anonymous pipe in the file system. That may be close to what a named
pipe is, but a /dev/fd filename is not a named pipe (and has some
annoyingly system-specific differences).
Why should these be different in practice?
(1)
mkfifo named-pipe
child process command < named-pipe &
{
foreground shell commands
} > named-pipe
(2)
{
foreground shell commands
} > >( child process command )
Because you create a job with one and not the other, explicitly allowing
you to manipulate `child' directly?
In my actual use cases, I have:
(1)
A couple different scripts that alternate reading from multiple
different processes, not entirely unlike
sort -- <( command-1 ) <( command-2 ) <( command-3 )
except it's using exec and automatic fds.
Hypothetically, it could work like this:
{
commands
} {fd[0]}< <( command-1 ) {fd[1]}< <( command-2 ) {fd[2]}< <( command-3 )
But then again, *I can't get the pids for the processes if I do it this way*.
If you have to get the pids for the individual processes, do it a different
way. That's just not part of what process substitutions provide: they are
word expansions that expand to a filename. If the semantics make it more
convenient for you to use named pipes, then use named pipes.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
OpenPGP_signature.asc
Description: OpenPGP digital signature
Re: waiting for process substitutions
On Fri, Jul 05, 2024 at 15:16:31 -0400, Chet Ramey wrote: > They're similar, but they're not jobs. They run in the background, but you > can't use the same set of job control primitives to manipulate them. > Their scope is expected to be the lifetime of the command they're a part > of, not run in the background until they're wanted. Some scripts use something like this: #!/bin/bash exec > >(tee /some/logfile) 2>&1 logpid=$! ... exec >&- wait "$logpid" Your expectations might be different from those of bash's users.
Re: Comments on bash 5.2's undocumented <((
Emanuele Torre writes: > Bash 5.2 apparently added <(< file) that expand to the path to a fifo > (openable only for read on BSD) to which the contents of file are > written to, without documenting it. I suspect that this is a consequence of The comâ mand substitution $(cat file) can be replaced by the equivalent but faster $(< file). (Which oddly enough, I suggested for Bash some decades ago.) In the latter form, Bash doesn't set up a subprocess and then read the pipe from it but instead just reads the named file. Or rather, that was the initial implementation. I suspect that the code has been updated so that an "inner command" of "< file" now copies "file" to stdout (as if it was cat), and the various results you see are based on what the parent process does with that output. Dale
Re: Comments on bash 5.2's undocumented <((
On Fri, Jul 05, 2024 at 04:10:55PM -0400, Dale R. Worley wrote:
> Emanuele Torre writes:
> > Bash 5.2 apparently added <(< file) that expand to the path to a fifo
> > (openable only for read on BSD) to which the contents of file are
> > written to, without documenting it.
>
> I suspect that this is a consequence of
>
>The com‐
>mand substitution $(cat file) can be replaced by the equivalent but
>faster $(< file).
>
Yes, clearly that is influencing this new behaviour, but this is new:
<(( (Which oddly enough, I suggested for Bash some decades ago.) In the
> latter form, Bash doesn't set up a subprocess and then read the pipe
> from it but instead just reads the named file. Or rather, that was the
> initial implementation. I suspect that the code has been updated so
> that an "inner command" of "< file" now copies "file" to stdout (as if
> it was cat), and the various results you see are based on what the
> parent process does with that output.
More funny things have been discovered since.
It has been brought up when discussing this in the #bash IRC channel of
irc.libera.chat, that if you run eval ' f
bash-5.2$ echo ih > g
bash-5.2$ x=$(eval ' file
bash-5.2$ { _=$(eval '< file' >&3) ;} 3<&1
hi
hellobash-5.2$
I have also noticed that if you source a file from $() and the last
logical line of that file contains only a simple command with only one
read-only stdin redirection, bash will also print out that file, again,
in all versions of bash with $(<):
bash-5.2$ printf hello > myfile
bash-5.2$ cat <<'EOF' > tosource
echo foo
< myfile
EOF
bash-5.2$ x=$(. ./tosource; echo hey)
bash-5.2$ declare -p x
declare -- x=$'foo\nhellohey'
>
> Dale
o/
emanuele6
Re: Comments on bash 5.2's undocumented <((
On Fri, Jul 05, 2024 at 10:38:59PM +0200, Emanuele Torre wrote: > Yes, clearly that is influencing this new behaviour, but this is new: > <(((
