unsetting associative array executes commands
This behavior is quite surprising:
$ declare -A blah
$ blah['$(DOESNOTEXIST)']=broken
$ for i in "${!blah[@]}"; do echo "$i"; done
$(DOESNOTEXIST)
$ for i in "${!blah[@]}"; do unset blah["$i"]; done
bash: DOESNOTEXIST: command not found
bash: unset: [$(DOESNOTEXIST)]: bad array subscript
I wouldn't have anticipated the double expansion of "$i" there.
Is this a bug? Bash 5.1p4.
Re: unsetting associative array executes commands
On Thu, Mar 11, 2021 at 08:06:55AM -0700, Jason A. Donenfeld wrote:
> This behavior is quite surprising:
>
> $ declare -A blah
> $ blah['$(DOESNOTEXIST)']=broken
> $ for i in "${!blah[@]}"; do echo "$i"; done
> $(DOESNOTEXIST)
> $ for i in "${!blah[@]}"; do unset blah["$i"]; done
> bash: DOESNOTEXIST: command not found
> bash: unset: [$(DOESNOTEXIST)]: bad array subscript
unicorn:~$ declare -A blah
unicorn:~$ blah['$(DOESNOTEXIST)']=broken
unicorn:~$ for i in "${!blah[@]}"; do unset 'blah[$i]'; done
unicorn:~$ declare -p blah
declare -A blah=()
so-called pipe files (sh-np-*) do not get deleted when processes close.
Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' ``` back to ksh: ``` $ cat ~/build.log Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' ``` I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. I have forced some debug info: a) unset USE_MKTEMP b) added the PID of the process creating the special file c) additional debug info using fprintf(FILE, ...) fprintf(unlink_lst,"%d:%s:%d:%16s:%2d:%12d:%s\n", getpid(), __FILE__, __LINE__, "unlink_fifo_list", i, fifo_list[i].proc, fifo_list[i].file); That gives me the following information from the command above: 25231596:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) 21233868:../src/bash-5.0.18/subst.c:5372: add_fifo_list: 0: 0:/tmp//sh-np-21233868-1115804781 27918500:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/tee -a /home/aixtools/build.log 21233868:../src/bash-5.0.18/subst.c:5372: add_fifo_list: 1: 0:/tmp//sh-np-21233868-3761770506 27918500:../src/bash-5.0.18/subst.c:5506:reap_procsubs-calls:23 27918500:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 0: 0:/tmp//sh-np-21233868-1115804781 26869806:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/tee -a /home/aixtools/build.log 1>&2 26869806:../src/bash-5.0.18/subst.c:5506:reap_procsubs-calls:23 26869806:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 0: 27918500:/tmp//sh-np-21233868-1115804781 26869806:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 1: 0:/tmp//sh-np-21233868-3761770506 And this is remaining: $ ls -l /tmp/sh-np* | grep 21233868 prw--- 1 aixtools staff 0 Mar 11 08:07 /tmp/sh-np-21233868-1115804781 prw--- 1 aixtools staff 0 Mar 11 08:07 /tmp/sh-np-21233868-3761770506 Getting back to AdoptOpenJDK - a build process has roughly 3750 of these commands - leaving 7500 files behind in /tmp. On a busy day this can lead to over 100k empty files in /tmp. Thanks for any assistance. Regards, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: unsetting associative array executes commands
Single quotes with the nested double quote? That's nuts. But okay. I guess there really is some double expansion eval-like logic happening with the unset operator.
Re: unsetting associative array executes commands
On Thu, Mar 11, 2021 at 09:38:37AM -0700, Jason A. Donenfeld wrote: > Single quotes with the nested double quote? That's nuts. But okay. I > guess there really is some double expansion eval-like logic happening > with the unset operator. I removed the double quotes, as they serve no purpose there. For an indexed array, use: unset 'a[i]' For an associative array, use: unset 'a[$i]' In both cases, use single quotes, and all will be well. (In the indexed array case, there is another pitfall involving code injection in the arithmetic context, so only use sanitized index values.)
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. Probably because the process substitution does not exit before the shell does. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: unsetting associative array executes commands
On 3/11/21 10:06 AM, Jason A. Donenfeld wrote:
This behavior is quite surprising:
The idea is that array subscripts undergo a uniform set of expansions when
they're used, no matter the context. Builtins already undergo word
expansions as part of command execution. The result is double expansion
in certain cases (there are more; this is a staple of discussion here on
the list).
You can use the `assoc_expand_once' shell option to reduce the number of
expansions; it would have prevented the behavior you observed here, for
instance. But it's not perfect, as many people on the list will tell you.
$ declare -A blah
$ blah['$(DOESNOTEXIST)']=broken
$ for i in "${!blah[@]}"; do echo "$i"; done
$(DOESNOTEXIST)
$ for i in "${!blah[@]}"; do unset blah["$i"]; done
bash: DOESNOTEXIST: command not found
bash: unset: [$(DOESNOTEXIST)]: bad array subscript
$ cat x1
declare -A blah
blah['$(DOESNOTEXIST)']=broken
shopt -s assoc_expand_once
for i in "${!blah[@]}"; do unset blah["$i"]; done
declare -p blah
$ ../bash-5.1-patched/bash ./x1
declare -A blah=()
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: unsetting associative array executes commands
On 3/11/21 11:38 AM, Jason A. Donenfeld wrote: Single quotes with the nested double quote? That's nuts. But okay. I guess there really is some double expansion eval-like logic happening with the unset operator. It's not an `operator' per se, it's a builtin command. That means its arguments undergo the standard word expansions before `unset' even sees them. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: unsetting associative array executes commands
maybe implent a shopt -s no_expand_subshells
On Thu, Mar 11, 2021 at 9:20 PM Chet Ramey wrote:
> On 3/11/21 10:06 AM, Jason A. Donenfeld wrote:
> > This behavior is quite surprising:
>
> The idea is that array subscripts undergo a uniform set of expansions when
> they're used, no matter the context. Builtins already undergo word
> expansions as part of command execution. The result is double expansion
> in certain cases (there are more; this is a staple of discussion here on
> the list).
>
> You can use the `assoc_expand_once' shell option to reduce the number of
> expansions; it would have prevented the behavior you observed here, for
> instance. But it's not perfect, as many people on the list will tell you.
>
> >
> > $ declare -A blah
> > $ blah['$(DOESNOTEXIST)']=broken
> > $ for i in "${!blah[@]}"; do echo "$i"; done
> > $(DOESNOTEXIST)
> > $ for i in "${!blah[@]}"; do unset blah["$i"]; done
> > bash: DOESNOTEXIST: command not found
> > bash: unset: [$(DOESNOTEXIST)]: bad array subscript
>
>
> $ cat x1
> declare -A blah
> blah['$(DOESNOTEXIST)']=broken
> shopt -s assoc_expand_once
> for i in "${!blah[@]}"; do unset blah["$i"]; done
> declare -p blah
> $ ../bash-5.1-patched/bash ./x1
> declare -A blah=()
>
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>
Re: unsetting associative array executes commands
On Mär 11 2021, Chet Ramey wrote:
> $ cat x1
> declare -A blah
> blah['$(DOESNOTEXIST)']=broken
> shopt -s assoc_expand_once
touch blah\$
> for i in "${!blah[@]}"; do unset blah["$i"]; done
> declare -p blah
> $ ../bash-5.1-patched/bash ./x1
> declare -A blah=()
Andreas.
--
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
Sent from my iPhone > On 11 Mar 2021, at 18:15, Chet Ramey wrote: > > On 3/11/21 11:28 AM, Michael Felt wrote: >> Hi, >> Issue: AdoptOpenJDK build process makes bash calls in a particular way. An >> abbreviated (shorter pathnames) example is: >> ``` >> bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image >> test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >> >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a >> /home/aixtools/build.log >&2) >> Building targets 'product-images legacy-jre-image test-image' in >> configuration 'aix-ppc64-normal-server-release' > > I believe this is fixed in bash-5.1. Would it be difficult to give me a hint for 5.0. I could test further now i have a command that generates the issue. > > >> I added some debug statements to try and catch what is not happening. It >> seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so >> any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the >> special file created. > > Probably because the process substitution does not exit before the shell does. I was hoping that is what the wait routines were for. Also noticed that the second fifo never gets a pid. > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer >``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: unsetting associative array executes commands
2021年3月12日(金) 4:37 Alex fxmbsw7 Ratchev : > maybe implent a shopt -s no_expand_subshells There is already `shopt -s assoc_expand_once'. With this shell option, one can write `unset "blah[$i]"'. But such an option actually doesn't completely solve all similar cases including the associative array access in arithmetic contexts. Instead one can always write as $ unset 'blah[$i]' $ ((blah[\$i])) $ etc. Also, there was a thread on exactly the same topic last week: https://lists.gnu.org/archive/html/bug-bash/2021-03/msg4.html https://lists.gnu.org/archive/html/bug-bash/2021-03/msg5.html https://lists.gnu.org/archive/html/bug-bash/2021-03/msg8.html -- Koichi
Re: unsetting associative array executes commands
On 3/11/21 3:50 PM, Andreas Schwab wrote:
On Mär 11 2021, Chet Ramey wrote:
$ cat x1
declare -A blah
blah['$(DOESNOTEXIST)']=broken
shopt -s assoc_expand_once
touch blah\$
for i in "${!blah[@]}"; do unset blah["$i"]; done
declare -p blah
$ ../bash-5.1-patched/bash ./x1
declare -A blah=()
Sure:
"When using a variable name with a subscript as an argument to a com-
mand, such as with unset, without using the word expansion syntax de-
scribed above, the argument is subject to pathname expansion. If path-
name expansion is not desired, the argument should be quoted."
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 3/11/21 3:55 PM, Michael Felt (aixtools) wrote: Sent from my iPhone On 11 Mar 2021, at 18:15, Chet Ramey wrote: On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. Would it be difficult to give me a hint for 5.0. I could test further now i have a command that generates the issue. I can't reproduce it, but you can look at unlink_all_fifos() in bash-5.1. It's defined in subst.c and called in shell.c. Probably because the process substitution does not exit before the shell does. I was hoping that is what the wait routines were for. Also noticed that the second fifo never gets a pid. Bash doesn't wait for asynchronous processes before it exits unless you use `wait'. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
