"bashbug" script
Here was an interesting bug which was some what unexpected.
cat <(find ./ -iname t{1,2,3})
this is a valid command according to bash due to a bugged expansion of
{1,2,3} and the process expansion. It becomes three commands:
find ./ -iname t1
find ./ -iname t2
find ./ -iname t3
but as we know
find ./ -iname t{1,2,3}
is not a valid command, and the expansion behaviour is drasticly changed
while in the <(...)
although this
behaviour can be avoided by wrapping it up in things like "..." or $(...)
cat <(eval "find ./ -iname t{1,2,3}")
errors as expected and does not launch the three processes.
I feel that the expansion should be protected wheither wrapped or not to
preserve the expected results.
Please keep me posted on this bug.
Michael
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
remove extra blank lines in info table of contents
Please remove the extra line spacing between items in the main table of contents. While it looks reasonable in the HTML version, it looks dreadful in the info viewer and is inconsistent with the other tables of links. Configuration Information [Automatically generated, do not change]: Machine: i386 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i386' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i386-pc-linux-gnu' -DCONF_VENDOR='pc' -DSHELL -DHAVE_CONFIG_H -I. -I../bash -I../bash/include -I../bash/lib -g -O2 uname output: Linux eagle.endbracket.net 2.6.7eagle-20040916 #1 Thu Sep 16 23:36:29 EST 2004 i686 GNU/Linux Machine Type: i386-pc-linux-gnu Bash Version: 2.05b Patch Level: 0 Release Status: release Description: [Detailed description of the problem, suggestion, or complaint.] Repeat-By: [Describe the sequence of events that causes the problem to occur.] Fix: [Description of how to fix the problem. If you don't know a fix for the problem, don't include this section.] ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
nullglob option breaks complex parameter expansion/deletion
Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i386'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i386-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I../bash -I../bash/include -I../bash/lib -g -O2
uname output: Linux eagle.endbracket.net 2.6.7eagle-20040916 #1 Thu Sep 16
23:36:29 EST 2004 i686 GNU/Linux
Machine Type: i386-pc-linux-gnu
Bash Version: 3.0
Patch Level: 16
Release Status: release
Description:
I had been using ${HOSTNAME%%.*} in my prompt to show the local
host name portion of my full host name (e.g. localhost instead
of localhost.localdomain). After enabling the nullglob shell
option, this pattern is being replaced by a null string. I
don't think this behavior is desireable.
I don't fully understand the wording of the relevant portion of
the POSIX standard on this topic, but I would expect this
pattern to always function as it does without nullglob set:
- the /word/ to delete is a shell glob pattern to apply to the
result of expanding the /parameter/, regardless of whether the
word matches any file names in the current directory
- the deletion occurs regardless of whether the expansion of
/parameter/ is a file name
- the expansion of /parameter/ remains intact if the /word/ does
not match any portion of /parameter/
The standard also suggests that quoting /word/ shall cause it to
be regarded as literal rather than a pattern, but quoting /word/
exhibits the same problem.
Repeat-By:
$ shopt -u nullglob
$ connectioninfo='${HOST%%.*} ${USER}'
$ echo $connectioninfo
${HOST%%.*} ${USER}
$ shopt -s nullglob
$ echo $connectioninfo
${USER}
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
FYI! - bash-5.0-alpha - on AIX 6.1 , with xlc as compiler
"rltty.c", line 398.1: 1506-485 (S) Parameter declaration list is incompatible with declarator for rltty_warning. make[1]: *** [Makefile:72: rltty.o] Error 1 make: *** [Makefile:663: lib/readline/libreadline.a] Error 1 /opt/bin/make returned an error make -i compiles the rest of the files. Obviously, the final link is not going to work. FYI: It says: root@x066:[/data/prj/gnu/bash/bash-5.0-beta]make *** * * * GNU bash, version 5.0.0(1)-beta (powerpc-ibm-aix6.1.7.0) * * *** rm -f bash xlc_r -L./builtins -L./lib/readline -L./lib/readline -L./lib/glob -L./lib/tilde -L./lib/sh -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -o bash shell.o eval.o y.tab.o general.o make_cmd.o print_cmd.o dispose_cmd.o execute_cmd.o variables.o copy_cmd.o error.o expr.o flags.o jobs.o subst.o hashcmd.o hashlib.o mailcheck.o trap.o input.o unwind_prot.o pathexp.o sig.o test.o version.o alias.o array.o arrayfunc.o assoc.o braces.o bracecomp.o bashhist.o bashline.o list.o stringlib.o locale.o findcmd.o redir.o pcomplete.o pcomplib.o syntax.o xmalloc.o -lbuiltins -lglob -lsh -lreadline -lhistory -lcurses -ltilde /opt/lib/libintl.a /opt/lib/libiconv.a -ldl ld: 0711-317 ERROR: Undefined symbol: .rl_tty_set_default_bindings ld: 0711-317 ERROR: Undefined symbol: rl_prep_term_function ld: 0711-317 ERROR: Undefined symbol: rl_deprep_term_function ld: 0711-317 ERROR: Undefined symbol: .rl_restart_output ld: 0711-317 ERROR: Undefined symbol: ._rl_restore_tty_signals ld: 0711-317 ERROR: Undefined symbol: ._rl_disable_tty_signals ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information. signature.asc Description: OpenPGP digital signature
Building --enable-minimal-config fails: undefined reference to `glob_patscan'
Configuration Information [Automatically generated, do not change]: Machine: i586 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i586' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i586-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -g -O2 uname output: Linux fangorn 3.14.3 #10 SMP Fri Jun 27 23:41:04 CEST 2014 i586 GNU/Linux Machine Type: i586-pc-linux-gnu Bash Version: 4.3 Patch Level: 25 Release Status: release Description: Building bash as minimal shell fails due to glob_patscan missing The configuration information refers to a full features build, not to the minimal build. Repeat-By: ./configure --enable-minimal-config --disable-debugger make While linking, I get: ./lib/glob/libglob.a(gmisc.o): In function `glob_dirscan': bash-4.3/lib/glob/gmisc.c:398: undefined reference to `glob_patscan' Fix: Workaround: Add --enable-extended-glob
so-called pipe files (sh-np-*) do not get deleted when processes close.
Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' ``` back to ksh: ``` $ cat ~/build.log Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' ``` I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. I have forced some debug info: a) unset USE_MKTEMP b) added the PID of the process creating the special file c) additional debug info using fprintf(FILE, ...) fprintf(unlink_lst,"%d:%s:%d:%16s:%2d:%12d:%s\n", getpid(), __FILE__, __LINE__, "unlink_fifo_list", i, fifo_list[i].proc, fifo_list[i].file); That gives me the following information from the command above: 25231596:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) 21233868:../src/bash-5.0.18/subst.c:5372: add_fifo_list: 0: 0:/tmp//sh-np-21233868-1115804781 27918500:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/tee -a /home/aixtools/build.log 21233868:../src/bash-5.0.18/subst.c:5372: add_fifo_list: 1: 0:/tmp//sh-np-21233868-3761770506 27918500:../src/bash-5.0.18/subst.c:5506:reap_procsubs-calls:23 27918500:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 0: 0:/tmp//sh-np-21233868-1115804781 26869806:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/tee -a /home/aixtools/build.log 1>&2 26869806:../src/bash-5.0.18/subst.c:5506:reap_procsubs-calls:23 26869806:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 0: 27918500:/tmp//sh-np-21233868-1115804781 26869806:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 1: 0:/tmp//sh-np-21233868-3761770506 And this is remaining: $ ls -l /tmp/sh-np* | grep 21233868 prw--- 1 aixtools staff 0 Mar 11 08:07 /tmp/sh-np-21233868-1115804781 prw--- 1 aixtools staff 0 Mar 11 08:07 /tmp/sh-np-21233868-3761770506 Getting back to AdoptOpenJDK - a build process has roughly 3750 of these commands - leaving 7500 files behind in /tmp. On a busy day this can lead to over 100k empty files in /tmp. Thanks for any assistance. Regards, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 11/03/2021 22:27, Chet Ramey wrote: On 3/11/21 3:55 PM, Michael Felt (aixtools) wrote: Sent from my iPhone On 11 Mar 2021, at 18:15, Chet Ramey wrote: On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. Would it be difficult to give me a hint for 5.0. I could test further now i have a command that generates the issue. I can't reproduce it, but you can look at unlink_all_fifos() in bash-5.1. It's defined in subst.c and called in shell.c. FYI: Been digging in bash-5.0.18 - learning... Decided to give bash-5.1 a try. I doubt it is major, but I get as far as: "../../../src/bash-5.1.0/lib/sh/tmpfile.c", line 289.11: 1506-068 (W) Operation between types "char*" and "int" is not allowed. ld: 0711-317 ERROR: Undefined symbol: .mkdtemp ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information. make: 1254-004 The error code from the last command is 8. Using AIX 5.3 TL12 and xlc/xlC-v11, and a largely stripped system Of other OSS). I'll also give it a go on a more public server. Michael Probably because the process substitution does not exit before the shell does. I was hoping that is what the wait routines were for. Also noticed that the second fifo never gets a pid. Bash doesn't wait for asynchronous processes before it exits unless you use `wait'. OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 16/03/2021 14:38, Chet Ramey wrote: On 3/16/21 8:04 AM, Michael Felt wrote: Decided to give bash-5.1 a try. I doubt it is major, but I get as far as: "../../../src/bash-5.1.0/lib/sh/tmpfile.c", line 289.11: 1506-068 (W) Operation between types "char*" and "int" is not allowed. ld: 0711-317 ERROR: Undefined symbol: .mkdtemp Then how does configure find it? It's a POSIX function, and that file includes the appropriate headers. Haven't looked at configure yet - but do not find it in the usual places: root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/*.h root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/sys/*.h Also, not found on AIX 6.1 (TL9), but did find on AIX 7.1 TL4. Hope this helps, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 16/03/2021 16:21, Chet Ramey wrote: On 3/16/21 11:07 AM, Michael Felt wrote: On 16/03/2021 14:38, Chet Ramey wrote: On 3/16/21 8:04 AM, Michael Felt wrote: Decided to give bash-5.1 a try. I doubt it is major, but I get as far as: "../../../src/bash-5.1.0/lib/sh/tmpfile.c", line 289.11: 1506-068 (W) Operation between types "char*" and "int" is not allowed. ld: 0711-317 ERROR: Undefined symbol: .mkdtemp Then how does configure find it? It's a POSIX function, and that file includes the appropriate headers. Haven't looked at configure yet - but do not find it in the usual places: root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/*.h root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/sys/*.h Also, not found on AIX 6.1 (TL9), but did find on AIX 7.1 TL4. Sure, but configure checks for it, and bash only uses mkdtemp if configure finds it. Why does configure find it? Not sure. I'll have time again Thursday. Will be back. OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 11/03/2021 18:11, Chet Ramey wrote: On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. Probably because the process substitution does not exit before the shell does. I spent several days debugging - and, basically, they never get cleared because the fifo_struct never gets the (pid_t) -1 value assigned. Although the `reap` function does get called - there is never anything to do. The routine that does assign the (pid_t) -1 value is `wait`*something - and this is only called via an interrupt (aka signal) - as far as I could see. in the end I came up with a very simple - basically historical solution - for working with tempoary files that do not need to survive the process - unlink() the file immediately after open()> As I need to document for AdoptOpenJDK I created a mirror of savannah (git) and created a PR: https://github.com/aixtools/bash/pull/2 I expect much more testing is warrented - as to potential side-effects with the fifo struct (that is no longer accurate as the file may (read should) already be unlinked. Hope this helps, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
OK - this process on github has not gone exactly as I intended - merged
with master - while I wanted to update, ie., merge with branch 5.0.18.
So, the link may not be accurate.
The change is simple:
diff --git a/subst.c b/subst.c
index 843c9d39..3792e45c 100644
--- a/subst.c
+++ b/subst.c
@@ -5926,6 +5926,8 @@ process_substitute (string, open_for_read_in_child)
#if !defined (HAVE_DEV_FD)
/* Open the named pipe in the child. */
fd = open (pathname, open_for_read_in_child ? O_RDONLY : O_WRONLY);
+ /* now that the file is open (or not) * unlink it to keep garbage down */
+ unlink(pathname);
if (fd < 0)
{
/* Two separate strings for ease of translation. */
On 17/03/2021 16:17, Michael Felt wrote:
On 11/03/2021 18:11, Chet Ramey wrote:
On 3/11/21 11:28 AM, Michael Felt wrote:
Hi,
Issue: AdoptOpenJDK build process makes bash calls in a particular
way. An abbreviated (shorter pathnames) example is:
```
bash-5.0$ /usr/bin/printf "Building targets 'product-images
legacy-jre-image test-image' in configuration
'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a
/home/aixtools/build.log) 2> >(/usr/bin/tee -a
/home/aixtools/build.log >&2)
Building targets 'product-images legacy-jre-image test-image' in
configuration 'aix-ppc64-normal-server-release'
I believe this is fixed in bash-5.1.
I added some debug statements to try and catch what is not
happening. It seems that the fifo_list[i].proc value is never being
set to (pid_t)-1 so any call to `unlink_fifo()` or
`unlink_fifo_list()` does not unlink the special file created.
Probably because the process substitution does not exit before the
shell does.
I spent several days debugging - and, basically, they never get
cleared because the fifo_struct never gets the (pid_t) -1 value assigned.
Although the `reap` function does get called - there is never anything
to do.
The routine that does assign the (pid_t) -1 value is `wait`*something
- and this is only called via an interrupt (aka signal) - as far as I
could see.
in the end I came up with a very simple - basically historical
solution - for working with tempoary files that do not need to survive
the process - unlink() the file immediately after open()>
As I need to document for AdoptOpenJDK I created a mirror of savannah
(git) and created a PR: https://github.com/aixtools/bash/pull/2
I expect much more testing is warrented - as to potential side-effects
with the fifo struct (that is no longer accurate as the file may (read
should) already be unlinked.
Hope this helps,
Michael
OpenPGP_0x722BFDB61F396FC2.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
I tried as many combinations of commands as I could - and it seems that the regular behavior of dup2 on the opened fifo is enough to maintain communication. Going into a system test (ie. a normal AdoptOpemJDK build process) that has nearly 3500 commands, each with two process_substitution commands. Maybe that fails - and then I'll have yet another test. btw: other than the one open in the middle of process_substitution() I did not see anywhere where another process even tries to open the file. what I also noticed is that the process, (iirc) that opens the file - never 'returns' - it ends via sh_exit() and the end of the routine. Next time - I'll save all of my debug changes. Got a bit too rigorous when I cleaned up. On 17/03/2021 19:03, Chet Ramey wrote: On 3/17/21 11:52 AM, Michael Felt wrote: OK - this process on github has not gone exactly as I intended - merged with master - while I wanted to update, ie., merge with branch 5.0.18. So, the link may not be accurate. This is not correct. Process substitution is a word expansion that results in a pathname. You can't just remove the pathname after the child opens it. How will other processes that want to communicate with the process substitution use it? OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 17/03/2021 23:12, Chet Ramey wrote: On 3/17/21 3:29 PM, Michael Felt wrote: I tried as many combinations of commands as I could - and it seems that the regular behavior of dup2 on the opened fifo is enough to maintain communication. It's not, since FIFOs exist in the file system and have to be available to open(2) when the other process (consumer, producer) wants to use them. I didn't expect it to be perfect (see thx etc below). Better: I needed help in telling when it would most like fail! :) Going into a system test (ie. a normal AdoptOpemJDK build process) that has nearly 3500 commands, each with two process_substitution commands. Consider the following scenario. You want to perform a regression test of two versions of a program, each of which produces textual output. You run diff <(program-version-1 args) <(program-version-2 args) Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log). So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way). This is defined to provide `diff' with two arguments. Let's call them /var/tmp/sh-np12345 and /var/tmp/sh-np67890 So diff runs, sees two arguments, opens both files, and does its thing. Diff has to see two filenames when it runs, otherwise it's an error. But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file. a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routine b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion. Now, let's say your change is there. The shell still runs diff /var/tmp/sh-np12345 /var/tmp/sh-np67890 but, depending on how processes get scheduled, the shell forked to run the process substitutions has already unlinked those FIFOs. Diff will error out. The user will be unhappy. I will get bug reports. You have introduced a race condition. You may not get hit by it, but you cannot guarantee that no one will. No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned. The shell can't unlink the FIFO until it can guarantee that the processes that need to open it have opened it, and it can't guarantee that in the general case. It has to wait until the process completes, at least, and even that might not be correct. Again, my issue was with >(command) substitution - where the `files` get written to by the parent - rather than reading them. p.s. it is not my call to ask why they do not use regular redirection or pipes. Feels much simpler - but some people cannot miss the opportunity to use something blinky and shiney. p.p.s. - If you have `words of wisdom` re: why this approach is much better than `standard` redirection - I am all ears! *** Thanks again for the time to reply *** That's why the last-ditch approach is to remove all remaining FIFOs when the shell exits. btw: other than the one open in the middle of process_substitution() I did not see anywhere where another process even tries to open the file. They are not necessarily shell processes, but what if they were? Since a FIFO is an object in the file system, you can just open(2) it. That's ostensibly the advantage of FIFOs. what I also noticed is that the process, (iirc) that opens the file - never 'returns' - it ends via sh_exit() and the end of the routine. Of course. It's a process that is forked to run the command specified in the process substitution. What else does it need to do? OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
Scraping through this - thanks for the lessons aka explanations. On 18/03/2021 16:08, Chet Ramey wrote: On 3/18/21 5:53 AM, Michael Felt wrote: Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log). So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way). It's the redirection. The shell does the open, since the filename resulting from process substitution is the target of a redirection operator. This is a common idiom -- so common, in fact, that people have interpreted it to mean that the entire `> >(xxx)' is a single operator. However, the shell expands redirections in the child process it forks to exec printf, so that child shell is what does the process substitution. That might be the problem here. The command itself doesn't do anything, though. `tee' just sits there waiting for data to write to log files. It has no purpose. I'm not sure what the intent is. If you wrapped that command into a script, it's unlikely that either `tee' would exit (why would they?) before `printf' exits and the script completes. In bash-5.0, there would be nothing to remove the FIFOs. If I understand correctly, the commands are generated by gmake as it processes targets. This is defined to provide `diff' with two arguments. Let's call them /var/tmp/sh-np12345 and /var/tmp/sh-np67890 So diff runs, sees two arguments, opens both files, and does its thing. Diff has to see two filenames when it runs, otherwise it's an error. But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file. Yes and no. Process substitution is a word expansion that results in a filename. The stuff between the parens defines a command that writes to or reads from a pipe expressed as a filename (/dev/fd/NN or a FIFO) that is the result of the word expansion. In this case, the process substitution is the target of a redirection, so the shell performs that word expansion before it execs diff. a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routine It executes the specified command and exits. Got it: must remember - initially it is bash busy with word expansion (a new bash child for each 'process substitution' b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion. It doesn't have to. The filename itself is the expansion: it's an object you can use to communicate with an asynchronous process. This is how you can have programs that expect a filename use program-generated output, for instance, without using a temp file. Yes - for me at least it is much easier to fathom as input - that ends and behaves/looks/feels like EOF. In this case, it opens the FIFO because it is the target of a redirection. Nods: just as it would if it was the output from a program than had just run 'moments' before. Now, let's say your change is there. The shell still runs diff /var/tmp/sh-np12345 /var/tmp/sh-np67890 but, depending on how processes get scheduled, the shell forked to run the process substitutions has already unlinked those FIFOs. Diff will error out. The user will be unhappy. I will get bug reports. You have introduced a race condition. You may not get hit by it, but you cannot guarantee that no one will. No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned. You mean in terms of using the filename as an argument to a shell builtin? Otherwise you'll have to trace into other child process execution. /usr/bin/printf is not a built-n (afaik) If I understand correctly - from printf perspective we have /usr/bin/printf "Some formatted message" > /tmp/sh-np.123456 2> /tmp/sh-np.9876543 & And, if for ease of discussion we say program1 is PID-123456 and program is PID-987654 - these programs have no way of knowing their stdin is named /tmp/sh-np-something? BING: as the dutch (used to) say - the quarter drops - the other programs (e.g., tee) have no fifo knowledge - they are who/what they are. What maybe needed for this situation - is rather than directly execev() the program - yet another fork (for the execve() - and wait for that program to hit it's EOF on input and then the sleeping 'word expansion child' cleans up the fifo file it created for the communication path. * Am I getting closer? :) The shell can't unlink the FIFO until it can guarantee that the proce
Sort command doesn't sort '@' character correctly
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -g -O2 -fdebug-prefix-map=/build/bash-a6qmCk/bash-5.0=. -fstack-protector-stron> uname output: Linux ubuntu 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64 x86_> Machine Type: x86_64-pc-linux-gnu Bash Version: 5.0 Patch Level: 17 Release Status: release Description: sorting lines where some starts with @ and some doesn't, it will not sort the lines correct Repeat-By: echo "xxaxxon" > test.txt echo "@zorg" >> test.txt echo "@jupiterlander" >> test.txt cat test.txt | sort Note it prints out: @jupiterlander xxaxxon @zorg Expected: @jupiterlander @zorg xxaxxon
Re: Revisiting Error handling (errexit)
Sounds great to me. I also use Bash for mission-critical processes.
Philip
On Mon, Jul 4, 2022 at 8:22 AM Yair Lenga wrote:
>
> Hi,
>
> In my projects, I'm using bash to manage large scale jobs. Works very well,
> especially, when access to servers is limited to ssh. One annoying issue is
> the error handling - the limits/shortcomings of the 'errexit', which has
> been documented and discussed to the Nth degree in multiple forums.
>
> Needless to say, trying to extend bash to support try/catch clauses, (like
> other scripting solutions: Python, Groovy, Perl, ...), will be a major
> effort, which may never happen. Instead, I've tried to look into a minimal
> solution that will address the most common pitfall of errexit, where many
> sequences (e.g., series of commands in a function) will not properly
> "break" with 'errexit'. For example:
>
> function foo {
> cat /missing/file # e.g.: cat non-existing file.
> action2 # Executed even if action 1 fail.
> action3
> }
>
> set -oerrexit # want to catch errors in 'foo'
> if ! foo ; then
> # Error handling for foo failure
> fi
>
> I was able to change Bash source and build a version that supports the new
> option 'errfail' (following the 'pipefail' naming), which will do the
> "right" thing in many cases - including the above - 'foo' will return 1,
> and will NOT proceed to action2. The implementation changes the processing
> of command-list ( '{ action1 ; action2 ; ... }') to break of the list, if
> any command returns a non-zero code, that is
>
> set -oerrfail
> { echo BEFORE ; false ; echo AFTER ; }
>
> Will print 'BEFORE', and return 1 (false), when executed under 'errfail'
>
> I'm looking for feedback on this implementation. Will be happy to share the
> code, if there is a chance that this will be accepted into the bash core
> code - I believe it will make it easier to strengthen many production
> systems that use Bash.
>
> To emphasize, this is a minimal proposal, with no intention of expanding it
> into full support for exceptions handling, finally blocks, or any of the
> other features implemented in other (scripting) languages.
>
> Looking for any feedback.
>
> Yair
Re: Light weight support for JSON
He has a point, though. To have some of the functionality of jq inside Bash
may be very useful.
If he can supply a patch, why not?
Philip Orleans
On Sun, Aug 28, 2022, 3:22 PM John Passaro wrote:
> interfacing with an external tool absolutely seems like the correct answer
> to me. a fact worth mentioning to back that up is that `jq` exists. billed
> as a sed/awk for json, it fills all the functions you'd expect such an
> external tool to have and many many more. interfacing from curl to jq to
> bash is something i do on a near daily basis.
>
> https://stedolan.github.io/jq/
>
> On Sun, Aug 28, 2022, 09:25 Yair Lenga wrote:
>
> > Hi,
> >
> > Over the last few years, JSON data becomes a integral part of processing.
> > In many cases, I find myself having to automate tasks that require
> > inspection of JSON response, and in few cases, construction of JSON. So
> > far, I've taken one of two approaches:
> > * For simple parsing, using 'jq' to extract elements of the JSON
> > * For more complex tasks, switching to python or Javascript.
> >
> > Wanted to get feedback about the following "extensions" to bash that will
> > make it easier to work with simple JSON object. To emphasize, the goal is
> > NOT to "compete" with Python/Javascript (and other full scale language) -
> > just to make it easier to build bash scripts that cover the very common
> use
> > case of submitting REST requests with curl (checking results, etc), and
> to
> > perform simple processing of JSON files.
> >
> > Proposal:
> > * Minimal - Lightweight "json parser" that will convert JSON files to
> bash
> > associative array (see below)
> > * Convert bash associative array to JSON
> >
> > To the extent possible, prefer to borrow from jsonpath syntax.
> >
> > Parsing JSON into an associative array.
> >
> > Consider the following, showing all possible JSON values (boolean,
> number,
> > string, object and array).
> > {
> > "b": false,
> > "n": 10.2,
> > "s: "foobar",
> > x: null,
> > "o" : { "n": 10.2, "s: "xyz" },
> > "a": [
> > { "n": 10.2, "s: "abc", x: false },
> > { "n": 10.2, "s": "def" x: true},
> > ],
> > }
> >
> > This should be converted into the following array:
> >
> > -
> >
> > # Top level
> > [_length] = 6# Number of keys in object/array
> > [_keys] = b n s x o a# Direct keys
> > [b] = false
> > [n] = 10.2
> > [s] = foobar
> > [x] = null
> >
> > # This is object 'o'
> > [o._length] = 2
> > [o._keys] = n s
> > [o.n] = 10.2
> > [o.s] = xyz
> >
> > # Array 'a'
> > [a._count] = 2 # Number of elements in array
> >
> > # Element a[0] (object)
> > [a.0._length] = 3
> > [a.0._keys] = n s x
> > [a.0.n] = 10.2
> > [a.0.s] = abc
> > [a.0_x] = false
> >
> > -
> >
> > I hope that example above is sufficient. There are few other items that
> are
> > worth exploring - e.g., how to store the type (specifically, separate the
> > quoted strings vs value so that "5.2" is different than 5.2, and "null"
> is
> > different from null.
> >
> > I will leave the second part to a different post, once I have some
> > feedback. I have some prototype that i've written in python - POC - that
> > make it possible to write things like
> >
> > declare -a foo
> > curl http://www.api.com/weather/US/10013 | readjson foo
> >
> > printf "temperature(F) : %.1f Wind(MPH)=%d" ${foo[temp_f]}, ${foo[wind]}
> >
> > Yair
> >
>
Re: IFS field splitting doesn't conform with POSIX
There is an additional problem with IFS and the command read
Suppose I have variable $line with a string "a,b,c,d"
IFS=',' read -r x1 <<< $line
Bash will assign the whole line to x1
echo $x1
line="a,b,c,d";IFS=',' read -r x1 <<< $line;echo $x1;
a,b,c,d
but if I use two variables
line="a,b,c,d";IFS=',' read -r x1 x2 <<< $line;echo "$x1 ---> $x2";
a ---> b,c,d
this is incorrect. If IFS=",", then a read -r statement must assign the
first value to the single variable, and disregard the rest.
and so on, with (n) variables.
The compelling reason is: I may not know how many values are stored in the
comma-separated list.
GNU AWK, for instance, acts responsibly in the same exact situation:
line="a,b,c,d";awk -F, '{print $1}' <<< $line
a
We need to fix this
On Sat, Apr 1, 2023, 6:11 PM Mike Jonkmans wrote:
> On Sat, Apr 01, 2023 at 03:27:47PM -0400, Lawrence Velázquez wrote:
> > On Fri, Mar 31, 2023, at 2:10 PM, Chet Ramey wrote:
> > > kre filed an interpretation request to get the language cleaned up.
> >
> > For those who might be interested:
> >
> > https://austingroupbugs.net/view.php?id=1649
>
> Thanks for the link.
>
> And well done, kre!
>
> --
> Regards, Mike Jonkmans
>
>
Re: FEATURE REQUEST : shell option to automatically terminate child processes on parent death
I support this feature. On Sat, Nov 11, 2023, 11:29 AM Corto Beau wrote: > Configuration Information [Automatically generated, do not change]: > Machine: x86_64 > OS: linux-gnu > Compiler: gcc > Compilation CFLAGS: -g -O2 > uname output: Linux zinc 6.6.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 08 > Nov 2023 16:05:38 + x86_64 GNU/Linux > Machine Type: x86_64-pc-linux-gnu > > Bash Version: 5.2 Patch > Level: 21 > Release Status: release > > Description: > Hi, > > I would like to suggest a new shell option to ensure child processes are > automatically killed when the parent dies. > > Right now, it's already possible to emulate this feature by setting a > trap to kill all child processes on exit (trap "kill 0" EXIT), but > obviously it doesn't work if the process is terminated by a signal that > cannot be caught (like SIGKILL). > > On Linux, it can be done by setting the PR_SET_PDEATHSIG flag to > propagate the parent termination signal to the child, regardless of > whether the signal can be caught or not. > > The rationale for this feature is that scripts spawning background > processes to listen to various events (udev events, window manager > events, etc) often leave orphan processes behind when terminated > forcefully. > > I've attached a proof-of-concept patch.
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 17:51, Bob Proulx wrote: > Are you thinking that setting shopts should in some way be persistent > across program invocations? That would be pretty annoying and a > severe bug if it did. > > Are you forgetting to put your desired configuration into ~/.bashrc > where it is loaded when bash starts? > > Are you forgetting to put > source "$HOME/.bashrc" > into your ~/.bash_profile where it will source your .bashrc when you > log into your account? No. Read his email again.
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 15:56, Slevin McGuigan wrote: > I am unsure whether or not this a bug. >From what I can tell, it's not so much a bug as it is an inadequacy: When you quit bash, the history is stored very naively in "$HISTFILE"; the history is simply dumped to it line-by-line, and each line of the file is later interpreted by a new bash instance as a complete command line input. Hence, the multi-line nature of your command input is lost between dumping it to "$HISTFILE" and later populating the history from "$HISTFILE". The solution would be to invent a more robust file format for "$HISTFILE".
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 18:02, Jon Seymour wrote: > The version I tried on Linux 3.2.25 does have a .bash_history > format that could support it, but it still behaved the same way. How do you mean? I'm running bash version "4.1.9(2)-release" on GNU/Linux, and the resulting history file doesn't seem like it's storing anything more than lines of text naively dumped from the multi-line example.
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 19:15, Jon Seymour wrote: > Here's the format I see in my history. > > #1296950184 > for i in 1 2 > do > echo $i > done > #1296950194 > exit > > HISTTIMEFORMAT is: > > HISTTIMEFORMAT='[%m.%d.%y] %T ' > > > bash -version is: > > GNU bash, version 3.2.25(1)-release (i686-redhat-linux-gnu) > Copyright (C) 2005 Free Software Foundation, Inc. > > jon. As you can see, the timestamp that is actually recorded is the format '%s' rather than the one given by "$HISTTIMEFORMAT". >From `man bash': HISTTIMEFORMAT If this variable is set and not null, its value is used as a format string for strftime(3) to print the time stamp associated with each history entry displayed by the history builtin. If this variable is set, time stamps are written to the history file so they may be preserved across shell sessions. This uses the history comment character to distinguish timestamps from other history lines. namely: its value is used as a format string for strftime(3) to print the time stamp associated with each history entry displayed by the history builtin. So, if you run `history', you'll not only get the commands in the history list, but you'll also get the time at which the commands were last run (formatted according to "$HISTTIMEFORMAT"). In other words, it's not helpeful in this case.
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 20:02, Michael Witten wrote: > So, if you run `history', you'll not only get the commands in the > history list, but you'll also get the time at which the commands > were last run (formatted according to "$HISTTIMEFORMAT"). > > In other words, it's not helpeful in this case. Of course, I suppose bash could be taught to build multi-line comments from command lines that share the same timestamp, but given the nature of how that information is recorded, it seems like it may not be a very robust solution.
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 20:12, Jon Seymour wrote: > You don't have to do that - the timestamp is encoded in a "comment" > line between entries. See the example below. One could simply assume > all lines between two lines beginning with # are part of the one > entry, That's what I was saying. However, it seems like an unrobust way have handling the situation (both for multi-line comments and for the existing timestamp purposes).
Re: multi-line commands in the history get split when bash is quit
On Sat, Feb 5, 2011 at 20:09, Jon Seymour wrote: > I guess the point is that in versions of bash that do store the > timestamp in the .bash_history file To clarify, the timestamp is stored whenever HISTTIMEFORMAT has a non-null value; the bash version doesn't particularly matter unless you're suggesting that HISTTIMEFORMAT is non-null by default under some bash versions. If there aren't really any concerns about using the same history file with older versions of bash, then wouldn't it be better to have a new file format that can handle multi-line commands more directly?
Re: Bash not reacting to Ctrl-C
On Wed, Feb 9, 2011 at 08:53, Ingo Molnar wrote:
>
> * Michael Witten wrote:
>
>> On Mon, Feb 7, 2011 at 07:08, Oleg Nesterov wrote:
>> > Now that it is clear what happens, the test-case becomes even more
>> > trivial:
>> >
>> > bash-4.1$ ./bash -c 'while true; do /bin/true; done'
>> > ^C^C
>> >
>> > needs 4-5 attempts on my machine.
>>
>> I feel like the odd penguin out.
>>
>> I can't reproduce the behavior in question when using that example (I
>> haven't tried the other).
>>
>> I'm running:
>>
>> * bash version 4.1.9(2)-release (i686-pc-linux-gnu)
>>
>> * linux 2.6.38-rc4 (100b33c8bd8a3235fd0b7948338d6cbb3db3c63d)
>
> Oleg provided another testcase, can you reproduce the Ctrl-C problem with this
> it?
>
> #!/bin/bash
>
> perl -we '$SIG{INT} = sub {exit}; sleep'
>
> echo "Hehe, I am going to sleep after ^C"
> sleep 100
>
>
> Thanks,
>
> Ingo
>
Yes, that requires me to press Ctrl-C twice in order to escape the
entire script. However, what do you expect the following to do:
#!/bin/bash
perl -we '$SIG{INT} = "IGNORE"; sleep 10'
echo "Hehe, I am going to sleep after ^C"
sleep 100
bash tab variable expansion question?
Configuration Information [Automatically generated, do not change]: Machine: i686 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/local/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -g -O2 uname output: Linux penguin.localdomain 2.6.18-194.26.1.el5.028stab079.2 #1 SMP Fri Dec 17 19:25:14 MSK 2010 i686 i686 i386 GNU/Linux Machine Type: i686-pc-linux-gnu Bash Version: 4.2 Patch Level: 0 Release Status: release Description: Hi! Example: In bash, version 4.1.5(1)-release: $ echo $PWD/ will expand the $PWD variable to your current directory while in bash, version 4.2.0(1)-release: $ echo $PWD/ will just escape the $ in front of the $ variable i.e: $ echo \$PWD/ The shell-expand-line (Ctrl-Alt-e) works but before I could use just TAB Any hints why? Any way to get the 4.1 behavior in 4.2? Can someone confirm... Is this a bug or a feature? Thanks in advance Michael
Re: bash tab variable expansion question?
2011/2/27 Clark J. Wang > > A workaround is fine but is the 4.2 behavior bug or not? > I agree...Would be nice if someone could confirm if this is a bug or not? I'm betting that this is a bug :-) //Michael
[PATCH 0/6] Off-by-one bug fix and clean up
The parse_token_word() function (parse.y:4297) has a couple of off-by-one assignments to the `token' array, which are really 2 manifestations of the same bug. These patches first mask the bug by solving the manifestations independently, and then eradicate the bug by uniformly dismantling the noodliness in which it is hiding. Here is the entire patch series at a glance: Solve the manifestations: [PATCH 1/6] Bug: extglob: Fix off-by-one assignment in read_token_word() [PATCH 2/6] Bug: shellexp: Fix off-by-one assignment in read_token_word() Slice through the noodles: [PATCH 3/6] Clean: parse_token_word(): relax memory requirements (off by one) [PATCH 4/6] Clean: More direct coupling between assignment and allocation Extraneous tidying up: [PATCH 5/6] Clean: Remove unnecessary xmalloc()/strcpy() [PATCH 6/6] Clean: Remove unnecessary backslashes (line continuation) All told, the changes are fairly minimal: parse.y | 50 ++ 1 files changed, 26 insertions(+), 24 deletions(-)
[PATCH 2/6] Bug: shellexp: Fix off-by-one assignment in read_token_word()
This fixes a parsing bug that would probably never be of trouble in
practice; it is an edge case of edge cases. Nevertheless, bash should
be able to handle it without corrupting memory.
Consider the following:
$ TOKEN_DEFAULT_INITIAL_SIZE=496 # parse.y:1282
$ n=TOKEN_DEFAULT_INITIAL_SIZE # notational convenience
$ unset BASH_ENV
$ { for ((i=0;iword = (char *)xmalloc (1 + token_index);
the_word->flags = 0;
strcpy (the_word->word, token);
if (dollar_present)
the_word->flags |= W_HASDOLLAR;
yylval.word = the_word;
return WORD;
}
Before the `$' is reached, the `for(;;)' loop looks essentially like this:
for (;;)
token[token_index++] = shell_getc (1);
That is, the `a' characters are copied into the token array.
When the `$' is read, `token_index' has the value:
TOKEN_DEFAULT_INITIAL_SIZE-5 = 491
and the following code is executed:
if (character == '$')
{
/** simulate peek_char and parse_matched_pair() **/
shell_getc (1);
shell_getc (1);
/*/
token[491] = character;
token[492] = '{';
strcpy (token + 493, "a}");
token_index = 495;
dollar_present = 1;
goto next_character;
}
The value of `token_index' represents the number of characters that have
already been stored in `token'; in this case, 495 characters have been
stored. As per the `goto', the next character is read:
next_character:
character = shell_getc (1);
This reads the `\177' character, CTLNUL. which is handled with this code:
if (character == CTLESC || character == CTLNUL)
{
token[495] = CTLESC;
token_index=496; /* assignment below: token[497] = CTLNUL; */
}
token[token_index++] = character;
Thus, the last assignment is equivalent to:
token[496] = character;
token_index=497;
which is, of course, one past the end of `token', because there are only
496 indicies for `token' according to its current allocation (the maximum
`token' index is 495).
The most "conservative" fix is the one presented in this patch:
Allocate more space than is necessary by imposing
an unnecessarily strict requirement, thereby "masking"
the bug.
In my opinion, this is wrongheaded, but it only requires the change of
one character and it's exactly what other code paths in read_token_word()
[currently] do, probably by mistake. :-P
Signed-off-by: Michael Witten
---
parse.y |2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/parse.y b/parse.y
index 6c095f5..55874aa 100644
--- a/parse.y
+++ b/parse.y
@@ -4473,7 +4473,7 @@ read_token_word (character)
ttok = parse_matched_pair (cd, '[', ']', &ttoklen, 0);
if (ttok == &matched_pair_error)
return -1; /* Bail immediately. */
- RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 2,
+ RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 3,
token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
token[token_index++] = character;
--
1.7.4.22.g14b16.dirty
[PATCH 3/6] Clean: parse_token_word(): relax memory requirements (off by one)
This doesn't really fix a bug; it just relaxes the required `room' that is
necessary in the allocated buffer.
>From general.h:164:
#define RESIZE_MALLOCED_BUFFER(str, cind, room, csize, sincr) \
do { \
if ((cind) + (room) >= csize) \
{ \
while ((cind) + (room) >= csize) \
csize += (sincr); \
str = xrealloc (str, csize); \
} \
} while (0)
Firstly, RESIZE_MALLOCED_BUFFER() is written to assume that `cindex'
is the largest index such that `str[index]' is live data. However, in
parse_token_word(), RESIZE_MALLOCED_BUFFER() is given `token_index' as
`cindex' when the required index is actually `token_index-1', thereby
inflating the measurement of used space by one.
An alternative interpretation is that the value of `token_index' is the
number of live characters already stored in the `token' array. Then the
macro's allocation condition:
(cind) + (room) >= csize
Can be reinterpreted as:
(# of live slots) + (# of slots to be made live) >= (# of slots in buffer)
or:
`str' must have at least 1 unused slot after all live data
has been slotted.
What is this 1 unused slot for? Well, in this case, it seems to
be for the null character '\0' placed there by strcpy() or
when the token is complete.
Thus, the code in question:
RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen + 2,
token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
says:
`token' must have at least 1 unused slot after all live data
has been slotted.
However, it lists the number of live slots as:
cind + room = token_index + ttranslen + 2
when the actual number of live slots is:
(# of live slots) + (# of slots to be made live) = token_index + ttranslen
So, it would seem that there would be enough for the final '\0' character
if the requirement were reduced to:
RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen,
token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
However, there's one more wrinkle: parse_token_word() has the
following code for handling CTLESC and CTLNUL:
if (character == CTLESC || character == CTLNUL)
token[token_index++] = CTLESC;
/* ... minor stuff in between ... */
token[token_index++] = character;
RESIZE_MALLOCED_BUFFER (token, token_index, 1, token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
As you can see, each CTLESC or CTLNUL encountered on the input is added to
the token only after a CTLESC has been artificially inserted first; this
means that there must actually be room not just for the '\0' added in the
last slot when the token is complete, but also for this extra CTLESC. That
is, parse_token_word()'s overall requirement is:
`token' must have at least 2 unused slots while a CTLESC or CTLNUL
may still be read, one for the extra CTLESC and one for the final
'\0' character.
Thus, we need to inflate the `room' value by 1, from `ttranslen' to
`ttranslen + 1':
RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen + 1,
token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
This patch achieves that final result by deflating the `room' value by 1,
from `ttranslen + 2' to `ttranslen + 1'.
Signed-off-by: Michael Witten
---
parse.y |2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/parse.y b/parse.y
index 55874aa..38c4330 100644
--- a/parse.y
+++ b/parse.y
@@ -4524,7 +4524,7 @@ read_token_word (character)
ttrans = ttok;
}
- RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen + 2,
+ RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen + 1,
token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
strcpy (token + token_index, ttrans);
--
1.7.4.22.g14b16.dirty
[PATCH 1/6] Bug: extglob: Fix off-by-one assignment in read_token_word()
This fixes a parsing bug that would probably never be of trouble in
practice; it is an edge case of edge cases. Nevertheless, bash should
be able to handle it without corrupting memory.
Consider the following:
$ TOKEN_DEFAULT_INITIAL_SIZE=496 # parse.y:1282
$ n=TOKEN_DEFAULT_INITIAL_SIZE # notational convenience
$ unset BASH_ENV
$ { for ((i=0;i bash -O extglob
That is, giving bash input composed of TOKEN_DEFAULT_INITIAL_SIZE `a'
characters followed by `*(a)' followed by CTLNUL followed by EOF, all
while the extglob shell option is in effect.
The environment variable `BASH_ENV' is unset to inhibit the reading of any
startup script, which might otherwise alter the particular case that would
trigger the bug (due to changes in buffer allocations [see below]).
The bash program eventually calls read_token_word(), which, under the
given input, is essentially equivalent to the following:
#define TOKEN_DEFAULT_INITIAL_SIZE 496
#define TOKEN_DEFAULT_GROW_SIZE 512
static int
read_token_word (character)
int character;
{
/* The value for YYLVAL when a WORD is read. */
WORD_DESC *the_word;
/* Index into the token that we are building. */
int token_index;
if (token_buffer_size < TOKEN_DEFAULT_INITIAL_SIZE)
token = (char *)xrealloc (token, token_buffer_size =
TOKEN_DEFAULT_INITIAL_SIZE);
token_index = 0;
for (;;)
{
if (character == EOF)
goto got_token;
/* Parse a ksh-style extended pattern matching specification. */
if (character == '*')
{
/** simulate peek_char and parse_matched_pair() **/
shell_getc (1);
shell_getc (1);
/*/
token[491] = '*';
token[492] = '(';
strcpy (token + 493, "a)");
token_index = 495;
goto next_character;
}
if (character == CTLESC || character == CTLNUL)
{
token[495] = CTLESC;
token_index=496; /* assignment below: token[497] = CTLNUL; */
}
token[token_index++] = character;
RESIZE_MALLOCED_BUFFER (token, token_index, 1, token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
next_character:
character = shell_getc (1);
} /* end for (;;) */
got_token:
token[token_index] = '\0';
the_word = (WORD_DESC *)xmalloc (sizeof (WORD_DESC));
the_word->word = (char *)xmalloc (1 + token_index);
the_word->flags = 0;
strcpy (the_word->word, token);
yylval.word = the_word;
return WORD;
}
Before the `*' is reached, the `for(;;)' loop looks essentially like this:
for (;;)
token[token_index++] = shell_getc (1);
That is, the `a' characters are copied into the token array.
When the `*' is read, `token_index' has the value:
TOKEN_DEFAULT_INITIAL_SIZE-5 = 491
and the following code is executed:
if (character == '*')
{
/** simulate peek_char and parse_matched_pair() **/
shell_getc (1);
shell_getc (1);
/*/
token[491] = '*';
token[492] = '(';
strcpy (token + 493, "a)");
token_index = 495;
goto next_character;
}
The value of `token_index' represents the number of characters that have
already been stored in `token'; in this case, 495 characters have been
stored. As per the `goto', the next character is read:
next_character:
character = shell_getc (1);
This reads the `\177' character, CTLNUL. which is handled with this code:
if (character == CTLESC || character == CTLNUL)
{
token[495] = CTLESC;
token_index=496; /* assignment below: token[497] = CTLNUL; */
}
token[token_index++] = character;
Thus, the last assignment is equivalent to:
token[496] = character;
token_index=497;
which is, of course, one past the end of `token', because there are only
496 indicies for `token' according to its current allocation (the maximum
`token' index is 495).
The most "conservative" fix is the one presented in this patch:
Allocate more space than is necessary by imposing
an unnecessarily strict requirement, thereby "masking"
the bug.
In my opinion, this is wrongheaded, but it only requires the change of
one character and it's exactly what other code paths in read_token_word()
[currently] do, probably by mistake. :-P
Signed-off-by: Michael Witten
---
parse.y |2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/parse.y b/parse.y
index b5c94e7..6c095f5 100644
--- a/parse.y
+++ b/parse.y
@@ -4431,7 +4431,7 @@ read_token_word (character)
po
[PATCH 6/6] Clean: Remove unnecessary backslashes (line continuation)
Signed-off-by: Michael Witten
---
parse.y |6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/parse.y b/parse.y
index b61c4d0..eaae077 100644
--- a/parse.y
+++ b/parse.y
@@ -4453,7 +4453,7 @@ read_token_word (character)
{
peek_char = shell_getc (1);
/* $(...), <(...), >(...), $((...)), ${...}, and $[...] constructs */
- if MBTEST(peek_char == '(' || \
+ if MBTEST(peek_char == '(' ||
((peek_char == '{' || peek_char == '[') && character == '$'))
/* ) ] } */
{
if (peek_char == '{') /* } */
@@ -4654,8 +4654,8 @@ got_token:
is a `<', or a `&', or the character which ended this token is
a '>' or '<', then, and ONLY then, is this input token a NUMBER.
Otherwise, it is just a word, and should be returned as such. */
- if MBTEST(all_digit_token && (character == '<' || character == '>' || \
- last_read_token == LESS_AND || \
+ if MBTEST(all_digit_token && (character == '<' || character == '>' ||
+ last_read_token == LESS_AND ||
last_read_token == GREATER_AND))
{
if (legal_number (token, &lvalue) && (int)lvalue == lvalue)
--
1.7.4.22.g14b16.dirty
[PATCH 5/6] Clean: Remove unnecessary xmalloc()/strcpy()
Signed-off-by: Michael Witten
---
parse.y |8 ++--
1 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/parse.y b/parse.y
index a12c4d0..b61c4d0 100644
--- a/parse.y
+++ b/parse.y
@@ -4538,17 +4538,13 @@ read_token_word (character)
shell's single-character parameter expansions, and set flags.*/
else if MBTEST(character == '$' && peek_char == '$')
{
- ttok = (char *)xmalloc (3);
- ttok[0] = ttok[1] = '$';
- ttok[2] = '\0';
RESIZE_MALLOCED_BUFFER (token, token_index, 2,
token_buffer_size,
TOKEN_DEFAULT_GROW_SIZE);
- strcpy (token + token_index, ttok);
- token_index += 2;
+ token[token_index++] = '$';
+ token[token_index++] = '$';
dollar_present = 1;
all_digit_token = 0;
- FREE (ttok);
goto next_character;
}
else
--
1.7.4.22.g14b16.dirty
[PATCH 4/6] Clean: More direct coupling between assignment and allocation
In parse_token_word(), RESIZE_MALLOCED_BUFFER() was used in such a way as to ensure that there is always enough room for the final '\0' character and for a potential unquoted CTLESC or CTLNUL character (which causes an additional CTLESC to be appended to `token'). Now, by introducing a call to RESIZE_MALLOCED_BUFFER() before the extra CTLESC is appended to the `token' array, and by moving the last call to RESIZE_MALLOCED_BUFFER() from below to above the last token character assignment, this contorted connection between allocation and assignment has been reduced to a rule that is more familiar: Ensure space is allocated before attempting an assignment to that space. As an added benefit, the last RESIZE_MALLOCED_BUFFER() condition has been automatically relaxed now that a potentially unquoted CTLESC or CTLNUL character need not be speculated about; this particular benefit has been extended to the other invocations of RESIZE_MALLOCED_BUFFER() by reducing the `room' argument by 1 (in fact, this argument can now be interpreted as "the number of characters that will be immediately appended to `token'", which is a much more natural and transparent understanding). Signed-off-by: Michael Witten --- parse.y | 36 +--- 1 files changed, 21 insertions(+), 15 deletions(-) diff --git a/parse.y b/parse.y index 38c4330..a12c4d0 100644 --- a/parse.y +++ b/parse.y @@ -4382,7 +4382,7 @@ read_token_word (character) pop_delimiter (dstack); if (ttok == &matched_pair_error) return -1; /* Bail immediately. */ - RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 2, + RESIZE_MALLOCED_BUFFER (token, token_index, 1 + ttoklen, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); token[token_index++] = character; strcpy (token + token_index, ttok); @@ -4408,7 +4408,7 @@ read_token_word (character) pop_delimiter (dstack); if (ttok == &matched_pair_error) return -1; /* Bail immediately. */ - RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 2, + RESIZE_MALLOCED_BUFFER (token, token_index, 1 + ttoklen, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); token[token_index++] = character; strcpy (token + token_index, ttok); @@ -4431,7 +4431,7 @@ read_token_word (character) pop_delimiter (dstack); if (ttok == &matched_pair_error) return -1; /* Bail immediately. */ - RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 3, + RESIZE_MALLOCED_BUFFER (token, token_index, 2 + ttoklen, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); token[token_index++] = character; @@ -4473,7 +4473,7 @@ read_token_word (character) ttok = parse_matched_pair (cd, '[', ']', &ttoklen, 0); if (ttok == &matched_pair_error) return -1; /* Bail immediately. */ - RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 3, + RESIZE_MALLOCED_BUFFER (token, token_index, 2 + ttoklen, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); token[token_index++] = character; @@ -4524,7 +4524,7 @@ read_token_word (character) ttrans = ttok; } - RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen + 1, + RESIZE_MALLOCED_BUFFER (token, token_index, ttranslen, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); strcpy (token + token_index, ttrans); @@ -4541,7 +4541,7 @@ read_token_word (character) ttok = (char *)xmalloc (3); ttok[0] = ttok[1] = '$'; ttok[2] = '\0'; - RESIZE_MALLOCED_BUFFER (token, token_index, 3, + RESIZE_MALLOCED_BUFFER (token, token_index, 2, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); strcpy (token + token_index, ttok); @@ -4566,7 +4566,7 @@ read_token_word (character) ttok = parse_matched_pair (cd, '[', ']', &ttoklen, P_ARRAYSUB); if (ttok == &matched_pair_error) return -1; /* Bail immediately. */ - RESIZE_MALLOCED_BUFFER (token, token_index, ttoklen + 2, + RESIZE_MALLOCED_BUFFER (token, token_index, 1 + ttoklen, token_buffer_size, TOKEN_DEFAULT_GROW_SIZE); token[token_index++] = character; @@ -4584,7 +4584,7 @@ read_token_word (c
Re: [PATCH 1/6] Bug: extglob: Fix off-by-one assignment in read_token_word()
On looking over this patch, I found a number of `bugs' in my description,
but they don't change the conclusions. I should have proofread more carefully.
On Sat, 26 Feb 2011 11:48:06 -0500, Michael Witten wrote:
> /** simulate peek_char and parse_matched_pair() **/
> shell_getc (1);
> shell_getc (1);
> /*/
That's actually missing one `shell_getc (1);' statement; it should be:
/* Simulate peek_char */
shell_getc (1);
/* Simulate parse_matched_pair() */
shell_getc (1);
shell_getc (1);
> if (character == CTLESC || character == CTLNUL)
> {
> token[495] = CTLESC;
> token_index=496; /* assignment below: token[497] = CTLNUL; */
> }
That comment should read `496' rather than `497':
/* assignment below: token[496] = CTLNUL; */
Fortunately, the non-code description that followed got it right.
Sorry for the confusion.
Re: [PATCH 2/6] Bug: shellexp: Fix off-by-one assignment in read_token_word()
On looking over this patch, I found a number of `bugs' in my description,
but they don't change the conclusions. I should have proofread more carefully.
On Sat, 26 Feb 2011 09:48:06 -0700, Michael Witten wrote:
> /** simulate peek_char and parse_matched_pair() **/
> shell_getc (1);
> shell_getc (1);
> /*/
That's actually missing one `shell_getc (1);' statement; it should be:
/* Simulate peek_char */
shell_getc (1);
/* Simulate parse_matched_pair() */
shell_getc (1);
shell_getc (1);
> if (character == CTLESC || character == CTLNUL)
> {
> token[495] = CTLESC;
> token_index=496; /* assignment below: token[497] = CTLNUL; */
> }
That comment should read `496' rather than `497':
/* assignment below: token[496] = CTLNUL; */
Fortunately, the non-code description that followed got it right.
Also, it would have been helpful if these lines had been included, as
in the previous patch:
#define TOKEN_DEFAULT_INITIAL_SIZE 496
#define TOKEN_DEFAULT_GROW_SIZE 512
Sorry for the confusion.
Command substition failure?
Could somebody please tell me what's going on here?
First, here is a minimal example:
LINE|
| $ echo "$(echo '"' | awk '{sub(/a/,$0)}')"
0001| awk: cmd. line:1: sub(/a/
0002| awk: cmd. line:1:^ unexpected newline or end of string
0003| awk: $0)
0004| awk: ^ syntax error
0005|
Where is this newline coming from? What happened to the comma?
Look what happens if we add the action `NR=1'; notice that the error
message seems to show that the `{' has been removed:
LINE|
| $ echo "$(echo '"' | awk 'NR==1 {sub(/a/,$0)}')"
0001| awk: cmd. line:1: NR==1 sub(/a/
0002| awk: cmd. line:1: ^ unexpected newline or end of string
0003| awk: NR==1 $0)
0004| awk: ^ syntax error
0005|
Interestingly, the whole thing seems to become syntactically correct
when I put a space between the `{' and the `sub' (and experimentation
shows that it is working):
LINE|
| $ echo "$(echo '"' | awk 'NR==1 { sub(/a/,$0)}')"
0001|
The problem also goes away when I just remove the outer quotes:
LINE|
| $ echo $(echo '"' | awk 'NR==1 {sub(/a/,$0)}')
0001|
However, it should be noted that the outer quotes should not have an
effect on any characters within the command substition `$(...)';
consider what the POSIX shell documentation here:
http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_02_03
says:
Enclosing characters in double-quotes ("") shall preserve the
literal value of all characters within the double-quotes, with
the exception of the characters dollar sign, backquote, and
backslash, as follows:
$
The dollar sign shall retain its special meaning
introducing parameter expansion (see Parameter
Expansion), a form of command substitution (see
Command Substitution), and arithmetic expansion
(see Arithmetic Expansion).
The input characters within the quoted string
that are also enclosed between "$(" and the
matching ')' shall not be affected by the
double-quotes, but rather shall define that
command whose output replaces the "$(...)" when
the word is expanded. The tokenizing rules in
Token Recognition , not including the alias
substitutions in Alias Substitution , shall be
applied recursively to find the matching ')'.
...
...
In particular:
The input characters within the quoted string
that are also enclosed between "$(" and the
matching ')' shall not be affected by the
double-quotes, but rather shall define that
command whose output replaces the "$(...)" when
the word is expanded.
>From what I can tell, bash follows the same semantics; the manual here:
$ info '(bash)Double Quotes'
or here:
http://www.gnu.org/software/bash/manual/html_node/Double-Quotes.html#Double-Quotes
says:
3.1.2.3 Double Quotes
.
Enclosing characters in double quotes (`"') preserves the
literal value of all characters within the quotes, with the
exception of `$', ``', `\', and, when history expansion is
enabled, `!'. The characters `$' and ``' retain their special
meaning within double quotes (*note Shell Expansions::).
and the manual here:
$ info '(bash)Command Substitution'
or here:
http://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html#Shell-Expansions
says:
3.5.4 Command Substitution
--
...
... When using the `$(COMMAND)' form, all characters between
the parentheses make up the command; none are treated
specially.
...
If the substitution appears within double quotes, word
splitting and filename expansion are not performed on the
results.
Could somebody please tell me what's going on here?
Sincerely,
Michael Witten
Re: Command substition failure?
On Sat, Apr 2, 2011 at 08:20, Andreas Schwab wrote: > Brace expansion. HAH! :-D Damn :-/
Re: Command substition failure?
On Sat, 02 Apr 2011 15:20:23 +0200, Andreas Schwab wrote:
>> Could somebody please tell me what's going on here?
>
> Brace expansion.
>
> $ set -x
> $ echo "$(echo '"' | awk '{sub(/a/,$0)}')"
> ++ echo '"'
> ++ awk 'sub(/a/'
> awk: cmd. line:1: sub(/a/
> awk: cmd. line:1:^ unexpected newline or end of string
> ++ echo '"'
> ++ awk '$0)'
> awk: $0)
> awk: ^ syntax error
> + echo '' ''
So, basically, bash is viewing this:
echo "$(echo '"' | awk '{sub(/a/,$0)}')"
as:
echo PREAMBLE{x,y}POSTSCRIPT
where PREAMBLE is the string:
"$(echo '"' | awk '
and the `x' is the string:
sub(/a/
and the `y' is the string:
$0)
and the POSTSCRIPT is the string:
')"
Thus, bash expands the line to the following:
echo "$(echo '"' | awk 'sub(/a/')" "$(echo '"' | awk '$0)')"
Bash runs the first substitution command:
echo '"' | awk 'sub(/a/'
which fails with:
awk: cmd. line:1: sub(/a/
awk: cmd. line:1:^ unexpected newline or end of string
and produces an empty string as output.
Bash runs the second substitution command:
echo '"' | awk '$0)'
which fails with:
awk: $0)
awk: ^ syntax error
and produces an empty string as output.
Thus, the whole line becomes:
echo "" ""
which outputs 2 characters: A space followed by a newline.
Phew!
OK, now, I think that's a bug!
The bash manual here:
info '(bash)Brace Expansion'
or here:
http://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html#Brace-Expansion
says:
3.5.1 Brace Expansion
-
...
A sequence expression takes the form `{X..Y[..INCR]}', where X and Y
are either integers or single characters, and INCR, an optional
increment, is an integer...
Brace expansion is performed before any other expansions,
and any characters special to other expansions are preserved
in the result. It is strictly textual. Bash does not apply any
syntactic interpretation to the context of the expansion or
the text between the braces. To avoid conflicts with parameter
expansion, the string `${' is not considered eligible for brace
expansion.
A correctly-formed brace expansion must contain unquoted
opening and closing braces, and at least one unquoted comma
or a valid sequence expression. Any incorrectly formed brace
expansion is left unchanged.
A { or `,' may be quoted with a backslash to prevent its
being considered part of a brace expression. To avoid conflicts
with parameter expansion, the string `${' is not considered
eligible for brace expansion.
...
In particular:
A correctly-formed brace expansion must contain unquoted
opening and closing braces, and at least one unquoted comma
or a valid sequence expression. Any incorrectly formed brace
expansion is left unchanged.
Now, my email with `Message-ID':
ecdf0813-3e67-4bb7-bfaf-6bfc12336c31-mfwit...@gmail.com
shows:
The input characters within the quoted string
that are also enclosed between "$(" and the
matching ')' shall not be affected by the
double-quotes, but rather shall define that
command whose output replaces the "$(...)" when
the word is expanded.
and this:
... When using the `$(COMMAND)' form, all characters between
the parentheses make up the command; none are treated
specially.
So, there should be no brace expansion in this case, because
the entire brace construct is quoted:
'{sub(/a/,$0)}'
Thus, bash has a bug.
My guess is the nature of the problem is that the combination
of the outer-most quotes (which would render most characters
as literal) and the command substitution (which in some sense
is probably parsed in a `top-level' context) works to confuse
the brace-expansion logic.
Sincerely,
Michael Witteng
Re: Bash source repository
On Mon, May 30, 2011 at 02:18, Bradley M. Kuhn wrote: > Chet Ramey wrote on 2009-11-02: >> Jari Aalto was setting up a git repository of current and older bash >> versions on savannah. I'll keep him up to date with public versions >> of bash > > Bob Proulx wrote on 2009-11-02: >>> It looks like it has just recently been partially implemented. >>> http://git.savannah.gnu.org/cgit/bash.git >>> But only at the major release points. No patches have been applied. >>> But the available history seems to all be there. > > Jan Schampera wrote on 2009-11-22: The official patches should be there as individual commits. Though, I admit it's not a small amount of work to do all that for the past releases. > > Chet Ramey wrote on 2009-11-23: >> All the release tarfiles and official patches dating back to at least >> bash-1.14 are on ftp.gnu.org in pub/gnu/bash (though the form of the >> patches has changed). This is at least a 15-year chronology. > > Eric Blake wrote on 2010-03-13: > the savannah repository is tracking only public releases and > patches (and could use some TLC in reflecting recent patches, or > even being modified to show patches in finer granularity). > > It's been two years since this discussion began, and a year since this > thread was alive on the mailing list. > > I did some research this weekend and collected all the old versions of Bash > I could find. I've carefully organized all this into a repository, > making the commits as find-grained as possible given what's available in > the permanent public record. My work is available at: > https://gitorious.org/bash/bash > > I humbly suggest that http://git.savannah.gnu.org/cgit/bash.git be > replaced with this repository above that I've created. The new > repository contains everything that the current Savannah one does, but I > put much more effort into making commits fine-grained, rather than > merely importing the public releases blindly. (For example, I did 'git > mv' where it was obvious a move occurred, so that changes in file > movements were properly tracked historically). > > I'm also willing to maintain the git repository going forward, > both on savannah and gitorious. Would folks like me to do this? > > Finally, I wish I had older data, particularly versions before 1.05, and > also intermediate releases before 1.14. If anyone has any such > versions, please let me know, and I'll rebase the master branch on top > of them. As I recall, the diffs for versions prior to the 1.14.3 tarball produce incorrect results (somewhat silently), so those commits are essentially going to be junk. In any case, it is my opinion that all further development should take place through a public, distributed repository such as the one you have created - regardless of Chet's objections. If Chet is not on board, then I suggest working to establish a new official maintainer. Part of being a maintainer is using tools that benefit and properly recognize the contributors; private repos, ancient patch formats, and attribution through [only] ChangeLogs are unnacceptable ways to work these days.
Re: Bash source repository
On Mon, May 30, 2011 at 03:09, Bradley M. Kuhn wrote: > > Michael Witten replied a few minutes ago: >> it is my opinion that all further development should take place through >> a public, distributed repository such as the one you have created - >> regardless of Chet's objections. > > It seems that two different conversations have been conflated here. My > goal was to simply finish what Jari was seeking to do with > http://git.savannah.gnu.org/cgit/bash.git two years ago. I believe I have > succeed in that: specifically, putting all the historical information > currently available into a single git repository. I'm also offering to > maintain the same indefinitely. > > What you've raised another issue entirely that perhaps should be > discussed, but I hope it won't be conflated with what I was trying to do > by creating the Git repository. It's true that perhaps this Git > repository I created could be used as a basis to explore what you are > proposing, but the two issues are ultimately only tangentially related. I take full responsibility for accusations of mutiny :-)
Re: Bash source repository
On Mon, May 30, 2011 at 05:06, Ben Pfaff wrote: > "Bradley M. Kuhn" writes: > >> The new repository contains everything that the current >> Savannah one does, but I put much more effort into making >> commits fine-grained, rather than merely importing the public >> releases blindly. (For example, I did 'git mv' where it was >> obvious a move occurred, so that changes in file movements were >> properly tracked historically). > > Git doesn't track file movement in any way, it just tracks file > content. It is "git diff" and other commands for viewing history > that figure out that a file moved, based on the change that > occurred. It is a waste of time to use "git mv" if some other > way is easier. More importantly, it is best to move files and commit the results without any other additional content changes occuring simultaneously, so that commands like "git diff" may figure this out more readily. Perhaps (and hopefully) Bradley meant that file moves were separated from any other content changes that might otherwise have occurred simultaneously. P.S. Ben, it is generally a good idea to maintain the `Cc' list unless explicitly asked.
Re: Bash source repository
On Thu, Jun 2, 2011 at 17:00, Bradley M. Kuhn wrote: > I'd suggest that we keep the master branch only to track the history of > releases and officially released patches as Chet posts them, and then we > can use separate branches for individual developers who want to use Git. > What do you think of this idea? There should perhaps be development branches (like `pu' and `next' in git's own development model[0]), but I don't like the idea of having separate PUBLIC branches for individual developers; they can have their own repositories with their own branches. I would also like to stress that whatever ends up in `master' should not be a surprise (that is, the result of private work). There should be a public progression through development branches and then ultimately a merge (or at least a cherry-picking) into the master branch. [0] http://git.kernel.org/?p=git/git.git;a=blob_plain;f=MaintNotes;hb=9f1017f6414d81e9dc561c2cc627b6fcd05d57b2
Re: Bash source repository
>On Thu, Jun 2, 2011 at 17:00, Bradley M. Kuhn wrote: >> I'd suggest that we keep the master branch only to track the history of >> releases and officially released patches as Chet posts them, and then we >> can use separate branches for individual developers who want to use Git. >> What do you think of this idea? > > There should perhaps be development branches (like `pu' and `next' in > git's own development model[0]), but I don't like the idea of having > separate PUBLIC branches for individual developers; they can have > their own repositories with their own branches. > > I would also like to stress that whatever ends up in `master' should not be > a surprise (that is, the result of private work). There should be a public > progression through development branches and then ultimately a merge (or at > least a cherry-picking) into the master branch. > > [0] > http://git.kernel.org/?p=git/git.git;a=blob_plain;f=MaintNotes;hb=9f1017f6414d81e9dc561c2cc627b6fcd05d57b2 Of course, in this scenario, that public progression may include the refinement of code via email patches rather than commits on development branches, and then the actual commit (perhaps directly on `master' in the case of trivial changes) would simply be formed from the finalized patch (including any appropriate authorship information).
Re: Bash source repository
On Fri, Jun 3, 2011 at 15:27, Bradley M. Kuhn wrote: > Sorry, I was imprecise in my wording in my email yesterday. By "use > separate branches for individual developers", I meant that "branches > would be created for those developers who wanted to develop publicly in > a Git repository". Such developers should use their own repositories.
Re: A Feature Request for History
On Mon, Jun 13, 2011 at 17:10, Bradley M. Kuhn wrote: > I have all my bash history going back to > 2003-10-15 accessible to me. Why?
Re: Bug, or someone wanna explain to me why this is a POSIX feature?
On Mon, Aug 8, 2011 at 18:44, Linda Walsh wrote:
>
> I was testing functions in my shell, I would cut/paste,
> thing is, with each past, I'd get my dir listed (sometimes multiple times)
> on each line entered.
>
> Now I have:
> shopt:
> no_empty_cmd_completion on
>
> i.e. it's not supposed to expand an empty line
>
> but type in
> function foo {
> return 1
> }
>
> When I hit tab it lists out all the files in my dir -- which
> explains why when I cut/paste, any tab-indented line will list
> out the dir, and if it is multiply indented, it will be listed
> once for each indent level!
>
> Now I'm sure someone will come up and tell me how POSIX this or that.
> But maybe POSIX needs to be retired if this is some required feature!
>
> When I tell it not to complete an empty line, and hit tab at the
> beginning of a line -- even inside a function def, I don't need my
> dir listed!
>
> So is this a bug, or can we retire POSIX??
>
> running bash 4.1.10(1)-release (x86_64-suse-linux-gnu)
> (released version in Suse 11.4).
Don't use tab characters on the interactive command line.
I'm not sure what you're mumbling about POSIX.
Re: Bug, or someone wanna explain to me why this is a POSIX feature?
On Mon, 8 Aug 2011 21:40:39 +0200, Davide Brini wrote:
> On Mon, 8 Aug 2011 21:14:50 +0200, Davide Brini wrote:
>
>> In fact, you could do the same thing with
>>
>> foo() { # hit tab here
>>
>> and I'm sure you wouldn't consider that an empty line.
>
> I have to take that back: it looks like bash treats the above differently
> depending on whether enter was pressed or not:
>
> foo() { # hit tab here
> Display all 2138 possibilities? (y or n)
>
> foo() { # hit enter here
> > # hit tab here
> Display all 112 possibilities? (y or n)
>
> The latter only attemps completion from names in the current directory.
>
> On the other hand, with no_empty_cmd_completion set, no completion at all is
> attempted in the first case, while the second case still attempts completion
> from local names.
This behavior does indeed seem buggy.
>From the following:
$ info '(bash)The Shopt Builtin'
we have:
`no_empty_cmd_completion'
If set, and Readline is being used, Bash will not attempt to
search the `PATH' for possible completions when completion is
attempted on an empty line.
Now, firstly, there is possibly a conceptual conflict between `empty_cmd' and
`empty line', but seeing as virtually everything that can be input is defined
as a command of some sort or another, maybe there's no problem.
At any rate, the input:
$ foo() { # hit tab here, right after the space character.
is not only a non-empty line, but it is also an incomplete function definition
*command*, according to the nomenclature and shell grammar specified by POSIX:
Base Definitions Issue 7, IEEE Std 1003.1-2008
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05
Hence, it seems to me that completion should be attempted regardless of
whether `no_empty_cmd_completion' is set; of course, searching `PATH'
for a completion would likely result in no available completions anyway.
Moreover, given that searching `PATH' is mentioned, one could easily
imagine that the completion in question is that of the names of simple
commands. In that capacity, there is no simple command to complete (in
other words, there is an `empty' command to complete). In this case, then,
it is correct for there to be no completion when `no_empty_cmd_completion'
is set, and it is correct for there to be a list of all available commands
when `no_empty_cmd_completion' is unset (which would appear to be the
current behavior).
Now, what about the continuation of the command line?
$ foo() {# hit enter here
> #hit tab here.
According to the same POSIX documentation:
PS2
Each time the user enters a prior to completing a
command line in an interactive shell, the value of this variable
shall be subjected to parameter expansion and written to standard
error. The default value is "> " . This volume of POSIX.1-2008
specifies the effects of the variable only for systems supporting
the User Portability Utilities option.
and together with bash's shopt options `cmdhist' and `lithist', it would
seem that bash [mostly] treats `multi-line' commands (as bash calls them)
as one large single command line. Hence, a tab on a continuation line might
be expected to invoke only file name completion (which does happen, and is
indeed orthogonal to the setting of `no_empty_cmd_completion').
Unfortunately, such behavior is still inconsistent with the current completion
behavior as described above, because bash basically converts an interactive
newline in a compound command to a semicolon, thereby initiating another
simple command.
For instance, regardless of `no_empty_cmd_completion', the following:
$ foo() { echo a; ech#tab
completes `echo' on my system.
When the option is set, the following:
$ foo() { echo a; #tab
doesn't attempt any completion (visibly, anyway). However, when the option is
UNset, completion is attempted based on all available commands (thousands).
So, based on the existing semantics and descriptions, one would expect
$ foo() {#enter
> #tab
to make no completion attempt when the option is set, and to make an attempt
based on all available command names when unset, but not the current behavior,
which is to suggest file name completions.
Now, of course, completion is much more complex than just command name and
file name completion, and I think it could be improved to be more useful.
For instance, according to shell grammar, any compound command is acceptable
as the body of a function, so that we could have this:
$ i=0
$ foo()
> while [ $i -lt 10 ]; do i=$((i + 1)); echo $i; done
$ foo#enter
1
2
3
4
5
6
7
8
9
10
$ bar()
> if [ $i -eq 10 ]; then echo 11; fi
$ bar
11
Thus, for instance, completing function definition commands should produce
results like this:
$ foo() #tab
( { case ifwhile until
Also, why doesn't `shopt' have option completion? etc.
In any case, e
Re: Bug, or someone wanna explain to me why this is a POSIX feature?
On Mon, 08 Aug 2011 15:56:30 -0700, Linda Walsh wrote:
> Michael Witten wrote:
>>
>> In any case, even if `no_empty_cmd_completion' were to behave as Linda
>> expected, her tabs would still get eaten when pasted on the interactive
>> command line.
>
> Which is what Linda expects... the function definition wouldn't
> hold spaces or tabs or whitespace unless it is quoted.
>
> She just doesn't expect, when pasting a function that is from a
> source file into her shell, scads of output that is unexpected,
> unwanted, and more than a bit confusing.
>
> Fortunately, if you have the function 'well formed' and 'well
> defined', it seems to make no difference as far as defining the
> actual function, BUT having all the names of my current dir
> blatted out for each tab char is a pain.
>
> So don't assume or infer that Linda wanted the tabs included in
> bash's internal function representation. She just didn't want the
> blather.
>
> Reasonable? Or is someone going to tell me why blather is a
> desired and wanted feature (under one standard or another! ;-))...
Reasonable, but naive.
Interactive bash can complete variable names, even when they are quoted.
On my system, these variables are in the environment:
TERM
TERMINFO
Now, when I try to paste the following ("$TERM$TERMINFO"):
foo() { echo "$TERM $TERMINFO"; }
this is the result in my terminal:
mfwitten$ foo() { echo "$TERM$TERMINFO"; }
which is not what I wanted; if I save that same line in a file
and then run a shell on that file, I get the expected output.
Similarly, if I paste the following ("$TERM$TERMINFO"):
foo() { echo "$TERM $TERMINFO"; }
the result is this in my terminal:
mfwitten$ foo() { echo "$TERM
$TERM $TERMINFO
mfwitten$ foo() { echo "$TERM$TERMINFO"; }
Here, the 2 tabs ask bash/readline to print the possible completions
for `$TERM' (hence the scads of output) and then the rest of the
pasted input gets written in as before.
If you want to work with your functions interactively, then just
save them to a file (like `/tmp/s') and source them in:
mfwitten$ source /tmp/s
mfwitten$ foo
rxvt-unicode-256color /usr/share/terminfo
or, equivalently:
mfwitten$ . /tmp/s
mfwitten$ foo
rxvt-unicode-256color /usr/share/terminfo
This is much more efficient than copying and pasting when you need
to do a lot of tinkering.
If you insist on copying and pasting into an interactive terminal,
then never ever use tab characters anywhere.
Re: Syntax Question...
On Sat, Aug 13, 2011 at 23:41, Linda Walsh wrote:
> ${#${!name}[*]}
> bash: ${#${!name}[*]}: bad substitution
It's probably what you're trying to avoid, but you'll probably have to
construct and then eval the right code by hand:
$(eval "echo \${#$name[*]}")
Re: Syntax Question...
On Sun, Aug 14, 2011 at 00:49, Dennis Williamson
wrote:
> On Sat, Aug 13, 2011 at 6:41 PM, Linda Walsh wrote:
>>
>>
>>
>> I want to have an array of 'names'.
>>
>> given a name, "X", I had a prefix, _p_, so have _p_X,
>> I want to access / manipulate it as an array.
>>
>> so given I pass in a name in a var, I have to use 'indirect' syntax:
>>
>> ${!name}
>>
>> But that doesn't seem to play well with array syntax:
>>
>> ${#${!name}[*]}
>> bash: ${#${!name}[*]}: bad substitution
>>
>> What syntax would I use for this?
>
> Please read BashFAQ/006: http://mywiki.wooledge.org/BashFAQ/006
"no force in the universe can put NUL bytes into shell strings usefully"
Ain't that the goddamn Truth!
Re: Syntax Question...
On Sun, Aug 14, 2011 at 04:55, Linda Walsh wrote:
>
>
>
> ` Michael Witten wrote:
>>
>> On Sat, Aug 13, 2011 at 23:41, Linda Walsh wrote:
>>
>>>
>>> ${#${!name}[*]}
>>> bash: ${#${!name}[*]}: bad substitution
>>>
>>
>> It's probably what you're trying to avoid, but you'll probably have to
>> construct and then eval the right code by hand:
>>
>> $(eval "echo \${#$name[*]}")
>>
>
> bingo. (sigh)
>
> I refactored and am essentially using spaces to store the values in
> a string, then when I want to check the, I throw them into an array and
> manip.
> e.g. (not showing all supporting code)
>
> declare -axr _plst_t='_plist_'
> declare -axl defined_PLists=()
>
> add things to the list...
>
> plist="$_plst_t$plist"
> local -al plist_deps=( $plist )
>
> {search for value in list, ...]
>
> unless ((found)) ; then
> plist_deps[${#plist_deps[*]}]="$subdep"
> eval "$plist=( "$plist_deps[@]" )"
> fi
You want this:
unless ((found)) ; then
plist_deps+=("$subdep")
eval $plist='("${plist_deps[@]}")'
fi
Of course, isn't the variable `plist' supposed to be a reference to
your *string* variable that holds a space-separated list of items? In
that case, the last line should be:
IFS=' ' eval $plist='"${a[*]}"'
Re: Syntax Question...
On Sun, Aug 14, 2011 at 05:56, Michael Witten wrote:
> On Sun, Aug 14, 2011 at 04:55, Linda Walsh wrote:
>>
>>
>>
>> ` Michael Witten wrote:
>>>
>>> On Sat, Aug 13, 2011 at 23:41, Linda Walsh wrote:
>>>
>>>>
>>>> ${#${!name}[*]}
>>>> bash: ${#${!name}[*]}: bad substitution
>>>>
>>>
>>> It's probably what you're trying to avoid, but you'll probably have to
>>> construct and then eval the right code by hand:
>>>
>>> $(eval "echo \${#$name[*]}")
>>>
>>
>> bingo. (sigh)
>>
>> I refactored and am essentially using spaces to store the values in
>> a string, then when I want to check the, I throw them into an array and
>> manip.
>> e.g. (not showing all supporting code)
>>
>> declare -axr _plst_t='_plist_'
>> declare -axl defined_PLists=()
>>
>> add things to the list...
>>
>> plist="$_plst_t$plist"
>> local -al plist_deps=( $plist )
>>
>> {search for value in list, ...]
>>
>> unless ((found)) ; then
>> plist_deps[${#plist_deps[*]}]="$subdep"
>> eval "$plist=( "$plist_deps[@]" )"
>> fi
>
> You want this:
>
> unless ((found)) ; then
> plist_deps+=("$subdep")
> eval $plist='("${plist_deps[@]}")'
> fi
>
> Of course, isn't the variable `plist' supposed to be a reference to
> your *string* variable that holds a space-separated list of items? In
> that case, the last line should be:
>
> IFS=' ' eval $plist='"${a[*]}"'
Woops! That last line should read:
IFS=' ' eval $plist='"${plist_deps[*]}"'
Of course, if each list item is guaranteed never to contain a
word-delimiting character, then the quoting may be simplified.
In the first case:
unless ((found)) ; then
plist_deps+=($subdep)
eval $plist='(${plist_deps[@]})'
fi
and in the second case (where each list item is also guaranteed never
to contain a space character):
unless ((found)) ; then
plist_deps+=($subdep)
IFS=' ' eval $plist=\${plist_deps[*]}
fi
Re: Typo in po/de.po
sorry, i forgot something. 2531c2531 < "Führt eine eingebeute Shell Funktionen aus.\n" --- > "Führt eine eingebaute Shell Funktionen aus.\n" 2539c2539 < "Der Rückgabewert der eingebauten Schellfunkrion oder Falsch, wenn\n" --- > "Der Rückgabewert der eingebauten Shell Funktion oder Falsch, wenn\n"
Typo in po/de.po
Hello, Hope i am right here with that. Nothing special, but noticed this just now. Patch is attached. I am not registered in this list, so please cc replys to me. Greetings, Michael 2539c2539 < "Der Rückgabewert der eingebauten Schellfunkrion oder Falsch, wenn\n" --- > "Der Rückgabewert der eingebauten Shell Funktion oder Falsch, wenn\n"
Re: Append history from each shell upon exiting GNU Screen
On Mon, Sep 12, 2011 at 14:19, Roger wrote: >> On Mon, Sep 12, 2011 at 08:36:07AM -0400, Greg Wooledge wrote: >>On Sun, Sep 11, 2011 at 11:23:48PM -0800, Roger wrote: >>> When using GNU Screen (or other terminal multiplexer), I noticed the >>> terminal >>> multiplexer never gives each bash shell opened, a chance to write the >>> history >>> file on exit. >>> >>> The usual result after an initial install for several months, the history >>> file will >>> remain practically empty. >> >>That's got nothing to do with screen. Any long-lived terminal will >>have the same behavior. If you want to write to the history file, you >>can do so at any time: >> >>$ help history >>... >> -aappend history lines from this session to the history file > > However, on regular bash exit or logout, isn't history synced (or appended) at > that time automatically? The real question: Why aren't your bash processes automatically writing something to the history file? Mine certainly do, even when using screen, as I have bash configured to do so.
Re: bash tab variable expansion question?
Hi Chet, The shopt "direxpand" feature works as advertised (Thanks!) except that I noticed it seems to break the name-completion of executables which are not in you path. Example: (colortable16.sh is a script and executable) $ ll colortable16.sh -rwxr-xr-x 1 user group 1458 2009-12-03 19:18 colortable16.sh* $ shopt | grep direxpand direxpand off $ ./col (will expand the name to colortable16.sh => a correct behaviour) $ shopt -s direxpand $ shopt | grep direxpand direxpand on $ ./col (will NOT expand the name colortable16.sh => Incorrect behaviour?) B.t.w I have the bind 'set show-all-if-ambiguous on' so I only need to press once. Bug or feature? Any hints? Thanks in advance, Michael > On 2/24/11 5:14 PM, Michael Kalisz wrote: > >> Bash Version: 4.2 >> Patch Level: 0 >> Release Status: release >> >> Description: >> >> Hi! >> >> Example: >> >> In bash, version 4.1.5(1)-release: >> >> $ echo $PWD/ >> will expand the $PWD variable to your current directory >> >> while in bash, version 4.2.0(1)-release: >> >> $ echo $PWD/ >> will just escape the $ in front of the $ variable i.e: >> >> $ echo \$PWD/ >> The shell-expand-line (Ctrl-Alt-e) works but before I could use just TAB >> >> Any hints why? Any way to get the 4.1 behavior in 4.2? > > I wrote, in a subsequent message in a related thread: > >> The difference is that bash-4.1 expanded $HOME and left the expansion >> as part of the replacement text. Bash-4.2 tries to leave what the >> user typed alone, but that leads to an inherently ambiguous situation: >> when do you quote the `$' in a filename (or, in this case, a directory >> name)? It could be a shell variable, and it could be a character in >> the filename. > > That is the problem, in a nutshell. I posted a partial patch at the > end of March that applied a heuristic to avoid quoting variable > expansions in most cases, but there was no way to get the bash-4.1 > behavior back. > > The attached patch adds a new shell option that, when enabled, is > intended to restore the bash-4.1 behavior of expanding directory names > in filenames being completed. I have done some testing, and it seems > to work the way I intend. This, or some later version, will be part > of the next bash release. I am soliciting feedback on this iteration. > > I'm sending this directly to everyone who's commented negatively about > the default bash-4.2 behavior, as well as bug-bash. Please try the new > option (`direxpand') and let me know if it's missing anything. The patch > includes the original heuristic I sent out back in March, the new shopt > option, and updates to the documentation and test suite. It should apply > cleanly to bash-4.2.10. > > Chet > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer >``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.edu > http://cnswww.cns.cwru.edu/~chet/ >
Re: String behaviour
On Fri, Jun 24, 2011 at 10:38, BX wrote:
> #/bin/bash
> # file1
> import()
> {
> source file2
> }
>
> import
>
> echo v1=$v1, v2=$v2, v3=$v3
>
> #/bin/bash
> # file2
> v1=v1
> declare -x v2=v2
> declare v3=v3
>
> Run above script by
> $ bash file1
>
> Expected result: v1=v1, v2=v2, v3=v3
> Real result: v1=v1, v2=, v3=
>From the documentation here:
info '(bash)Shell Functions'
we have:
When the name of a shell function is used as a
simple command name, the list of commands
associated with that function name is executed.
Shell functions are executed in the current shell
context; no new process is created to interpret
them.
...
The BODY of the function is the compound
command COMPOUND-COMMAND
and from:
info '(bash)Command Grouping'
we have:
Placing a list of commands between curly braces
causes the list to be executed in the current
shell context. No subshell is created.
and from:
info '(bash)Bourne Shell Builtins'
we know that the `source' builtin works as follows:
Read and execute commands from the FILENAME
argument in the current shell context
and from:
info '(bash)Bash Builtins'
we have
When used in a function, `declare' makes each
NAME local, as with the `local' command, unless
the `-g' option is used.
So, basically, the results are as expected: You execute the function
`import', which executes `source file2' in the current shell context,
which creates the variable `v1' in the current shell context, but
creates the variables `v2' and `v3' local to the function being
executed in the current shell context, so that by the echo statement,
only `v1' is defined.
Sincerely,
Michael Witten
AIX and Interix also do early PID recycling.
Configuration Information [Automatically generated, do not change]:
Machine: powerpc
OS: aix5.3.0.0
Compiler: powerpc-ibm-aix5.3.0.0-gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='powerpc'
-DCONF_OSTYPE='aix5.3.0.0' -DCONF_MACHTYPE='powerpc-ibm-aix5.3.0.0'
-DCONF_VENDOR='ibm' -DLOCALEDIR='/tools/haubi/gentoo/sauxz3/usr/share/locale'
-DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I./include -I. -I./include
-I./lib
-DDEFAULT_PATH_VALUE='/tools/haubi/gentoo/sauxz3/usr/sbin:/tools/haubi/gentoo/sauxz3/usr/bin:/tools/haubi/gentoo/sauxz3/sbin:/tools/haubi/gentoo/sauxz3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
-DSTANDARD_UTILS_PATH='/tools/haubi/gentoo/sauxz3/bin:/tools/haubi/gentoo/sauxz3/usr/bin:/tools/haubi/gentoo/sauxz3/sbin:/tools/haubi/gentoo/sauxz3/usr/sbin:/bin:/usr/bin:/sbin:/usr/sbin'
-DSYS_BASHRC='/tools/haubi/gentoo/sauxz3/etc/bash/bashrc'
-DSYS_BASH_LOGOUT='/tools/haubi/gentoo/sauxz3/etc/bash/bash_logout'
-DNON_INTERACTIVE_LOGIN_SHELLS -DSSH_SOURCE_BASHRC
-I/tools/haubi/gentoo/sauxz3/usr/include -g -O2
uname output: AIX sauxz3 3 5 0A03D600
Machine Type: powerpc-ibm-aix5.3.0.0
Bash Version: 4.2
Patch Level: 36
Release Status: release
Description:
On AIX (5.3, 6.1, 7.1), as well as on Interix (any version) I do
encounter
some race condition in a code similar to:
if grep "unwanted" /some/nonexistent/filename
then
echo "bad"
exit 1
fi
echo "good"
Sometimes it does "bad" while it should do "good" always.
As this is part of some large build process, and occurs every once in a
while, I've been unable to create a small testcase.
However, I've found /very/ similar problem description with Cygwin,
even if I don't grok why defining RECYCLES_PIDS shouldn't be enough
here:
http://www.cygwin.com/ml/cygwin/2004-09/msg00882.html
Also, with an older version of libtool, I've had this problem too:
http://lists.gnu.org/archive/html/bug-bash/2008-07/msg00117.html
While this doesn't happen with more recent libtool any more,
I've found an identical problem description with Cygwin again,
even if that test script doesn't expose the problem to me:
http://sources.redhat.com/ml/cygwin/2002-08/msg00449.html
However, here's a rough script to show when PIDs get recycled.
---
#! /usr/bin/env bash
count=0
min=
max=
while true
do
/bin/true &
last=$!
[[ ${min:=${last}} -gt ${last} ]] && min=${last}
[[ ${max:=${last}} -lt ${last} ]] && max=${last}
[[ ${#last} > 4 ]] && used=used_${last::((${#last}-4))} ||
used=used_0
if [[ ${!used} == *" ${last} "* ]]; then
break
fi
(( count+=1 ))
eval "${used}+=' ${last} '"
if [[ "${count}" == *000 ]]; then
echo ${count}
fi
done
echo "reused pid ${last} (min ${min}, max ${max}) after ${count}
trials."
---
On AIX, this script shows something like:
"reused pid 121692 (min 88110, max 121854) after 254 trials."
On Interix, this is something like:
"reused pid 1805 (min 135, max 2107) after 121 trials."
Running this script multiple times in parallel reduces number of trials,
especially on Interix.
Linux, HP-UX and Solaris need something near 32k trials, depending
on how the kernel is configured (Linux: kernel.pid_max).
Repeat-By:
Unable to repeat in a small testcase, but adding some debug fprintf's
to bash's job.c and execute_cmd.c near fork() and waitpid() allowed
me to identify pid-recycling as the root problem.
Fix:
Define RECYCLES_PIDS for AIX and Interix too (like for Cygwin and
LynxOS).
Re: AIX and Interix also do early PID recycling.
On 07/24/2012 05:49 PM, Greg Wooledge wrote:
> On Tue, Jul 24, 2012 at 05:03:36PM +0200, michael.haubenwall...@salomon.at
> wrote:
>> Description:
>> On AIX (5.3, 6.1, 7.1), as well as on Interix (any version) I do
>> encounter
>> some race condition in a code similar to:
>> if grep "unwanted" /some/nonexistent/filename
>> then
>>echo "bad"
>>exit 1
>> fi
>> echo "good"
There is nothing multiprocessing nor asynchronous in this script snippet, there
isn't even a pipe or subshell somewhere. Copy&pasted the code actually is:
# Verify that the libtool files don't contain bogus $D entries.
local abort=no gentoo_bug=no
for a in "${ED}"usr/lib*/*.la ; do
s=${a##*/}
if grep -qs "${D}" "${a}" ; then
vecho -ne '\a\n'
eqawarn "QA Notice: ${s} appears to contain PORTAGE_TMPDIR
paths"
abort="yes"
fi
done
[[ ${abort} == "yes" ]] && die "soiled libtool library files found"
When it erroneously fails, the message is "QA Notice: *.la appears to contain
...",
however there is no filename '*.la'.
Agreeed, a bug here is that nullglob should be set to not run grep at all when
there is no *.la file, but this just would hide the bash bug...
>> Sometimes it does "bad" while it should do "good" always.
>
> If that happens, then I don't see how it is related to recyling PIDs.
> In fact, if grep is failing to produce the correct result, it it a
> problem with your OS's implementation of grep, and not with bash.
Adding some debug-printfs to bash itself around fork, execve, waitpid shows:
Bash usually does fork()+execve("grep"), as well as waitpid(-1, ...).
Whenever waitpid() returns this "grep" one's PID, the reported exitstatus
always is 2, even when bash goes "bad"...
Adding more debug-printfs to bash's wait_for() and execute_command_internal()
shows:
Usually, execute_command_internal() does wait_for(this one grep's PID) before
executing anything else, correctly evaluating the returnvalue to be "not true",
skipping the "bad" part.
But when there was some previous but unrelated command, where fork() returned
the
same PID than for this "grep", execute_command_internal() does /not/ wait_for()
at all,
because last_made_pid is equal to last_pid, and the path to "bad" is gone
instead,
as another exitstatus is evaluated instead of this grep's one.
However, in a subsequent wait_for(another child), waitpid() does report
exitstatus 2
for this grep's PID, but bash has gone "bad" already and ignores that
exitstatus.
>>[[ ${#last} > 4 ]] && used=used_${last::((${#last}-4))} ||
>> used=used_0
>
> That [[ ${#last} > 4 ]] check is incorrect. You're doing a string
> comparison there; [[ 10 > 4 ]] is false. Either use ((...)) or use -gt.
Indeed! (but irrelevant - is just a performance optimisation)
> In any case, if your script breaks because PIDs are recycled sooner than
> you expect, then it is a bug in your script, and not in bash.
It's not me nor my script to expect anything about PIDs at all here.
> (What
> would you expect bash to do about it in the first place?) It may also
> interest you to know that there are some operating systems that use
> random PID allocation, instead of sequential (OpenBSD for example).
PID randomisation isn't a problem at all, as long as a previously used PID
is not reused too early.
> http://mywiki.wooledge.org/ProcessManagement has some tips on how to
> deal with multiple processes.
Interesting page, but there's nothing that applies here.
Thank you anyway!
/haubi/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 03:05 AM, Chet Ramey wrote: > Bash assumes that there's a PID space at least as > large as CHILD_MAX, and that the kernel will use all of it before reusing > any PID in the space. Posix says that shells must remember up to CHILD_MAX > statuses of terminated asynchronous children (the description of `wait'), > so implicitly the kernel is not allowed to reuse process IDs until it has > exhausted CHILD_MAX PIDs. What about grand-childs? They do count for the kernel, but not for the toplevel shell... > The description of fork() doesn't mention this, > however. The Posix fork() requirement that the PID returned can't > correspond to an existing process or process group is not sufficient to > satisfy the requirement on `wait'. OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by fork(). However, I'm unable to find that in the POSIX spec. > Bash holds on to the status of all terminated processes, not just > background ones, and only checks for the presence of a newly-forked PID > in that list if the list size exceeds CHILD_MAX. One of the results of > defining RECYCLES_PIDS is that the check is performed on every created > process. What if the shell does not do waitpid(-1), but waitpid(known-child-PID). That would mean to waitpid(synchronous-child-PID) immediately, and waitpid(asynchronous-child-PID) upon some "wait $!" shell command, rendering to waitpid(-1) when there's no PID passed to "wait". > I'd be interested in knowing the value of CHILD_MAX (or even `ulimit -c') > on the system where you're seeing this problem. The AIX 6.1 I've debugged on has: #define CHILD_MAX 128 #define _POSIX_CHILD_MAX 25 sysconf(_SC_CHILD_MAX) = 1024 $ ulimit -H -c -u core file size (blocks, -c) unlimited max user processes (-u) unlimited $ ulimit -S -c -u core file size (blocks, -c) 1048575 max user processes (-u) unlimited The Interix 6.1 we do have similar-looking stability problems has: CHILD_MAX not defined #define _POSIX_CHILD_MAX 6 sysconf(_SC_CHILD_MAX) = 512 $ ulimit -H -c -u core file size (blocks, -c) unlimited max user processes (-u) 512 $ ulimit -S -c -u core file size (blocks, -c) unlimited max user processes (-u) 512 > The case where last_made_pid is equal to last_pid is a problem only when > the PID space is extremely small -- on the order of, say, 4 -- as long as > the kernel behaves as described above. I'm going to run this build job with 'truss -t kfork' again, to eventually find some too small count of different PIDs before PID-recycling by the kernel... Anyway - defining RECYCLES_PIDS for that AIX 6.1 has reduced the error rate for this one build job from ~37 to 0 when run 50 times. /haubi/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 09:59 AM, Michael Haubenwallner wrote: > On 07/25/2012 03:05 AM, Chet Ramey wrote: >> Bash holds on to the status of all terminated processes, not just >> background ones, and only checks for the presence of a newly-forked PID >> in that list if the list size exceeds CHILD_MAX. > The AIX 6.1 I've debugged on has: > #define CHILD_MAX 128 > I'm going to run this build job with 'truss -t kfork' again, to eventually > find > some too small count of different PIDs before PID-recycling by the kernel... Tracing shows: The minimum fork count (including grand-childs to any depth) before PID recycling starts looks like 255 (once), but usually 256 and more. However, one process does see a PID recycled after *at least* 128 forks, that is exactly the value of CHILD_MAX. First thought is of some off-by-one bug, but reducing js.c_childmax in jobs.c (2 times) by one doesn't help. Investigating further... any hints what to look out for? /haubi/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 02:14 PM, Greg Wooledge wrote: > On Wed, Jul 25, 2012 at 09:59:28AM +0200, Michael Haubenwallner wrote: >> OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by >> fork(). >> However, I'm unable to find that in the POSIX spec. > > A process that hasn't been waited for should become a zombie, which > should be sufficient to prevent its PID being reused. Are you saying > that AIX and Interix don't have zombies? Nope. My thought was that bash eventually could postpone waiting for a specific child PID until required by the driving shell script. That is: immediately for synchronous childs to set $?, and on "wait" for asynchronous childs. The idea was to render storing CHILD_MAX returnvalues obsolete. However, I'm investigating why respecting CHILD_MAX by bash doesn't work when the kernel starts reusing PIDs after CHILD_MAX different ones. /haubi/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 03:20 PM, Michael Haubenwallner wrote: > On 07/25/2012 09:59 AM, Michael Haubenwallner wrote: >> On 07/25/2012 03:05 AM, Chet Ramey wrote: >>> Bash holds on to the status of all terminated processes, not just >>> background ones, and only checks for the presence of a newly-forked PID >>> in that list if the list size exceeds CHILD_MAX. > >> The AIX 6.1 I've debugged on has: >> #define CHILD_MAX 128 >> #define _POSIX_CHILD_MAX 25 >> sysconf(_SC_CHILD_MAX) = 1024 > Tracing shows: > > The minimum fork count (including grand-childs to any depth) before PID > recycling starts > looks like 255 (once), but usually 256 and more. > > However, one process does see a PID recycled after *at least* 128 forks, > that is exactly the value of CHILD_MAX. Got it: The value used for js.c_childmax isn't 128, but 1024. In lib/sh/oslib.c, getmaxchild() prefers sysconf(_SC_CHILD_MAX) over CHILD_MAX over MAXUPRC. But sysconf(_SC_CHILD_MAX) does return the number of "processes per real user id" (similar to ulimit -u), rather than the number of CHILD_MAX (whenever defined). For Interix, things are different though: There is no CHILD_MAX nor MAXUPRC defined, and sysconf(_SC_CHILD_MAX) does return 512, but PIDs start to be recycled at ~120 already... Any idea about the "correct" fix for getmaxchild() across platforms? /haubi/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 04:50 PM, Chet Ramey wrote: >> The AIX 6.1 I've debugged on has: >> #define CHILD_MAX 128 >> #define _POSIX_CHILD_MAX 25 >> sysconf(_SC_CHILD_MAX) = 1024 > Bash prefers sysconf(_SC_CHILD_MAX) and will use it over the other > defines (lib/sh/oslib.c:getmaxchild()). I don't know why AIX chooses > to return a different value via sysconf than it defines for CHILD_MAX, > especially when it seems to use the CHILD_MAX value to decide when it > can recycle the PID space. Well, _SC_CHILD_MAX is documented across platforms as: (Linux) "The max number of simultaneous processes per user ID." (HP-UX) "Maximum number of simultaneous processes per user ID." (Solaris) "Max processes allowed to a UID" (AIX) "Specifies the number of simultaneous processes per real user ID." (Interix) "Maximum number of simultaneous processes per user ID." Also, one Linux machine actually shows the _SC_CHILD_MAX value equal to kernel.pid_max (32768 here), so even Linux could see this problem in theory, because PIDs really are recycled before kernel.pid_max. > And I suspect that the single change of significance is to not check > against the childmax value when deciding whether or not to look for and > remove this pid from the list of saved termination status values. Agreed - but is this still different to defining RECYCLES_PIDS then? Thank you! /haubi/
Re: AIX and Interix also do early PID recycling.
On 07/25/12 19:06, Chet Ramey wrote: Well, _SC_CHILD_MAX is documented across platforms as: Heck, even POSIX specifies CHILD_MAX as: "Maximum number of simultaneous processes per real user ID." Also, one Linux machine actually shows the _SC_CHILD_MAX value equal to kernel.pid_max (32768 here), That's interesting, since Posix describes sysconf() as simply a way to retrieve values from limits.h or unistd.h that one wishes to get at run time rather than compile time. And interesting that it establishes a correspondence between CHILD_MAX and _SC_CHILD_MAX. There's this one sentence in sysconf spec: The value returned shall not be more restrictive than the corresponding value described to the application when it was compiled with the implementation's or . So CHILD_MAX is the /minimum/ value sysconf(_SC_CHILD_MAX) may return. And I suspect that the single change of significance is to not check against the childmax value when deciding whether or not to look for and remove this pid from the list of saved termination status values. Agreed - but is this still different to defining RECYCLES_PIDS then? It is not. It is one of the things that happens when you define RECYCLES_PIDS. The question is whether or not that is the single thing that makes a difference in this case. If it is, there is merit in removing the check against js.c_childmax entirely or making it dependent on something else. IMO, checking against js.c_childmax (sysconf's value) still makes sense to have some upper limit, while being large enough to be useful. However, defining the "useful" value is up to the kernel, which does guarantee for static CHILD_MAX (or _POSIX_CHILD_MAX) at least, while providing more than 100 in practice across various platforms. However, having the "useful" value unavailable to bash feels like rendering the RECYCLES_PIDS-implementation mandatory for /any/ platform. /haubi/
Re: AIX and Interix also do early PID recycling.
On 07/26/12 20:29, Chet Ramey wrote:
OK, we have some data, we have a hypothesis, and we have a way to test it.
Let's test it.
Michael, please apply the attached patch, disable RECYCLES_PIDS, and run
your tests again. This makes the check for previously-saved exit statuses
unconditional.
Let's see if this is the one change of significance.
Nope, doesn't fix the problem, even if it might be necessary though
to not mix up stored exitstates.
Somehow this is related to last_made_pid being preserved across childs
created for { $() } or { `` }.
In execute_command_internal(), last_made_pid still holds the 128 forks
old (first) PID, causing wait_for() to be not run when getting the same
PID by execute_simple_command() again.
However, I've been able to create a short testcase now:
---
#! /bin/bash
/bin/false # make first child
for x in {1..127}; do
x=$( : ) # make CHILD_MAX-1 more childs
done
# breaks when first child's PID is recycled here
if /bin/false; then
echo BOOM
exit 1
fi
echo GOOD
---
/haubi/
Re: AIX and Interix also do early PID recycling.
On 07/26/2012 11:37 PM, Michael Haubenwallner wrote:
> On 07/26/12 20:29, Chet Ramey wrote:
>> OK, we have some data, we have a hypothesis, and we have a way to test it.
>> Let's test it.
>>
>> Michael, please apply the attached patch, disable RECYCLES_PIDS, and run
>> your tests again. This makes the check for previously-saved exit statuses
>> unconditional.
>>
>> Let's see if this is the one change of significance.
>
> Nope, doesn't fix the problem, even if it might be necessary though
> to not mix up stored exitstates.
For mixing up stored exitstates: This patch isn't enough to get below testcase
working reliably - it also is necessary to drop the pid_wrap detection, as pids
aren't guaranteed to be (re)used in any particular order. However, this highly
depends on the machine's load.
With attached patch I haven't been able to break the testcase below so far
on that AIX 6.1 box here.
But still, the other one using the $()-childs still fails.
---
for job in {128..511} {0..127}
do
if [[ ${job} -lt 128 ]]; then
( exit 17 ) &
else
( exit 1 ) &
fi
eval "pidof_${job}=\$!"
done
for job in {127..0}; do
pid=pidof_${job}
pid=${!pid}
wait ${pid}
ret=$?
if [ ${ret} -ne 17 ]; then
echo "job ${job} failed with ret ${ret}"
fi
done
---
Thank you!
/haubi/ (away for next 3 weeks)
*** jobs.c.orig 2012-07-27 15:29:54.283862562 +0200
--- jobs.c 2012-07-27 15:29:51.960238374 +0200
***
*** 1897,1903
--- 1897,1906
#endif
+ #if 0
if (pid_wrap > 0)
+ #endif
delete_old_job (pid);
+ #if 0
#if !defined (RECYCLES_PIDS)
/* Only check for saved status if we've saved more than CHILD_MAX
***
*** 1905,1908
--- 1908,1912
if ((js.c_reaped + bgpids.npid) >= js.c_childmax)
#endif
+ #endif
bgp_delete (pid); /* new process, discard any saved status */
Re: AIX and Interix also do early PID recycling.
On 07/29/2012 12:46 AM, Chet Ramey wrote:
> On 7/27/12 9:50 AM, Michael Haubenwallner wrote:
>
>> With attached patch I haven't been able to break the testcase below so far
>> on that AIX 6.1 box here.
>>
>> But still, the other one using the $()-childs still fails.
>
> Try the attached patch for that.
Collecting the patches and cleaning up now unused code, attached patch
seems to fix both CHILD_MAX related problems on that AIX box here now,
without using the RECYCLES_PIDS workaround.
Thank you!
/haubi/
Bash assumes pids aren't reused before sysconf(_SC_CHILD_MAX) immediate childs
(the dynamic value), as well as ascending and wrapped around pid values.
However, as specified by POSIX, conforming kernels actually guarantee for
CHILD_MAX imediate childs (the static value) before reusing pids. Additionally,
AIX (at least) does not guarantee for ascending pid values at all. Actually,
AIX reuses pids after its CHILD_MAX value of 128 in somewhat random order in
some configuration- or load-cases, resulting in race conditions like these:
http://lists.gnu.org/archive/html/bug-bash/2008-07/msg00117.html
This looks like a similar problem with Cygwin, where RECYCLES_PIDS is defined
as the workaround, but that isn't really correct for AIX (and maybe Interix):
http://www.cygwin.com/ml/cygwin/2004-09/msg00882.html
http://www.cygwin.com/ml/cygwin/2002-08/msg00449.html
*** jobs.c.orig 2012-08-20 16:23:51 +0200
--- jobs.c 2012-08-20 16:51:36 +0200
***
*** 317,324
static char retcode_name_buffer[64];
- /* flags to detect pid wraparound */
- static pid_t first_pid = NO_PID;
- static int pid_wrap = -1;
-
#if !defined (_POSIX_VERSION)
--- 317,320
***
*** 347,352
{
js = zerojs;
- first_pid = NO_PID;
- pid_wrap = -1;
}
--- 343,346
***
*** 1823,1833
as the proper pgrp if this is the first child. */
- if (first_pid == NO_PID)
- first_pid = pid;
- else if (pid_wrap == -1 && pid < first_pid)
- pid_wrap = 0;
- else if (pid_wrap == 0 && pid >= first_pid)
- pid_wrap = 1;
-
if (job_control)
{
--- 1817,1820
***
*** 1863,1875
#endif
! if (pid_wrap > 0)
! delete_old_job (pid);
! #if !defined (RECYCLES_PIDS)
! /* Only check for saved status if we've saved more than CHILD_MAX
! statuses, unless the system recycles pids. */
! if ((js.c_reaped + bgpids.npid) >= js.c_childmax)
! #endif
! bgp_delete (pid); /* new process, discard any saved status */
last_made_pid = pid;
--- 1850,1856
#endif
! delete_old_job (pid);
! bgp_delete (pid); /* new process, discard any saved status */
last_made_pid = pid;
*** execute_cmd.c.orig 2012-08-20 16:36:10 +0200
--- execute_cmd.c 2012-08-20 16:51:14 +0200
***
*** 742,748
/* XXX - this is something to watch out for if there are problems
! when the shell is compiled without job control. */
! if (already_making_children && pipe_out == NO_PIPE &&
! last_made_pid != last_pid)
{
stop_pipeline (asynchronous, (COMMAND *)NULL);
--- 742,750
/* XXX - this is something to watch out for if there are problems
! when the shell is compiled without job control. Don't worry about
! whether or not last_made_pid == last_pid; already_making_children
! tells us whether or not there are unwaited-for children to wait
! for and reap. */
! if (already_making_children && pipe_out == NO_PIPE)
{
stop_pipeline (asynchronous, (COMMAND *)NULL);
Re: AIX and Interix also do early PID recycling.
On 08/28/2012 09:21 AM, Roman Rakus wrote: > On 08/01/2012 03:13 PM, Chet Ramey wrote: >> On 7/30/12 10:41 AM, Roman Rakus wrote: >> >>> Hmm... I don't know much about boundaries of maximum number of user >>> processes. But anyway - do you think that (re)changing js.c_childmax (when >>> `ulimit -u' is changed) is not good? >> Maybe it's ok up to some fixed upper bound. But if you're going to have >> that fixed upper bound, why not just use it as the number of job exit >> statuses to remember all the time? >> > I prepared a patch which add configure option to enable and set the number of > job exit statuses to remember. Why not simply use the static CHILD_MAX value instead? Feels like this is what the spec means - and conforming kernels do not guarantee for more than that anyway, counting synchronous, asynchronous and substituted commands together. However, Linux has stopped defining CHILD_MAX (not so) recently (value was 999), so _POSIX_CHILD_MAX (25 is current value, 6 is old value) would feel correct then... Anyway, now I do understand why people use pipes instead to get the child's exitstatus: http://thread.gmane.org/gmane.linux.gentoo.portage.devel/3446/focus=3451 /haubi/
How is being invoked via login different?
Hello,
I gain more knowledge about process controls, jobs and shells I figured it
would be interesting to throw together my own shell. Just a simple, throw away
test to help me gain knowledge.
I was basing a lot of the flow on how bash works however I have hit a problem.
My code only works when I run it after I am logged in, from a bash prompt :(
I read from stdin, parse then call fork and run the command in execvp().
If I set my test shell as the default and try to use it via ssh'ing in or by
"su - -s /my_test/shell" if fails after the fork. I can read stdin
and print out. However after the fork the command executed ("ls") seems to be
unable to access stdin, stdout, stderr and goes into defunct.
Any help or guidance is appreciated, is there somewhere in the source code of
bash that would point me in the right direction?
Thanks
Michael
No such file or directory
Hi, I have a complaint. Apparently, when unknowingly attempting to run a 32-bit executable file on a 64-bit computer, bash gives the error message "No such file or directory". That error message is baffling and frustratingly unhelpful. Is it possible for bash to provide a better error message in this case? Thanks, -Mike
Re: No such file or directory
Hi Aharon, Thanks for your explanation. Now I have another question. Why is the error message for ENOENT simply "No such file or directory", when the man page for execve has this complete description: "The file filename or a script or ELF interpreter does not exist, or a shared library needed for file or interpreter cannot be found."? Thanks, -Mike On 1/1/13, Aharon Robbins wrote: > In article , > Michael Williamson wrote: >>Hi, >> >>I have a complaint. Apparently, when unknowingly attempting to run a >>32-bit executable file on a 64-bit computer, bash gives the error message >>"No such file or directory". That error message is baffling and >> frustratingly >>unhelpful. Is it possible for bash to provide a better error message >>in this case? >> >>Thanks, >>-Mike > > It's not Bash. That is the error returned from the OS in errno when > it tries to do an exec(2) of the file. Bash merely translates the > error into words. > > Depending on the distro, it's usually not too hard to install the 32 bit > compatibility files to support 32 bit executables. The "enterprise" > distibutions tend to do that by default, whereas the more free ones > (Fedora, Ubuntu, ...) require that you do it yourself. > > I agree, it's confusing. But it's a decision made by the kernel guys, > not Bash. > > HTH, > > Arnold > -- > Aharon (Arnold) Robbins arnold AT skeeve DOT com > P.O. Box 354 Home Phone: +972 8 979-0381 > Nof AyalonCell Phone: +972 50 729-7545 > D.N. Shimshon 99785 ISRAEL >
Fwd: No such file or directory
-- Forwarded message -- From: Eric Blake Date: Wed, 02 Jan 2013 10:41:07 -0700 Subject: Re: No such file or directory To: Michael Williamson On 01/02/2013 10:30 AM, Michael Williamson wrote: > Hi Eric, > > Thanks for your explanation. I realize now that I should > submit my suggestion somewhere besides bug-bash, and > I am researching that. But, I have another question. > > Why does the man page have the complete, accurate error > message for ENOENT, but not glibc? Did you mean for this to go to the list (as other people might be interested in the question and answer)? And if so, please don't top-post. strerror() is full of historical baggage - changing the strings it outputs is not done lightly, even the string printed is not always the best fit for all documented cases in which a particular error is returned. Thus, the man pages will ALWAYS be a better fit than what glibc can cram into the limited abilities of strerror(). -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org OK, thanks for that explanation Eric. In that case, my naive suggestion would be to introduce a new glibc function, such as strerror_verbose. -Mike signature.asc Description: PGP signature
Subscribe
<>
[PATCH] Fix process substitution with named pipes.
When /dev/fd is missing, and named pipes are used instead (like on AIX),
this snippet sometimes does work right, wrong, or hang - depending on
the operating system's process scheduler timing:
for x in {0..9}; do echo $x; done > >(
cnt=0; while read line; do let cnt=cnt+1; done; echo $cnt
)
To reproduce this problem on Linux, add this line to subst.c to enforce
the problematic timing behaviour:
#if !defined (HAVE_DEV_FD)
+ sleep(1);
fifo_list[nfifo-1].proc = pid;
#endif
and enforce using named pipes: bash_cv_dev_fd=absent ./configure ...
---
subst.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/subst.c b/subst.c
index 48c89c1..afae3b7 100644
--- a/subst.c
+++ b/subst.c
@@ -5075,7 +5075,7 @@ process_substitute (string, open_for_read_in_child)
#if !defined (HAVE_DEV_FD)
/* Open the named pipe in the child. */
- fd = open (pathname, open_for_read_in_child ? O_RDONLY|O_NONBLOCK :
O_WRONLY);
+ fd = open (pathname, open_for_read_in_child ? O_RDONLY : O_WRONLY);
if (fd < 0)
{
/* Two separate strings for ease of translation. */
--
1.8.1.5
$"text" TEXTDOMAIN{,DIR} ordering relevance
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I../bash -I../bash/include -I../bash/lib
-D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Wformat-security -Werror=format-security -Wall
uname output: Linux rocket76 3.2.0-55-generic #85-Ubuntu SMP Wed Oct 2 12:29:27
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu
Bash Version: 4.2
Patch Level: 25
Release Status: release
Description:
TEXTDOMAIN must be defined before TEXTDOMAINDIR - otherwise it does not
work.
Use of $"text" in functions does not work if TEXTDOMAIN{,DIR} is defined
below the defined function - regardless of application flow.
Repeat-By:
#!/bin/bash
l1="/tmp/test/de/LC_MESSAGES" # adjust according to your needs
l2="/tmp/test" # short form of above
mkdir -p $l1
echo -e 'msgid "failed"\nmsgstr "worked"' | msgfmt -o $l1/test.mo -
echo test1
export TEXTDOMAINDIR=$l2
export TEXTDOMAIN=test
f() {
echo ' '$"failed"
}
echo ' '$"failed"
f
echo test2
unset TEXTDOMAINDIR TEXTDOMAIN
export TEXTDOMAIN=test
export TEXTDOMAINDIR=$l2
echo ' '$"failed"
f
echo test3
unset f
f() {
echo ' '$"failed"
}
echo ' '$"failed"
f
Fix:
You must set TEXTDOMAIN before TEXTDOMAINDIR (bug?!)
You must set TEXTDOMAIN{,DIR} above the function (unexpexted for me)
Re: Weird process substitution behavior
On 11/14/2013 08:56 PM, Chet Ramey wrote: > On 11/8/13 6:26 PM, John Dawson wrote: >> The following surprised me. I thought line 4 of the output, and certainly >> line 5 of the output, should have said "0 /dev/fd/63" too. Is this behavior >> a bug? > > I'm still looking at this. I have not had a great deal of time to > investigate. Maybe interesting: With named pipes (some non-Linux platforms) this does hang after the first line. Not sure which behaviour actually is the Right Thing though. /haubi/
Re: Pb bash with process substitution on AIX : compilation logs for bash 4.2
Hi! On 11/28/2013 02:32 PM, Flene TOUMANI wrote: > Is it possible to get a feedback on the issue? (E.g. a confirmation that this > is a bug). Sounds like you've run into this problem (patch available): http://lists.gnu.org/archive/html/bug-bash/2013-10/msg00114.html /haubi/
dynamic-complete-history documentation change
Hi It seems that the binding for Alt-Tab changed from dynamic-complate-history to tab-insert around Bash 3.0. The bashref.info file as found in Debian and the current version 3.0 Bash source tarball on http://ftp.gnu.org/gnu/bash/ both still state the default binding for dynamic-complete-history is Alt-Tab. It looks like this section of the Bash Reference Manual is generated from the file rluser.texi, which is provided by Readline. The current version on the web no longer mentions dynamic-complete-history. http://cnswww.cns.cwru.edu/~chet/readline/readline.html#SEC19 Could you please regenerate the file bashref.info in the upstream sources and/or update the Readline documentation if you get the time. Thanks ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
Change of the Makefile
Hello, Shell Community
My name is Michael Song
Recently I found shell is a nice program that can be extended to solve my
automatic regression test problem. So I started hacking it.
I found it would be easiler use $(wildcard) in the builtins/Makefile.in, in
stead of staticly specify all the source files:
Here is the snippet of code change to illustrate my idea:
In builtins/Makefile.in
Original Definition:
DEFSRC = $(srcdir)/alias.def $(srcdir)/bind.def $(srcdir)/break.def \
$(srcdir)/builtin.def $(srcdir)/caller.def \
$(srcdir)/cd.def $(srcdir)/colon.def \
$(srcdir)/command.def $(srcdir)/declare.def $(srcdir)/echo.def \
$(srcdir)/enable.def $(srcdir)/eval.def $(srcdir)/getopts.def \
$(srcdir)/exec.def $(srcdir)/exit.def $(srcdir)/fc.def \
$(srcdir)/fg_bg.def $(srcdir)/hash.def $(srcdir)/help.def \
$(srcdir)/history.def $(srcdir)/jobs.def $(srcdir)/kill.def \
$(srcdir)/let.def $(srcdir)/read.def $(srcdir)/return.def \
$(srcdir)/set.def $(srcdir)/setattr.def $(srcdir)/shift.def \
$(srcdir)/source.def $(srcdir)/suspend.def $(srcdir)/test.def \
$(srcdir)/times.def $(srcdir)/trap.def $(srcdir)/type.def \
$(srcdir)/ulimit.def $(srcdir)/umask.def $(srcdir)/wait.def \
$(srcdir)/pushd.def $(srcdir)/shopt.def \
$(srcdir)/printf.def $(srcdir)/complete.def
Proposed Definition:
DEFSRC = $(wildcard $(srcdir)/*.def)
clean:
$(RM) -rf $(patsubst %.def, %.c, $(DEFSRC)) $(CREATED_FILES)
$(MKBUILTINS) *.o libbuiltins.a -r helpfiles
This wild card works based on the assumption that the reserved.def file is
renamed to something else
In the Makefile.in under the root, the change is as follows:
Original Definition:
BUILTIN_DEFS = $(DEFSRC)/alias.def $(DEFSRC)/bind.def $(DEFSRC)/break.def \
$(DEFSRC)/builtin.def $(DEFSRC)/cd.def $(DEFSRC)/colon.def \
$(DEFSRC)/command.def ${DEFSRC}/complete.def \
$(DEFSRC)/caller.def $(DEFSRC)/declare.def \
$(DEFSRC)/echo.def $(DEFSRC)/enable.def $(DEFSRC)/eval.def \
$(DEFSRC)/exec.def $(DEFSRC)/exit.def $(DEFSRC)/fc.def \
$(DEFSRC)/fg_bg.def $(DEFSRC)/hash.def $(DEFSRC)/help.def \
$(DEFSRC)/history.def $(DEFSRC)/jobs.def $(DEFSRC)/kill.def \
$(DEFSRC)/let.def $(DEFSRC)/read.def $(DEFSRC)/return.def \
$(DEFSRC)/set.def $(DEFSRC)/setattr.def $(DEFSRC)/shift.def \
$(DEFSRC)/source.def $(DEFSRC)/suspend.def $(DEFSRC)/test.def \
$(DEFSRC)/times.def $(DEFSRC)/trap.def $(DEFSRC)/type.def \
$(DEFSRC)/ulimit.def $(DEFSRC)/umask.def $(DEFSRC)/wait.def \
$(DEFSRC)/getopts.def \
$(DEFSRC)/pushd.def $(DEFSRC)/shopt.def $(DEFSRC)/printf.def
New Definition:
BUILTIN_DEFS = $(wildcard $(DEFSRC)/*.def)
OFILES = builtins.o $(patsubst %.def, %.o, $(DEFSRC)) $(patsubst %.c, %.o,
$(STATIC_SOURCE))
Thanks
Michael Song
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
ksh style [[ conditional does not match patterns
In the SHELL GRAMMAR section of the bash man page, the [[ expression ]] syntax is described: When the == and != operators are used, the string to the right of the operator is considered a pattern and matched according to the rules described below under Pattern Matching. The Pattern Matching subsection describes the familiar file name globbing syntax: * Matches any string, including the null string. ? Matches any single character. etc. Yet when I attempt a simple match, it doesn't work: bash-3.00$ [[ "foo" == "foo" ]] bash-3.00$ echo $? 0 bash-3.00$ [[ "foo" == "fo?" ]] bash-3.00$ echo $? 1 (I expect the second command to return 0 exit status as well, since the question mark should match the single "o" character at position 3 in "foo".) Am I doing something wrong? ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
Re: Readline-5.1 released
Chet Ramey <[EMAIL PROTECTED]> writes: > q. Extensive changes to readline to add enough state so that commands > requiring additional characters (searches, multi-key sequences, numeric > arguments, commands requiring an additional specifier character like > vi-mode change-char, etc.) work without synchronously waiting for > additional input. If this is in response to my bug report (and, I guess, even if it's not :) thanks a lot! It fixes both my test case and my actual use case. Cheers, mwh -- Ya, ya, ya, except ... if I were built out of KSR chips, I'd be running at 25 or 50 MHz, and would be wrong about ALMOST EVERYTHING almost ALL THE TIME just due to being a computer! -- Tim Peters, 30 Apr 97 ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
Can create BASH functions that can't be deleted
Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: linux-gnu
Compiler: i386-redhat-linux-gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i386' -DCONF_OSTYPE='linu
x-gnu' -DCONF_MACHTYPE='i386-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDI
R='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./inc
lude -I./lib -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -m32 -marc
h=i386 -mtune=pentium4
uname output: Linux sonic-mike1 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT 20
05 i686 i686 i386 GNU/Linux
Machine Type: i386-redhat-linux-gnu
Bash Version: 3.0
Patch Level: 15
Release Status: release
Description:
Naming restriction inconsistencies allow us to create
BASH functions that can't be deleted.
Repeat-By:
function a.b() { echo My name is $FUNCNAME; }
a.b
My name is a.b
unset a.b
-bash: unset: `a.b': not a valid identifier
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
Can create function that can't be deleted
Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: linux-gnu
Compiler: i386-redhat-linux-gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i386' -DCONF_OSTYPE='linu
x-gnu' -DCONF_MACHTYPE='i386-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDI
R='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./inc
lude -I./lib -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -m32 -marc
h=i386 -mtune=pentium4
uname output: Linux sonic-mike1 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT 20
05 i686 i686 i386 GNU/Linux
Machine Type: i386-redhat-linux-gnu
Bash Version: 3.0
Patch Level: 15
Release Status: release
Description:
Naming restriction inconsistencies allow us to create
BASH functions that can't be deleted.
Repeat-By:
function a.b() { echo My name is $FUNCNAME; }
a.b
My name is a.b
unset a.b
-bash: unset: `a.b': not a valid identifier
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
-d option not working. . .?
Hi All,
I've got a script that I'm trying to set up, but it keeps telling me
that "[-d command not found". Can someone please explain what is
wrong with this?:
#!/bin/sh
for i in $*
do
{
if [-d $i]
then
echo "$i is a directory! Yay!"
else
echo "$i is not a directory!"
fi
}
done
Regards,
Michael
Re: -d option not working. . .?
On Sep 12, 2007, at 2:56 AM, Bob Proulx wrote: The [ is a shell builtin, not a shell metacharacter. Shell metacharacters do not need to be separated by whitespace but the test program needs to be apart or it won't be parsed right. That is why you are seeing "[-d" not found. It is not a parenthesis as in some programming languages. The [ is a synonym for the "test" operator. It is a command like grep, sed, awk, etc. Being that I'm not a bash (or any other shell for that matter) guru, is there any reason that parsing occurs this way? Why is it not more like other programming languages? [ -d /tmp ] && echo /tmp is a dir Do it this way. if [ -d "$i" ] Note that you should quote the argument to protect against whitespace there. Thanks, that worked perfectly!
Re: -d option not working. . .?
Two words: history and POSIX. It's been done that way for more than 20 years, so it was standardized that way. Changing it would break too many existing scripts. Forgive me for saying so, and please appreciate both the sarcasm and irony, but I've never been one for "that's the way it's always been". I mean, if we all thought that way, we'd be a bunch of bloodletting flat-earthers. . .no? ;) That said, I'm sure these BASH and POSIX guys were (and still are) a lot smarter than myself. So I'll rest my faith on that and see about doing a bit more getting familiar. Best Regards All! Michael
Array Elements with Spaces
Group,
I have having a problem with spaces in individual elements of an array.
The space causes a single element to be seen as multiple elements.
Here is a sample run:
-
[EMAIL PROTECTED]:~/bin> ./arraywithspace.sh
3.1.17(1)-release
got 6 parms
parm: arg1
parm: arg2
parm: arg
parm: 3
parm: arg
parm: four
[EMAIL PROTECTED]:~/bin> cat ./arraywithspace.sh
#!/bin/bash
declare -a Arguments=("arg1" "arg2" "arg 3" "arg four")
function countparms
{
echo "got $# parms"
while (($# != 0 ))
do
echo "parm: $1"
shift
done
}
echo $BASH_VERSION
countparms ${Arguments[*]}
--
The real problem I am having is using the array as a list of arguments
to a command, maybe something like this:
grep -i mypattern ${FileList[*]}
--
potter
Re: Array Elements with Spaces
Thanks,
That fixed it.
For the benefit of others...
I ran my test program on a couple different versions of bash (including 2.x)
and It seems that this behavior (combination of quotes and * versus @)
changes based on the version of bash, and not just between 2.x and 3.x. I
did not do an exhaustive test because this solution will work for me.
--
potter
On Nov 10, 2007 12:27 PM, Andreas Schwab <[EMAIL PROTECTED]> wrote:
> "Michael Potter" <[EMAIL PROTECTED]> writes:
>
> > countparms ${Arguments[*]}
>
> Use "[EMAIL PROTECTED]" instead (including the quotes). See node Arrays
> in the Bash docs.
>
> Andreas.
>
> --
> Andreas Schwab, SuSE Labs, [EMAIL PROTECTED]
> SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
> PGP key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>
bash-shipped getcwd() replacement does not work on interix.
Machine: i586
OS: interix5.2
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i586'
-DCONF_OSTYPE='interix5.2' -DCONF_MACHTYPE='i586-pc-interix5.2'
-DCONF_VENDOR='pc'
-DLOCALEDIR='/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2/share/locale'
-DPACKAGE='bash'
-DLOCAL_PREFIX=/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2
-DSHELL -DHAVE_CONFIG_H -DNO_MAIN_ENV_ARG -DBROKEN_DIRENT_D_INO -D_POSIX_SOURCE
-I. -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2
-I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2/include
-I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2/lib -g -O2
uname output: Interix pc312001 5.2 SP-9.0.3790.3034 x86
Intel_x86_Family6_Model15_Stepping6
Machine Type: i586-pc-interix5.2
Bash Version: 3.2
Patch Level: 33
Release Status: release
Description:
Bash uses getcwd-replacement if libc provides getcwd without the
feature of allocating the buffer when called without one.
This override is done in config-bot.h, with an exception for
solaris already.
Problem now is that getcwd-replacement does not work on Interix
(SUA 5.2 here).
Now there's only one source location in builtins/common.c really
relying on getcwd(0,0) allocating the buffer.
But even here is some conditional code on GETCWD_BROKEN.
So why not simply don't require allocation-feature of getcwd at all
and use getcwd-replacement only if libc does not provide one ?
Repeat-By:
$ PWD=
/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2/bin/bash
shell-init: error retrieving current directory: getcwd: cannot access
parent directories: No such file or directory
Fix: (patch attached)
builtins/common.c:
Do not depend on getcwd() doing buffer allocation.
config-bot.h:
Ignore GETCWD_BROKEN, keep HAVE_GETCWD as is.
Additionally, the check for GETCWD_BROKEN can be dropped
from configure.in and aclocal.m4.
Thanks!
/haubi/
--
Michael Haubenwallner
Gentoo on a different level
diff -ru builtins/common.c builtins/common.c
--- builtins/common.c Wed Dec 19 10:30:07 2007
+++ builtins/common.c Wed Dec 19 10:34:58 2007
@@ -479,11 +479,8 @@
if (the_current_working_directory == 0)
{
-#if defined (GETCWD_BROKEN)
- the_current_working_directory = getcwd (0, PATH_MAX);
-#else
- the_current_working_directory = getcwd (0, 0);
-#endif
+ char *t = xmalloc(PATH_MAX);
+ the_current_working_directory = getcwd (t, PATH_MAX);
if (the_current_working_directory == 0)
{
fprintf (stderr, _("%s: error retrieving current directory: %s: %s\n"),
diff -ru config-bot.h config-bot.h
--- config-bot.h Wed Dec 19 10:30:06 2007
+++ config-bot.h Wed Dec 19 10:31:16 2007
@@ -70,14 +70,6 @@
# define TERMIOS_MISSING
#endif
-/* If we have a getcwd(3), but one that does not dynamically allocate memory,
- #undef HAVE_GETCWD so the replacement in getcwd.c will be built. We do
- not do this on Solaris, because their implementation of loopback mounts
- breaks the traditional file system assumptions that getcwd uses. */
-#if defined (HAVE_GETCWD) && defined (GETCWD_BROKEN) && !defined (SOLARIS)
-# undef HAVE_GETCWD
-#endif
-
#if !defined (HAVE_DEV_FD) && defined (NAMED_PIPES_MISSING)
# undef PROCESS_SUBSTITUTION
#endif
diff -ru configure.in configure.in
--- configure.in Wed Dec 19 10:30:09 2007
+++ configure.in Wed Dec 19 10:37:08 2007
@@ -894,9 +894,6 @@
BASH_FUNC_OPENDIR_CHECK
BASH_FUNC_ULIMIT_MAXFDS
BASH_FUNC_GETENV
-if test "$ac_cv_func_getcwd" = "yes"; then
-BASH_FUNC_GETCWD
-fi
BASH_FUNC_POSIX_SETJMP
BASH_FUNC_STRCOLL
diff -ru aclocal.m4 aclocal.m4
--- aclocal.m4 Tue Sep 12 23:18:07 2006
+++ aclocal.m4 Wed Dec 19 10:37:33 2007
@@ -684,32 +684,6 @@
fi
])
-AC_DEFUN(BASH_FUNC_GETCWD,
-[AC_MSG_CHECKING([if getcwd() will dynamically allocate memory])
-AC_CACHE_VAL(bash_cv_getcwd_malloc,
-[AC_TRY_RUN([
-#include
-#ifdef HAVE_UNISTD_H
-#include
-#endif
-
-main()
-{
- char *xpwd;
- xpwd = getcwd(0, 0);
- exit (xpwd == 0);
-}
-], bash_cv_getcwd_malloc=yes, bash_cv_getcwd_malloc=no,
- [AC_MSG_WARN(cannot check whether getcwd allocates memory when cross-compiling -- defaulting to no)
-bash_cv_getcwd_malloc=no]
-)])
-AC_MSG_RESULT($bash_cv_getcwd_malloc)
-if test $bash_cv_getcwd_malloc = no; then
-AC_DEFINE(GETCWD_BROKEN)
-AC_LIBOBJ(getcwd)
-fi
-])
-
dnl
dnl This needs BASH_CHECK_SOCKLIB, but since that's not called on every
dnl system, we can't use AC_PREREQ
Re: bash-shipped getcwd() replacement does not work on interix.
On Thu, 2007-12-20 at 12:30 +0100, Andreas Schwab wrote:
> Michael Haubenwallner <[EMAIL PROTECTED]> writes:
>
> > diff -ru builtins/common.c builtins/common.c
> > --- builtins/common.c Wed Dec 19 10:30:07 2007
> > +++ builtins/common.c Wed Dec 19 10:34:58 2007
> > @@ -479,11 +479,8 @@
> >
> >if (the_current_working_directory == 0)
> > {
> > -#if defined (GETCWD_BROKEN)
> > - the_current_working_directory = getcwd (0, PATH_MAX);
> > -#else
> > - the_current_working_directory = getcwd (0, 0);
> > -#endif
> > + char *t = xmalloc(PATH_MAX);
> > + the_current_working_directory = getcwd (t, PATH_MAX);
>
> The length of the cwd may be bigger than PATH_MAX.
Eventually - but there are three (ok, two) other locations in bash-3.2
where buffer[PATH_MAX] is passed to getcwd():
1) jobs.c:
current_working_directory()
2) parse.y:
decode_prompt_string()
3) lib/readline/examples/fileman.c:
com_pwd()
ok, this just is an example, and uses 1024 instead of PATH_MAX.
Instead of using PATH_MAX, why not have some xgetcwd() instead, doing
malloc (when getcwd does not allocate itself), and increase the buffer
when getcwd() returns ERANGE ?
/haubi/
--
Michael Haubenwallner
Gentoo on a different level
Re: bash-shipped getcwd() replacement does not work on interix.
On Thu, 2007-12-20 at 08:08 -0500, Chet Ramey wrote:
> Michael Haubenwallner wrote:
> > Machine: i586
> > OS: interix5.2
> > Compiler: gcc
> > Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i586'
> > -DCONF_OSTYPE='interix5.2' -DCONF_MACHTYPE='i586-pc-interix5.2'
> > -DCONF_VENDOR='pc'
> > -DLOCALEDIR='/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2/share/locale'
> > -DPACKAGE='bash'
> > -DLOCAL_PREFIX=/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2
> > -DSHELL -DHAVE_CONFIG_H -DNO_MAIN_ENV_ARG -DBROKEN_DIRENT_D_INO
> > -D_POSIX_SOURCE -I.
> > -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2
> > -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2/include
> > -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2/lib -g -O2
> > uname output: Interix pc312001 5.2 SP-9.0.3790.3034 x86
> > Intel_x86_Family6_Model15_Stepping6
> > Machine Type: i586-pc-interix5.2
> >
> > Bash Version: 3.2
> > Patch Level: 33
> > Release Status: release
> >
> > Description:
> > Bash uses getcwd-replacement if libc provides getcwd without the
> > feature of allocating the buffer when called without one.
> > This override is done in config-bot.h, with an exception for
> > solaris already.
> > Problem now is that getcwd-replacement does not work on Interix
> > (SUA 5.2 here).
>
> I'd be more interested in knowing why it doesn't work in this case,
> instead of discarding it. Since I neither have nor use Interix, I
> need someone who does to investigate the issue a little bit.
It is because readdir() returns 0 (zero) for (struct dirent).(d_ino),
while stat() returns useful values for (struct stat).(st_ino), so their
equal-comparison never succeeds.
Now, while trying to get inode number from stat() rather than readdir(),
I've seen another bug unrelated to readdir()/stat(), but still in
getcwd() replacement, causing a coredump here.
It is with the memcpy() from the internal buffer to the allocated return
buffer, but only when there is a minimal buffer size specified - wth. is
this done in get_working_directory() when GETCWD_BROKEN is defined:
Does Solaris (config-bot.h) allocate the buffer when a size is passed ?
Attached patch fixes this one issue, by still allocating at least
provided buffer size, but doing the memcpy with real path length.
When done with buffer size, memcpy reads beyond the end of the source
buffer on the stack. The SIGSEGV was caused here because it has read
beyond the whole stack frame page.
/haubi/
--
Michael Haubenwallner
Gentoo on a different level
--- lib/sh/getcwd.c.orig Fri Dec 21 11:34:00 2007
+++ lib/sh/getcwd.c Fri Dec 21 11:58:41 2007
@@ -252,9 +256,9 @@
size_t len = pathbuf + pathsize - pathp;
if (buf == NULL)
{
- if (len < (size_t) size)
- len = size;
- buf = (char *) malloc (len);
+ if (len > (size_t) size)
+ size = len;
+ buf = (char *) malloc (size);
if (buf == NULL)
goto lose2;
}
Re: bash-shipped getcwd() replacement does not work on interix.
On Fri, 2007-12-21 at 13:51 +0100, Michael Haubenwallner wrote: > On Thu, 2007-12-20 at 08:08 -0500, Chet Ramey wrote: > > Michael Haubenwallner wrote: > > > Machine: i586 > > > OS: interix5.2 > > > Compiler: gcc > > > Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i586' > > > -DCONF_OSTYPE='interix5.2' -DCONF_MACHTYPE='i586-pc-interix5.2' > > > -DCONF_VENDOR='pc' > > > -DLOCALEDIR='/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2/share/locale' > > > -DPACKAGE='bash' > > > -DLOCAL_PREFIX=/tools/snapshot/prefix-launcher-1pre.20071219/i586-pc-interix5.2 > > > -DSHELL -DHAVE_CONFIG_H -DNO_MAIN_ENV_ARG -DBROKEN_DIRENT_D_INO > > > -D_POSIX_SOURCE -I. > > > -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2 > > > -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2/include > > > -I/tss/prefix-launcher-1pre.20071219/buildroot/bash/bash-3.2/lib -g -O2 > > > uname output: Interix pc312001 5.2 SP-9.0.3790.3034 x86 > > > Intel_x86_Family6_Model15_Stepping6 > > > Machine Type: i586-pc-interix5.2 > > > > > > Bash Version: 3.2 > > > Patch Level: 33 > > > Release Status: release > > > > > > Description: > > > Bash uses getcwd-replacement if libc provides getcwd without the > > > feature of allocating the buffer when called without one. > > > This override is done in config-bot.h, with an exception for > > > solaris already. > > > Problem now is that getcwd-replacement does not work on Interix > > > (SUA 5.2 here). > > > > I'd be more interested in knowing why it doesn't work in this case, > > instead of discarding it. Since I neither have nor use Interix, I > > need someone who does to investigate the issue a little bit. > > It is because readdir() returns 0 (zero) for (struct dirent).(d_ino), > while stat() returns useful values for (struct stat).(st_ino), so their > equal-comparison never succeeds. Attached patch should fix this issue, not relying on readdir() returning valid d_ino, but doing stat() always instead. But eventually there should be a configure-check or sth. like that if readdir returns valid d_ino, and subsequently avoid the additional stat. Moving alloca() into separate function was necessary because there is no realloca() or sth. like that, and wasting stack for each iteration is bad. /haubi/ -- Michael Haubenwallner Gentoo on a different level
Re: bash-shipped getcwd() replacement does not work on interix.
On Sat, 2007-12-22 at 10:13 -0500, Chet Ramey wrote:
> Michael Haubenwallner wrote:
> >> It is because readdir() returns 0 (zero) for (struct dirent).(d_ino),
> >> while stat() returns useful values for (struct stat).(st_ino), so their
> >> equal-comparison never succeeds.
> >
> > Attached patch should fix this issue, not relying on readdir() returning
> > valid d_ino, but doing stat() always instead.
>
> You didn't attach one.
Uh oh, indeed, sorry. Here it is.
/haubi/
--
Michael Haubenwallner
Gentoo on a different level
--- lib/sh/getcwd.c.orig Fri Dec 21 11:34:00 2007
+++ lib/sh/getcwd.c Fri Dec 21 14:37:57 2007
@@ -58,6 +58,24 @@
# define NULL 0
#endif
+static int concat_path_and_stat(char *dotp, size_t dotlen,
+ char *nam, size_t namlen,
+ struct stat *st, char mount_point, ino_t thisino,
+ int *saved_errno)
+{
+ char *name;
+ name = alloca(dotlen + 1 + namlen + 1);
+ memcpy(name, dotp, dotlen);
+ name[dotlen] = '/';
+ memcpy(&name[dotlen+1], nam, namlen+1);
+ if (stat(name, st) < 0)
+return -1;
+ if (mount_point || st->st_ino == thisino)
+ if (lstat(name, st) < 0)
+ *saved_errno = errno;
+ return 0;
+}
+
/* Get the pathname of the current working directory,
and put it in SIZE bytes of BUF. Returns NULL if the
directory couldn't be determined or SIZE was too small.
@@ -169,31 +187,15 @@
(d->d_name[1] == '\0' ||
(d->d_name[1] == '.' && d->d_name[2] == '\0')))
continue;
- if (mount_point || d->d_fileno == thisino)
- {
- char *name;
-
- namlen = D_NAMLEN(d);
- name = (char *)
- alloca (dotlist + dotsize - dotp + 1 + namlen + 1);
- memcpy (name, dotp, dotlist + dotsize - dotp);
- name[dotlist + dotsize - dotp] = '/';
- memcpy (&name[dotlist + dotsize - dotp + 1],
- d->d_name, namlen + 1);
- if (lstat (name, &st) < 0)
- {
-#if 0
- int save = errno;
- (void) closedir (dirstream);
- errno = save;
- goto lose;
-#else
- saved_errno = errno;
-#endif
- }
- if (st.st_dev == thisdev && st.st_ino == thisino)
- break;
- }
+ namlen = D_NAMLEN(d);
+ if (concat_path_and_stat(dotp, dotlist + dotsize - dotp,
+ d->d_name, namlen,
+ &st, mount_point, thisino,
+ &saved_errno
+ ) < 0)
+ goto lose;
+ if (st.st_dev == thisdev && st.st_ino == thisino)
+ break;
}
if (d == NULL)
{
Re: bash's own getcwd reads uninitialized/nonexistent memory
On Wed, 2008-01-23 at 17:45 +0100, Philippe De Muyter wrote: > here is a patch : LOL - this is a very similar patch as http://lists.gnu.org/archive/html/bug-bash/2007-12/msg00084.html /haubi/ -- Michael Haubenwallner Gentoo on a different level
Re: read output of process into a variable
It is not a bug in bash. it is just how it works. the while loop creates a subshell and changes to the variables are not visable outside of the subshell. if you put the while loop first, then it will not create the subshell. do this: result="" while read line; do extracteddata=`echo "$line" | sed -e 's/X/Y/'` result="$result $extracteddata" done < <(/usr/bin/output_generator) /usr/bin/another_tool "$result" the <() is syntax for a named pipes. it makes a command look like a file. Be aware that this may leave files in your /tmp directory. BTW: I would use $() syntax instead of the backtic syntax; just easier to see. -- potter On 30 Jan 2008 11:21:34 GMT, Stefan Palme <[EMAIL PROTECTED]> wrote: > Hi, > don't know if this is the right newsgroup, but it's the only one > I can find with "bash" in its name :-) > > I want to do something like this: > > result="" > /usr/bin/output_generator | while read line; do > extracteddata=`echo "$line" | sed -e 's/X/Y/'` > result="$result $extracteddata" > done > /usr/bin/another_tool "$result" > > In the last line is "result" as empty as at the start of the > whole thing - I guess because the inner "while" loop is executed > in a subshell, so that changing the value of "result" in this > loop does not affect the "outer result". > > How can I solve this? I have some very ugly solutions, but > I guess there must be something "nice" :-) > > (using bash-3.2.17(1)-release) > > Thanks and regards > -stefan- > > >
errexit does not exit script in subshells
Bash Bunch,
Not surprisingly, bash does not exit the script when an error is detected in
a subshell.
I am hopeful that someone has a solution to this (other than: be careful to
not use subshells).
Here is the test run:
---
./subshellnofail.sh
BEGIN
ERROR: ./subshellnofail.sh 10 <- would like the script to stop here.
ONE
ERROR: ./subshellnofail.sh 10
---
Here is the sample script:
--
#!/bin/bash
set -o errexit
set -o noclobber
set -o nounset
set -o pipefail# if you fail on this line, get a newer version of bash.
function traperr
{
echo "ERROR: ${BASH_SOURCE[0]} ${LINENO}"
# exit 1 # does not seem to affect the program.
# return 1 # does not seem to affect the program.
}
set -o errtrace
trap traperr ERR
echo "BEGIN"
cat /etc/passwd |while read aLine
do
# in a subshell
test=$(true | false | true |false);
done
echo "ONE"
while read aLine
do
# not in a subshell
test=$(true | false | true |false);
done
