timeout
Hi, I have a question. Why is the "timeout" command disappearing from recent Linux distributions? I am referring to the command that allows a time limit to be set for another command: # timeout 10 ftp ... It is useful to me. I don't think it is a bash built-in command, but why not? Thanks, -Mike
COMP_WORDBREAKS and shopt hostcomplete cause crash
Configuration Information [Automatically generated, do not change]: Machine: i486 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i486' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i486-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I../bash -I../bash/include -I../bash/lib -g -O2 -Wall uname output: Linux kvm1 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:20 UTC 2008 i686 GNU/Linux Machine Type: i486-pc-linux-gnu Bash Version: 3.2 Patch Level: 39 Release Status: release Description: get_comp_wordbreaks() in variables.c records the value of rl_completer_word_break_characters into the COMP_WORDBREAKS variable value field. But enable_hostname_completion() frees it and allocates new memory for rl_completer_word_break_characters without updating the COMP_WORDBREAKS variable value field. Eventually shell_reinitialize is used and calls delete_all_contexts(shell_variables), freeing the COMP_WORDBREAKS value memory a second time. That can happen when the shell is running a shell script that doesn't have a "#!/bin/bash" line. The corruption causes a shell crash or hang. The exact results depend on what (if any) unfortunate code allocated the freed memory before the extra call to free it. Repeat-By: $ echo date > btest.sh; chmod +x btest.sh $ /bin/bash -c 'shopt -u hostcomplete;echo $COMP_WORDBREAKS;shopt -s hostcomplete;exec ./btest.sh' "'><=;|&(: malloc: ../bash/variables.c:2296: assertion botched free: called with already freed block argument Aborting...Aborted Fix: Make a copy of the rl_completer_word_break_characters value instead of using the original address. diff --git a/variables.c b/variables.c index 072a590..e9705ba 100644 --- a/variables.c +++ b/variables.c @@ -1314,7 +1314,8 @@ get_comp_wordbreaks (var) if (rl_completer_word_break_characters == 0 && bash_readline_initialized == 0) enable_hostname_completion (perform_hostname_completion); - var_setvalue (var, rl_completer_word_break_characters); + FREE (value_cell (var)); + var_setvalue (var, savestring (rl_completer_word_break_characters)); return (var); } -- Mike Stroyan <[EMAIL PROTECTED]>
Shell Grammar man page
Hi,
It seems some things are missing in the bash manual.
Notably definition of command and placements of coproc- and function-definition.
The section 'SHELL GRAMMAR' describes:
- simple-command
- pipeline
- list
- compound-command
- coproc
- function-definition
Simplified, a pipeline is:
[ ! ] command1 [ | command2 ]
A list is a sequence of pipelines separated by ;, &, &&, or ||
and optionally terminated by ;, & or NL.
Within the list description, the and-list and or-list are described as:
and-list: command1 && command2
or-list: command1 || command2
Though the text says that and/or-lists are sequences of pipelines
(separated by && or ||).
That is slightly inconsistent.
A pipeline can be reduced to a command.
I don't think a command can be a pipeline nor a list.
In that case, the and/or list should read:
and-list: pipeline1 && pipeline2
or-list: pipeline1 || pipeline2
These are valid:
: | { :; }
list: pipeline: command1 s|' command2
command1: simple-command
command2: compound-command
{ :; } || :
list: or-list: pipeline1 || pipeline2
pipeline1: command : compound-command
pipeline2: command: simple-command
:() if [ x = x ]; then echo; fi
Re: Shell Grammar man page
On Tue, Feb 23, 2021 at 04:33:44PM -0500, Chet Ramey wrote: > On 2/22/21 8:11 AM, Mike Jonkmans wrote: > > > > Hi, > > > > It seems some things are missing in the bash manual. > > Notably definition of command and placements of coproc- and > > function-definition. > > > > The section 'SHELL GRAMMAR' describes: > > - simple-command > > - pipeline > > - list > > - compound-command > > - coproc > > - function-definition > > These are basically the `command' productions from the grammar. The > exception is `list', and it's there because it would be strange for > the reader if it were presented before simple commands and pipelines, > which users are more likely to encounter, even though a complete > command is technically a list. It is mostly recursive, so you have to start somewhere. Your reasoning is understandable. Start with what the reader knows. Though the readers are likely to be familiar with grammars/rules/start symbols. Because 'command' is not described, a reader might think that all these items are commands. > > Simplified, a pipeline is: > > [ ! ] command1 [ | command2 ] > > > > A list is a sequence of pipelines separated by ;, &, &&, or || > > and optionally terminated by ;, & or NL. > > > > Within the list description, the and-list and or-list are described as: > > and-list: command1 && command2 > > or-list: command1 || command2 > > Though the text says that and/or-lists are sequences of pipelines > > (separated by && or ||). > > `command1' and `command2' are meta-notation, not a grammar production. I know. It is just as the man page describes it. The numbering is quite handy for referring in the text. > > That is slightly inconsistent. > Not really. > > > A pipeline can be reduced to a command. > In a sense, yes, depending on your definition of `command'. That is the point. The documentation never describes what a 'command' is. The inconsistency comes when you try to follow the documentation's use of 'command'. It is only in the words, not in the real grammar of course. > > In that case, the and/or list should read: > > and-list: pipeline1 && pipeline2 > > or-list: pipeline1 || pipeline2 > > No, it really shouldn't. You're right. These pipelines should be lists. In the mean time i have looked into parse.y The grammar is quite readable. Good. > The bash grammar is a little messier than that POSIX grammar in > https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_10_02 > but it's basically the same. Thanks for the link. > Where are you trying to go with this? I have been wondering about this (missing command description) for some time. Before even trying to submit a rewrite, i wanted to make it more clear. For me, it is clear now. But i am not sure whether a rewrite is worth the trouble. It is also a bit problematic that the word 'command' is overloaded. We give commands, not lists. Though a pipeline might be entered ;) Regards, Mike Jonkmans
Re: Shell Grammar man page
On Wed, Feb 24, 2021 at 12:08:50PM -0500, Chet Ramey wrote: > On 2/23/21 7:31 PM, Mike Jonkmans wrote: > > On Tue, Feb 23, 2021 at 04:33:44PM -0500, Chet Ramey wrote: > > > On 2/22/21 8:11 AM, Mike Jonkmans wrote: > > > > > > > > Hi, > > > > > > > > It seems some things are missing in the bash manual. > > > > Notably definition of command and placements of coproc- and > > > > function-definition. > > > > > > > > The section 'SHELL GRAMMAR' describes: > > > > - simple-command > > > > - pipeline > > > > - list > > > > - compound-command > > > > - coproc > > > > - function-definition > > > > > > These are basically the `command' productions from the grammar. The > > > exception is `list', and it's there because it would be strange for > > > the reader if it were presented before simple commands and pipelines, > > > which users are more likely to encounter, even though a complete > > > command is technically a list. > > > > It is mostly recursive, so you have to start somewhere. > > Sure. We start with lists. That's just not something that will resonate > with a reader. > > > Your reasoning is understandable. Start with what the reader knows. > > Though the readers are likely to be familiar with grammars/rules/start > > symbols. > > Readers are likely to be familiar with simple commands and pipelines, > especially the ones who primarily use interactive shells. I disagree that > the average reader is likely to be familiar with grammars. Aren't grammars taught already in primary school? It is in the Netherlands. But i agree that it is a good thing to let the text not depend on that. Starting with 'Statements' might be an option. E.g. Shell Grammar Statements Input to bash is a sequence of statements separated by ;, &, or newline. A statement sequence is parsed first. Only after a newline does bash perform any effects, like command execution, defined by these statements. A statement consists of simple-command, pipeline, compound-command, ... (as defined below). Simple Commmand ... > > Because 'command' is not described, a reader might think that all these > > items > > are commands. > > > > > > Simplified, a pipeline is: > > > > [ ! ] command1 [ | command2 ] > > > > > > > > A list is a sequence of pipelines separated by ;, &, &&, or || > > > > and optionally terminated by ;, & or NL. > > > > > > > > Within the list description, the and-list and or-list are described as: > > > > and-list: command1 && command2 > > > > or-list: command1 || command2 > > > > Though the text says that and/or-lists are sequences of pipelines > > > > (separated by && or ||). > > > > > > `command1' and `command2' are meta-notation, not a grammar production. > > > > I know. It is just as the man page describes it. > > The numbering is quite handy for referring in the text. > > > > > > That is slightly inconsistent. > > > Not really. > > > > > > > A pipeline can be reduced to a command. > > > In a sense, yes, depending on your definition of `command'. > > > > That is the point. The documentation never describes what a 'command' is. > > The inconsistency comes when you try to follow the documentation's > > use of 'command'. It is only in the words, not in the real grammar of > > course. > > So where would you suggest changes? I'd argue that the use of `command' in > the simple commands section is quite clear. I'd also argue that everyone > understands what `the last command in a pipeline' means, or `commands in a > list', and that those are more descriptive than `pipeline elements' or > `list elements'. I agree that the usage of 'command' is clear in these cases. But these commands are different things. The command in pipelines might be better described as compound-command. May I also suggest to make it more clear that coprocesses and function-definitions are compound-commands also? The indentation of these differ from the preceding while/if/etc. The description of Coprocesses uses the words 'shell command'. That looks like it came from shell_command in parse.y and is actually a compound-command without coproc and funcdef (limited-compound-comm
Re: Shell Grammar man page
On Thu, Feb 25, 2021 at 11:15:36PM +0100, Ángel wrote: > On 2021-02-25 at 16:13 +0100, Mike Jonkmans wrote: > > > > Aren't grammars taught already in primary school? It is in the > > Netherlands. > > But i agree that it is a good thing to let the text not depend on > > that. > > I really doubt so. I expect they would teach Dutch¹ grammar, but that's > quite different than formal grammars such as those of yacc. > Maybe you might use that in the teaching of a constructed language like > Esperanto, but I don't think that would be a good approach. Those grammars weren't all that different from yacc's grammar. Just simpler and incomplete. Minimal example: A Sentence can be "Pronoun Verb." or "Verb Pronoun?" Pronoun can be "I" or "You" etc. Verb would be "eat" or "walk". Sometimes you needed to 'parse' sentences in order to spell them correctly. > I expect people somewhat familiar with grammar constructs would rather > come from descriptions of commands or technical documentation if not > directly from a Linguistic, Mathematics or Computer Engineering > background. > Plus, the need to grasp concepts of variables and recursion must not be > undersestimated. It is always difficult to estimate the knowledge of your (reading) public. > An interesting point about primary school is that they probably do > teach the basic of number bases: 1234 = 1×1 + 2×1000 + 3×100 + 4×1 > (albeit they wouldn't use 10ⁿ syntax, or actual multiply signs), > which is exactly what you do with other bases. Yet it's something > nobody seems to remember when needing octal or hexadecimal years > later... > > Best regards > ¹ Or English, French, German… That is secondary school stuff (for the Dutch). As I recall there was less to do with grammar in those foreign languages. Regards, Mike Jonkmans
Re: Shell Grammar man page
On Thu, Feb 25, 2021 at 06:28:34PM -0500, Chet Ramey wrote:
> On 2/25/21 10:13 AM, Mike Jonkmans wrote:
> > Starting with 'Statements' might be an option.
>
> Maybe. Or a POSIX-like description that says a command can be a
>
> simple command
> list
> pipeline
> compound command
> function definition
> coproc
>
> just like the sections in the SHELL GRAMMAR section.
The missing definition of command was one of my initial points.
So that would be nice.
> > > So where would you suggest changes? I'd argue that the use of `command' in
> > > the simple commands section is quite clear. I'd also argue that everyone
> > > understands what `the last command in a pipeline' means, or `commands in a
> > > list', and that those are more descriptive than `pipeline elements' or
> > > `list elements'.
> >
> > I agree that the usage of 'command' is clear in these cases.
> > But these commands are different things.
> > The command in pipelines might be better described as compound-command.
>
> No. A simple command may be a pipeline element (and most often is, given
> how the shell is usually used). Compound commands are as described in the
> manual, basically equivalent to the shell_command production in the bash
> grammar.
Essential rules from parse.y (newlines and some equivalents left out):
simple_command: WORD+
pipeline: pipeline '|' pipeline
| command
command: simple_command | shell_command | function_def | coproc
shell_command: for_command | group_command | ...
function_def: WORD '(' ')' shell_command
coproc: 'coproc' shell_command | 'coproc' simple_command
simple_list: simple_list1 | simple_list1 ';'
simple_list1: pipeline
| simple_list1 '&&' simple_list1
| simple_list1 ';' simple_list1
| ...
group_command: '{' compound_list '}'
compound_list: pipeline## expanded from list0, list1
| pipeline '&&' compound_list
| pipeline ';' compound_list
| pipeline '\n' compound_list
| compound_list ';'
| compound_list '\n'
| ...
The correspondences between terms in the manual and parse.y are:
manual | parse.y
-+-
compound command | shell_command
command | command or simple_list or compound_list
list | simple_list or compound_list
pipeline | pipeline_command
> The compound command description is accurate. I think you're trying to make
> a compound comand more than it is.
Indeed. I mistakenly took functions and coprocs for compound commands. :(
In a later, separate, post, I will take the rules from parse.y
and transform these into an ECFG (with rules like: x: y+ [ nl | ';' ]* )
That should be shorter and look more like the descriptions in the manual.
Then maybe, if time (and wits) permits, I'll puzzle an ELL(1) parser into bash.
Just to see whether that can replace yacc/bison.
(see e.g. https://os.ghalkes.nl/LLnextgen/)
Regards, Mike Jonkmans
Re: Shell Grammar man page
On Fri, Feb 26, 2021 at 02:54:07AM +0100, Ángel wrote: > On 2021-02-26 at 00:45 +0100, Mike Jonkmans wrote: > > On Thu, Feb 25, 2021 at 11:15:36PM +0100, Ángel wrote: > > > > Those grammars weren't all that different from yacc's grammar. > > Just simpler and incomplete. > > > > Minimal example: > > A Sentence can be "Pronoun Verb." or "Verb Pronoun?" > > Pronoun can be "I" or "You" etc. > > Verb would be "eat" or "walk". > > > > Sometimes you needed to 'parse' sentences in order to spell them > > correctly. > > TEchnically yes. A linguist will happily study that one. Also, you > could have someone used to yacc understanding and using your simplified > grammar. The other way around? Based on the average grammar given on > primary school? I don't think so. I don't think that f.i. precedence was taught. Although you get that with arithmetic, which also has a grammar. Not sure if I knew that back then. I might though. > > > I expect people somewhat familiar with grammar constructs would rather > > > come from descriptions of commands or technical documentation if not > > > directly from a Linguistic, Mathematics or Computer Engineering > > > background. > > > Plus, the need to grasp concepts of variables and recursion must not be > > > undersestimated. > > > > It is always difficult to estimate the knowledge of your (reading) > > public. > > Heh, Sure. These are all unscientific estimations. > > > > > > > ¹ Or English, French, German… > > That is secondary school stuff (for the Dutch). > > I was trying to cover people which had their primary school on > different languages (and yes, it was a very limited, non-exhaustive > list) It is exactly the list of languages the Dutch are being teached. Apart from the optional Latin and Greek. > > As I recall there was less to do with grammar in those foreign > > languages. > > Interesting. I wonder if that was because their grammar are simpler or > just because it wasn't as present in the syllabus. I guess that, since the basic ideas of grammar had already been taught, it would not stand out anymore. > I wouldn't think French grammar to be simpler than English one, for > instance. French grammar was the most regular, iirc. English was the most irregular. The really easy thing with English though, is that it doesn't have all these male/female/neuter articles and nouns. > Can't give an opinion on your language though, it's all Dutch to me :-) It's Dutch to me too! > Best > Ángel Regards, Mike Jonkmans
Re: Shell Grammar man page
On Fri, Feb 26, 2021 at 12:41:44PM -0500, Chet Ramey wrote: > On 2/26/21 11:22 AM, Mike Jonkmans wrote: > > > I don't think that f.i. precedence was taught. > > Although you get that with arithmetic, which also has a grammar. > > It's not taught as such. Kids today are taught operator precedence, > phrased as "order of operations": PEMDAS. Yes, something like that we have also: "MVDWOA" The 1st letters of "Meneer Van Dalen Wacht Op Antwoord" In English (roughly): "Mister Van Dalen Waits On Answer" Regards, Mike Jonkmans
Shell Grammar man page function definition
Hi,
The manual page says:
If the function reserved word is used, but the parentheses are not
supplied,
the braces are required.
But it seems that from all the compound commands,
only a subshell is not possible.
Some examples that work:
function x { :; } ## as expected
function x if :; then :; fi
function x (( 42 ))
function x [[ 42 ]]
function x for x do :; done
function x for (( ; 42 - 42 ; )); do :; done
What does not work:
function x ( : )
>From looking at the bash grammar I am not sure why a subshell does not work.
As the subshell can be differentiated from the optional parentheses.
Though it may be handy to disallow a subshell.
If in the future named parameters would become a possibility.
E.g.
function x (a=$1 b=$2)
{ echo "$a$b; }
would become ambiguous.
Note that the Posix grammar doesn't mention the word 'function'.
Posix does mention it, as reserved word recognized by 'some implementations'
and causing undefined behavior.
Regards, Mike Jonkmans
Re: Shell Grammar man page function definition
On Sun, Feb 28, 2021 at 05:06:37PM -0500, Chet Ramey wrote: > On 2/28/21 12:38 PM, Mike Jonkmans wrote: > > > The manual page says: > > If the function reserved word is used, but the parentheses are not > > supplied, > > the braces are required. > > That text has been there since the earliest versions of the man page. It's > not strictly true any more, but it's a reasonable guideline. Maybe I'll > change the language to a recommendation. I think deleting , with one exception: If the function reserved word is used, but the parentheses are not supplied, the braces are required would suffice. > > But it seems that from all the compound commands, > > only a subshell is not possible. > > Subshells work fine in the current version of bash. Nice! Thanks, also to the others to point that out. I did not see that in the changelog (master-branch) though. But the 'culprit' is clear from the next post ;) Regards, Mike Jonkmans
[Patch] Makefile.in [ce]tags
The -x option to ctags generates 'human readable' stuff that vim can not use. This patch removes the -x and the '>$@', which is not needed. Refer to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ctags.html Also introducing $(ETAGS) $(ETAGSFLAGS) and same for CTAGS* Regards, Mike Jonkmans diff --git Makefile.in Makefile.in index 3e3a5d48..8d9883f6 100644 --- Makefile.in +++ Makefile.in @@ -79,6 +79,8 @@ ARFLAGS = @ARFLAGS@ RANLIB = @RANLIB@ SIZE = @SIZE@ STRIP = strip +CTAGS = ctags +ETAGS = etags INSTALL = @INSTALL@ INSTALL_PROGRAM = @INSTALL_PROGRAM@ @@ -807,10 +809,10 @@ force: # unused TAGS: $(SOURCES) $(BUILTIN_C_SRC) $(LIBRARY_SOURCE) - etags $(SOURCES) $(BUILTIN_C_SRC) $(LIBRARY_SOURCE) + $(ETAGS) $(ETAGSFLAGS) $(SOURCES) $(BUILTIN_C_SRC) $(LIBRARY_SOURCE) tags: $(SOURCES) $(BUILTIN_C_SRC) $(LIBRARY_SOURCE) - ctags -x $(SOURCES) $(BUILTIN_C_SRC) $(LIBRARY_SOURCE) > $@ + $(CTAGS) $(CTAGSFLAGS) $(SOURCES) $(BUILTIN_C_SRC) $(LIBRARY_SOURCE) # Targets that actually do things not part of the build
[Patch] .gitignore TAGS and tags
I assume that the TAGS and tags files will not go into the repo. Regards, Mike Jonkmans diff --git .gitignore .gitignore index 1512a9ab..f09f4eac 100644 --- .gitignore +++ .gitignore @@ -113,3 +113,6 @@ examples/loadables/tty examples/loadables/uname examples/loadables/unlink examples/loadables/whoami + +TAGS +tags
Re: [Patch] .gitignore TAGS and tags
On Mon, Mar 15, 2021 at 11:23:46AM -0400, Chet Ramey wrote: > On 3/15/21 3:29 AM, Mike Jonkmans wrote: > > I assume that the TAGS and tags files will not go into the repo. > > Why not? This is only the devel branch; they don't go into releases. Adding tags/TAGS to the repo would increase its size for almost no use. Creating the tags file takes less than a second. Drawback of not having these in the repo and not in .gitignore is that a 'git status' complains about untracked files. Regards, Mike Jonkmans
Re: [Patch] .gitignore TAGS and tags
On Mon, Mar 15, 2021 at 04:06:06PM -0500, Eric Blake wrote: > But even if the upstream repo doesn't want to ignore a file in the > (checked-in) .gitignore, you can always edit your (local-only) > .git/info/exclude to exclude your extra files locally. Thanks Eric, that is useful. Regards, Mike Jonkmans
[Patch] distclean should not rm lib/readline/doc/Makefile
A 'make distclean' removes lib/readline/doc/Makefile It is a handmade Makefile, so probably should not be removed. Idem for maintainer-clean. Regards, Mike Jonkmans diff --git lib/readline/doc/Makefile lib/readline/doc/Makefile index af5ee3e5..6bc2e5ea 100644 --- lib/readline/doc/Makefile +++ lib/readline/doc/Makefile @@ -130,14 +130,12 @@ clean: distclean: clean $(RM) $(CREATED_DOCS) $(RM) $(INTERMEDIATE_OBJ) - $(RM) Makefile mostlyclean: clean maintainer-clean: clean $(RM) $(CREATED_DOCS) $(RM) $(INTERMEDIATE_OBJ) - $(RM) Makefile install: @echo "This documentation should not be installed."
Re: why does this define bla() instead of respect the newline as command separator
On Fri, Mar 19, 2021 at 09:12:34AM +0100, Alex fxmbsw7 Ratchev wrote:
> eval $'alias n=bla\nn() { type $FUNCNAME ; }\nn'
> bla is a function
> bla ()
> {
> type $FUNCNAME
> }
>
> it was supposed to be n() ..
The eval $'...' can be left out.
$ echo $BASH_VERSION
5.0.17(1)-release
$ alias n=bla
$ n() { type $FUNCNAME; }
$ declare -pf n
bash: declare: n: not found
$ declare -pf bla
bla ()
{
type $FUNCNAME
}
It is the same on 5.1.4(1)-release.
I would have expected that n was a defined function.
But I would not have bet on it.
Regards, Mike Jonkmans
Re: [PATCH] Free unfreed string in assign_assoc_from_kvlist()
On Mon, Apr 19, 2021 at 11:27:13PM +0800, konsolebox wrote: > I looked at this code plenty of times and I really think the returned > value to expand_assignment_string_to_string (v, 0) needs to be freed > too when aval != 0. > > ... The patch seems to be correct. Moreover, I think that the line aval = expand_assignment_string_to_string (v, 0); should be moved down below the next if-statement. Because of the continue, there is a memory leak. Regards, Mike Jonkmans
Bash Builtin Read Function Hang
Configuration Information [Automatically generated, do not change]:
Machine: powerpc
OS: aix6.1.9.0
Compiler: gcc -maix64
Compilation CFLAGS: -g -Wno-parentheses -Wno-format-security
uname output: AIX aixutil07 2 7 00C07A304B00
Machine Type: powerpc-ibm-aix6.1.9.0
Bash Version: 5.0
Patch Level: 18
Release Status: release
Description:
I have a script that I wrote in bash and have been using for
years. It worked fine up until I updated bash to the latest version from
IBM, 5.0 Patch 18. Previously I ran it without issue on 5.0 with no
patches. What is happening is when I run a command (such as 'ls -al /etc')
with a while loop and read the line it works fine sometimes but sometimes
it does not. When it does not work fine it hangs once it reaches the end
of the input that is being fed into the loop, it should see the EOF and
automatically terminate but it does not. This previously worked fine 100%
of the time.
I trussed the output and this is what we are getting when we reach the end
of the line:
kfcntl(2, F_GETFL, 0x0FE8) = 67110914
kioctl(0, 22528, 0x, 0x) Err#25 ENOTTY
lseek(0, 0, 1) Err#29 ESPIPE
kread(0, "\t", 1) (sleeping...)
When it is working the kread function does not sleep, it just moves onto
the next iteration in the while loop.
Repeat-By:
Here is a sample piece of code to recreate the issue:
SGLIST=("testfile1" "testfile1" "testfile1" "testfile1") # List
of storage groups
for sg in ${SGLIST[@]}
do
while read line
do
echo $line
done < <(ls -al /etc 2>&1)
done
This particular sample code reliably fails every time. The piece of code I
have written fails with the same basic setup one out of every four times or
so.
Thanks
Re: Memory leak detected by Valgrind
On Tue, Jun 15, 2021 at 04:20:22PM -0400, Chet Ramey wrote: > It depends on the libc implementation of setlocale(3). > > This has come up a number of times before: > > https://lists.gnu.org/archive/html/bug-bash/2015-07/msg00073.html > https://lists.gnu.org/archive/html/bug-bash/2017-04/msg00063.html > https://lists.gnu.org/archive/html/bug-bash/2017-06/msg00052.html There is also uselocale, newlocale, duplocale and freelocale. The manual says POSIX.1-2008, so it should be reasonably portable. Not sure if it is worth the trouble. Regards, Mike Jonkmans
Re: does /etc/profile get sourced also when .profile exists ?
On Mon, Jul 26, 2021 at 03:57:16PM +0200, Alex fxmbsw7 Ratchev wrote: > .. ..: command not found Regards, Mike Jonkmans
Re: gettext feature request
On Tue, Aug 10, 2021 at 11:22:54AM -0400, Chet Ramey wrote: > On 7/31/21 4:17 PM, Jean-Jacques Brucker wrote: > > > > Le 29/07/2021 à 21:08, Chet Ramey a écrit : > > > >> How about `noexpand_translation'? > > > > It sounds good ! (imho). > > This is the current description: > >noexpand_translation >If set, bash encloses the translated results of $"..." >quoting in single quotes instead of double quotes. If >the string is not translated, this has no effect. > > It will be in the next devel branch push. So noexpand_translation is off by default. I suggest to use the positive version: expand_translation As that circumvents the double negation. expand_translation If set, the translated result of $"..." is enclosed in double quotes instead of single quotes. This might expose the script to the translated result. For compatibility reasons, this option is enabled by default. Regards, Mike Jonkmans
Re: efficient way to use matched string in variable substitution
On Tue, Aug 24, 2021 at 02:15:46PM +0200, Léa Gris wrote:
> Le 24/08/2021 à 14:06, Greg Wooledge écrivait :
>
> > unicorn:~$ f6() { local i n=${#1} arr; for ((i=0; i > arr[i]="${1:i:1}"; done; }
> > unicorn:~$ time for ((i=1; i<=1; i++)); do f6 682390; done
> > real 0.381 user 0.381 sys 0.000
> >
> > Looks like the efficiency of "read -ra" vs. a shell loop just about makes
> > up for the system calls used for the here string (f6 and f7 are almost
> > tied in overall speed, with f6 just a *tiny* bit faster). Good to know.
>
> See my featured version to also capture space and newlines:
>
> https://stackoverflow.com/a/68907322/7939871
Clever hack, Lea.
Nice thread, funny (but slow) solutions:
echo "abcdefg" | fold -w1
echo "abcdefg" | grep -o .
This seems to be the fastest:
f12 () { [[ "$1" =~ ${1//?/(.)} ]]; local arr=( "${BASH_REMATCH[@]:1}" ); }
time for ((i=1; i<=1; i++)); do f0 682390; done
real0m0,296s
user0m0,296s
sys 0m0,000s
Regards, Mike Jonkmans
Re: efficient way to use matched string in variable substitution
On Tue, Aug 24, 2021 at 09:24:35AM -0400, Greg Wooledge wrote:
> On Tue, Aug 24, 2021 at 03:09:55PM +0200, Mike Jonkmans wrote:
> > This seems to be the fastest:
> > f12 () { [[ "$1" =~ ${1//?/(.)} ]]; local arr=( "${BASH_REMATCH[@]:1}" ); }
> > time for ((i=1; i<=1; i++)); do f0 682390; done
> > real0m0,296s
> > user0m0,296s
> > sys 0m0,000s
>
> Your CPU is a bit faster than mine, since none of mine (nor yours) go
> quite that fast on my computer, but you've written "f0" in your time
> command, not f12.
oh oh, last minute edit - because i wasn't sure at which fN you started :)
Re: efficient way to use matched string in variable substitution
On Tue, Aug 24, 2021 at 04:16:46PM +0200, Léa Gris wrote:
> Le 24/08/2021 à 15:09, Mike Jonkmans écrivait :
> > This seems to be the fastest:
> > f12 () { [[ "$1" =~ ${1//?/(.)} ]]; local arr=( "${BASH_REMATCH[@]:1}" ); }
>
> Awesome Mike, would you like to add this answer to SO?
>
> It would be very useful there; but I don't want to be wrongly credited for
> this smart implementation.
Oh no, it is not my idea.
It came from that same thread.
Just wanted to tell that it is the fastest method, on my pc at least.
> Made it into a fancy utility function:
>
> string2array() {
> # Splits the string's characters into the array
> # $1: The input string
> # $2: The output array name
> [[ "$1" =~ ${1//?/(.)} ]]
> # shellcheck disable=SC2178 # shellcheck broken nameref type check
> local -n arr="$2"
> # shellcheck disable=SC2034 # shellcheck broken nameref usage check
> arr=("${BASH_REMATCH[@]:1}")
> }
shellcheck is very useful (in vim),
But it does not seem to be moving on :(
Regards, Mike Jonkmans
Re: Why tail coreutil cannot work as its input is from tee
On Mon, Sep 06, 2021 at 07:38:03AM +, Budi wrote: > How come tail coreutil cannot work as its input is from tee > > $ echo -e '9\n4\n3\n2\n8\n7\n3\n1' |tail -1 > 1 > > But : > > $ echo -e '9\n4\n3\n2\n8\n7\n3\n1' |tee >(head -1) |tail -1 > 9 > > Please help explain Not a bug. If you leave out the |tail -1', you will see that the output of head -1 gets appended to that of tee. -- Regards, Mike Jonkmans
Re: hash not restored after running command -p
On Sat, Oct 30, 2021 at 01:53:55PM -0400, Chet Ramey wrote: > On 10/29/21 6:06 PM, Roger Morris wrote: > > > Bash Version: 5.0 > > Patch Level: 17 > > Release Status: release > > > > Description: > > I believe there's a bug in bash 4.4 (and still in 5.1) that wasn't > > there in 4.3.30 > > > > When 'command -p' runs, it no longer seems to restore the hash to its > > previous value. > > This came in in mid-2015 as part of a set of fixes to `command -p'. I think > the bash-4.4 and later behavior is correct: just because you're telling > command to use a standard path it doesn't mean it should not act like it > would in any other circumstance. There's nothing in POSIX that contradicts > that. (Not using 'PATH=' as that would include the current directory). PATH=/dev/null command -p hostname hostname # executes /bin/hostname via the hash table I agree with OP that the behaviour is a bug, or at least unwanted behaviour. I could not find this in POSIX nor in the bash manual, but it seems true: Assignment to PATH clears the hash table. (Even when the value stays the same). Which could be an argument for clearing the hash table after 'commmand -p ...'. Though, preferably, the hash table is not modified in this case. A side note. The description, in the manual, on 'command' states: Run command with args suppressing the normal shell function lookup. Only builtin commands or commands found in the PATH are executed. ... I think the hash lookup should be mentioned, as the hash lookup is done before the builtin/PATH search, e.g.: PATH=/dev/null hash -p /bin/hostname foo command foo # executes /bin/hostname -- Regards, Mike Jonkmans This message was brought to you thanks to DST.
Re: hash not restored after running command -p
On Sun, Oct 31, 2021 at 03:33:07PM +0300, Oğuz wrote: > On Sun, Oct 31, 2021 at 2:15 PM Mike Jonkmans wrote: > > PATH=/dev/null > > command -p hostname > > hostname # executes /bin/hostname via the hash table > > > > I agree with OP that the behaviour is a bug, or at least unwanted behaviour. > > I'd say it's a feature, and a good one too. Not having to prefix each > invocation of a utility with `command -p ' is a convenience if there's > a lot of them. It might indeed be convenient. Using the hash as alias for commands, that are not in your PATH, seems risky though. > > I could not find this in POSIX nor in the bash manual, but it seems true: > > Assignment to PATH clears the hash table. > > (Even when the value stays the same). > > It's in XCU 2.9.1.4: Oops, missed that. Thanks. It is indeed there, under 1.e.i in https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01 -- Regards, Mike Jonkmans
Re: hash not restored after running command -p
On Sun, Oct 31, 2021 at 05:23:03PM +0200, Oğuz wrote: > 31 Ekim 2021 Pazar tarihinde Mike Jonkmans yazdı: > > > > Using the hash as alias for commands, that are not in your PATH, > > seems risky though. > > Why? Risky how? Risky, mostly on a cognitive level. 'command -p foo' doesn't look like it has a side effect. E.g. assume you have /usr/local/bin/foo and /bin/foo PATH=/usr/local/bin command -p foo foo You would not think that /bin/foo is executed for foo, given the value of PATH. It could also lead to bugs. E.g. PATH=/usr/local/bin source script foo When the 'command -p foo' is introduced in the sourced script, foo changes. -- Regards, Mike Jonkmans
Re: hash not restored after running command -p
On Sun, Oct 31, 2021 at 08:36:49PM +0300, Oğuz wrote: > On Sun, Oct 31, 2021 at 7:07 PM Mike Jonkmans wrote: > > > > On Sun, Oct 31, 2021 at 05:23:03PM +0200, Oğuz wrote: > > > 31 Ekim 2021 Pazar tarihinde Mike Jonkmans yazdı: > > > > > > > > Using the hash as alias for commands, that are not in your PATH, > > > > seems risky though. > > > > > > Why? Risky how? > > > > Risky, mostly on a cognitive level. > > > > 'command -p foo' doesn't look like it has a side effect. > > How does a command look like it has side effects? Changing PATH would be well known to have side effects. Also, i wouldn't be surprised of '/bin/rm -rf /bin' influencing things. > > E.g. assume you have /usr/local/bin/foo and /bin/foo > > PATH=/usr/local/bin > > command -p foo > > foo > > > > You would not think that /bin/foo is executed for foo, given the value of > > PATH. > > I suppose I would not, because the manual doesn't say `command -p' has > this effect. I'll give you that. POSIX is also silent on this. > > It could also lead to bugs. > > E.g. > > PATH=/usr/local/bin > > source script > > foo > > > > When the 'command -p foo' is introduced in the sourced script, foo changes. > > Yes, that's how `source' works; contents of the sourced file alters > the sourcing application's execution environment. In addition to > calling `command -p foo', if the sourced file changes `PATH', or > defines a function named `foo', or enables `expand_aliases' and > defines an alias named `foo', `foo' 'changes'. Note that it is not always the same person writing the code. The script might be written/changed by someone unaware of the side effects. I assume script writers would know about PATH and alias effects, less so about 'hash' and even fewer people would know about 'command -p'. -- Regards, Mike Jonkmans
Re: hash not restored after running command -p
On Mon, Nov 01, 2021 at 09:09:19AM +0300, Oğuz wrote: > On Sun, Oct 31, 2021 at 10:26 PM Mike Jonkmans wrote: > > POSIX is also silent on this. > > I think ``Once a utility has been searched for and found [...], an > implementation may remember its location and need not search for the > utility again unless the PATH variable has been the subject of an > assignment'' pretty much covers it. This wording does not cover it wholly, in my opinion. Because when the utility's hashed path is not in $PATH, then the utility should not have been searched for at all. It should not be found, even if it is remembered. The way it works now, is that the hashed keys are made into aliases. These hash-aliases circumvent PATH search. It is not specified by POSIX and I think it is an unwanted trap. > `command -p command_name' doesn't > alter `PATH', and the shell may or may not remember the location of > `command_name' obtained using the default value for `PATH'. Implicitly this sets PATH. So it should not mess with the outcome of later PATH searches. > > Note that it is not always the same person writing the code. > > The script might be written/changed by someone unaware of the side effects. > > I assume script writers would know about PATH and alias effects, > > less so about 'hash' and even fewer people would know about 'command -p'. > > Yeah, it'd be better if the feature in question were documented. Best would be to only return hash matches that are in PATH. Second best would be to let 'command -p' not mess with the hash. Third best is document current behaviour. -- Regards, Mike Jonkmans
Re: hash not restored after running command -p
On Mon, Nov 01, 2021 at 12:21:41PM +0300, Oğuz wrote: > On Mon, Nov 1, 2021 at 10:58 AM Mike Jonkmans wrote: > > This wording does not cover it wholly, in my opinion. > > Because when the utility's hashed path is not in $PATH, > > then the utility should not have been searched for at all. > > It should not be found, even if it is remembered. > > Is the rest of this paragraph your opinion too or did I miss where the > standard/bash manual says anything to this effect? Everything is opinion, in my opinion. ;) I think that you miss that remembering does not equal using. > > The way it works now, is that the hashed keys are made into aliases. > > These hash-aliases circumvent PATH search. > > It is not specified by POSIX and I think it is an unwanted trap. > > POSIX does not mandate that the directory portion of the pathname used > for executing a command be found in the current value of `PATH' > either. Perhaps this calls for an clarification request. Surely. > > Implicitly this sets PATH. > > So it should not mess with the outcome of later PATH searches. > > Again, nowhere it is said that `command -p' involves modifying `PATH'. Temporarily using a default value of PATH is akin to modifying it. -- Regards, Mike Jonkmans
Re: hash not restored after running command -p
On Mon, Nov 01, 2021 at 03:42:11PM +0700, Robert Elz wrote: > ... > which is how it should be - the hash table is intended to speed PATH > searches for commonly used commands, nothing should be found there > which wouldn't be found from a PATH search - with the sole exception > that the shell isn't required to check that the filesystem hasn't > changed out from under it (eg: putting a new "ls" in some directory > mentioned in PATH before /bin - if that happens, the user needs to > inform the shell, using "hash -r" (or possibly, an assignment to PATH). Is this 'sole exception' in POSIX? It is what bash currently does. Preferably the whole hash/remembering would be transparent, only an implementation speedup. Another case to think about: # Having /usr/local/bin/ls executable PATH=/usr/local/bin:/bin hash -p /usr/local/bin/ls ls ls # /usr/local/bin/ls rm /usr/local/bin/ls ls How should the last 'ls' be handled? Bash calls /bin/ls - which make the hash transparent. Good. It might also have failed with '/usr/local/bin/ls: No such file or directory'. > This is also somewhat inconsistent in bash, if one does, with the > same setup as above, but without having done the "command -p ls", > then: > PATH=/bin ls > > the ls command works, "ls" is found by the PATH search, but is not added > to the hash table (which is correct). A subsequent "ls" correctly > says: > bash: ls: command not found > > If it is to be considered correct to add the entry in the command -p > case, why would it not be correct in this case as well? The only > difference is where the (temporary) modified PATH comes from. Good point! > ps: should this be discussed for clarification in POSIX, I have no doubt > which way the result would go. There is the issue of 'command -p ...' changing the hash table. Another issue is, the usage of hashed values that are not based in the current PATH. This could be seen as a root cause for the 'command -p ...' problem. Not sure what the POSIX people would say about that. -- Regards, Mike Jonkmans
Re: Bash not escaping escape sequences in directory names
On Fri, Jan 21, 2022 at 03:29:47PM -0500, Chet Ramey wrote: > On 1/21/22 1:43 AM, Lawrence Velázquez wrote: > > > Personally, I would be > > less than pleased if my whole terminal turned red just because I > > changed into a directory that happened to have a weird name. > > A mild annoyance at best, don't you think? Mostly an annoyance, but it has potential to be a security issue. Regards, Mike Jonkmans
Re: bracketed paste on 4.4.20(1)-release
On Fri, May 06, 2022 at 02:16:55PM -0500, Sergei Gerasenko wrote: > Hello, > > I’m seeing strange behavior in bash on CentOS 8. The version of bash there is > 4.4.20(1). The 4.2.46(2) version on CentOS 7 doesn’t have this problem. The > versions of readline are 6.2 and 7.0 respectively. The problem is that when > vi-mode is on and bracketed paste is enabled in .bashrc with: > > bind 'set enable-bracketed-paste on’ > > … pasting while in the INSERT mode will result in: a bell + “(arg: 200)” > > I suspect this is because the bracketed paste escape sequence has 200 in it. > Namely, "ESC [ 200 ~”. Is this a known bug and is there a workaround? The > readline version on C8 is readline-7.0. > > Thanks for any ideas! > Sergei Fragments from my ~/.inputrc : set enable-bracketed-paste on set keymap vi-command "\e[200~": bracketed-paste-begin ## The initial escape is eaten, when pasting in insert mode: "[200~": bracketed-paste-begin ## Start with the last 'set keymap' mode set keymap vi-insert "\e[200~": bracketed-paste-begin Hope that it helps. -- Regards, Mike Jonkmans
Re: Bash 5.2 breaks our Arch Linux devtools
On Wed, Feb 15, 2023 at 02:29:29PM +0100, Tobias Powalowski via Bug reports for
the GNU Bourne Again SHell wrote:
I once had that same issue:
https://lists.gnu.org/archive/html/bug-bash/2015-10/msg00013.html
Conclusion: the shopt is not yet working until the function has been run.
(or after the line has been parsed).
> ,,,
> % bash <<'EOF'
>
> function x() {shopt -s extglobmapfile -t packages < <(printf "%s\n"
> "dir"/"dir"/"dir"/"{targetname}"-"${targetver}"-"${targetarch}".pkg.tar?(.!(sig|.*)))shopt
> -u extglob}xEOFResults in:
>
> bash: line 3: syntax error near unexpected token `('
>
> bash: line 3: `mapfile -t packages < <(printf "%s\n"
> "dir"/"dir"/"dir"/"{targetname}"-"${targetver}"-"${targetarch}".pkg.tar?(.!(sig|.*)))'
>
> Putting shopt options outside the function, but is this the wanted
> behaviour?
> ...
> Tobias Powalowski
--
Regards, Mike Jonkmans
Re: $SECONDS and timeout values use realtime `gettimeofday()`
On Wed, Mar 29, 2023 at 10:49:00AM -0400, Chet Ramey wrote: > On 3/24/23 6:19 PM, William Kennington wrote: > > Not worried, it is legitimately a problem for us to the point we are > > currently patching bash to use the monotonic clock for seconds. It times > > out stuff in our boot process which executes normally after the time > > sync has completed and jumped the clock forward. Rght now, we are just > > patching bash until we can fix all of our uses of $SECONDS. I don't > > really have a preference for how this gets exposed (updating SECONDS to > > use the monotonic timer, adding a new variable like MONOSECONDS, or some > > other call to get monotonic time via clock_gettime). > > Interesting. I was wondering how you got onto this, so I looked around for > any utilities (e.g., timeout, sleep), library functions, or system calls > that use the monotonic clock and couldn't find any. It must be too `new'; > everyone seems to use the realtime clock. > > I'll take it as a feature request. My earlier post, in help-bash: https://lists.gnu.org/archive/html/help-bash/2020-06/msg00031.html refers to a website pointing out problems with gettimeofday. That website was (presumably) written in 2010/09 : https://blog.habets.se/2010/09/gettimeofday-should-never-be-used-to-measure-time.html So it is not that new. It is another special case in code, which may explain the slow uptake in tools, many of which need to be portable. -- Regards, Mike Jonkmans
Re: Bash Reference Manual Type
On Fri, Mar 31, 2023 at 06:30:00PM +0200, Martin Schulte wrote: > Hi Chet! > > > Thanks for the report. The synopsis should read > > > > cd [-L|[-P [-e]]] [-@] [dir] > ^ ^ > But aren't these two brackets just superfluous? Then -L would get the -e too. -- Regards, Mike Jonkmans
Re: IFS field splitting doesn't conform with POSIX
On Sat, Apr 01, 2023 at 03:27:47PM -0400, Lawrence Velázquez wrote: > On Fri, Mar 31, 2023, at 2:10 PM, Chet Ramey wrote: > > kre filed an interpretation request to get the language cleaned up. > > For those who might be interested: > > https://austingroupbugs.net/view.php?id=1649 Thanks for the link. And well done, kre! -- Regards, Mike Jonkmans
hash -d 0/1 and other hash stuff
The following 'hash -d' statements have different exit statuses.
bash --noprofile --norc -c \
'hash -d ls; echo $?' ## 0
bash --noprofile --norc -c \
'cat /dev/null; echo $?' ## 1
This depends on a hashable command being given (e.g. cat https://pubs.opengroup.org/onlinepubs/9699919799/utilities/hash.html)
only has -r as option.
FWIW:
https://github.com/search?q=language%3AShell+content%3A%2F%28%5E%7C%5B%5B%3Ablank%3A%5D%5D%29hash%5B%5B%3Ablank%3A%5D%5D-d%2F+NOT+zsh&type=code
Simple patch attached, making 'hash -d ls' exit with 1 in the first case also.
But ...
Questions
1) Why is 'hash -d x' exiting with 1 at all? If the goal is to `unhash x',
'hash -d x' will do the job. Even if x was not hashed at all.
But I am not sure that such a change is feasible (backwards compatible).
2) 'hash x' and 'hash -l x' seem to do the same
(i.c. path search and add to hash if found).
It would be more useful to let 'hash -l x' give the 'hash -l' output for x.
Moreover, I find it unexpected that 'hash -l x' adds x to the hash.
A quick look on github, shows no use of 'hash -l x',
so changes might be ok.
'hash -lt x' outputs what I would have expected 'hash -l x' to do.
Note that 'hash -lt x' doesn't add x to the hash (unexpected).
3) These option combinations should error/exit 1:
Ignoring the '-p path' part:
hash -p path
hash -p path -t name
hash -p path -l
hash -p path -r
This adds name, using path, but ignores the other option:
hash -p path -l name
hash -p path -d name
Using -t ignores -d:
hash ls; hash -td ls
And -d ignores -l:
hash ls; hash -ld ls
Conclusion
I would say that:
- Combining options should error;
- Except for an added -r, which first clears;
- 'hash -l x' shouldn't add, but print what 'hash -lt x' currently prints;
- 'hash -d x' always exits 0, quietly.
Let me know if any of this stuff is wanted, I think I can implement this.
--
Regards, Mike Jonkmans
hashcmd.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git i/hashcmd.c w/hashcmd.c
index ebb92346..62e080be 100644
--- i/hashcmd.c
+++ w/hashcmd.c
@@ -66,9 +66,12 @@ phash_remove (const char *filename)
{
register BUCKET_CONTENTS *item;
- if (hashing_enabled == 0 || hashed_filenames == 0)
+ if (hashing_enabled == 0)
return 0;
+ if (hashed_filenames == 0)
+return 1;
+
item = hash_remove (filename, hashed_filenames, 0);
if (item)
{
Re: hash -d 0/1 and other hash stuff
On Mon, Sep 11, 2023 at 03:25:01PM -0400, Chet Ramey wrote: > On 9/6/23 10:55 AM, Mike Jonkmans wrote: > > The following 'hash -d' statements have different exit statuses. > > > > bash --noprofile --norc -c \ > > 'hash -d ls; echo $?' ## 0 > > bash --noprofile --norc -c \ > > 'cat /dev/null; echo $?' ## 1 > It's whether the hash table is empty. `hash -d' shortcuts if the hash > table is empty, but I agree that it should not. Was the supplied patch ok? > > Questions > > > > 1) Why is 'hash -d x' exiting with 1 at all? If the goal is to `unhash x', > > 'hash -d x' will do the job. Even if x was not hashed at all. > The parallel is with `unalias', which does the same thing if the alias > being removed isn't in the alias table. It is certainly useful to see a warning message (in case of typos). Changing the exit status to 0 may not be worth the trouble. > > 2) 'hash x' and 'hash -l x' seem to do the same > Yes. The `-l' option is an output format modifier; it takes -t to print > the hashed commands. It does not take -t per se, see `hash -l'. The `hash' and `hash -l' output are different, though. Adding a name should keep these differences. > > 'hash -lt x' outputs what I would have expected 'hash -l x' to do. > > Note that 'hash -lt x' doesn't add x to the hash (unexpected). > The `-t' option says to list the contents of the hash table; the `-l' > option modifies the output format. Yet `hash -t' errors, while `hash -l' lists all. Instead `hash -t' could be made to iterate over all names in the hashtable (prepending the name, even if there is only one command). No error on empty. `help hash' states hash: hash [-lr] [-p pathname] [-dt] [name ...] ... -ldisplay in a format that may be reused as input ... -tprint the remembered location of each NAME, ... Arguments: NAME Each NAME is searched for in $PATH and added to the list of remembered commands. This leaves open the question as to whether NAME is added in case of -t. In the synopsys, what does the separate placement of [-lr] and [-dt] mean? > > 3) These option combinations should error/exit 1: > Agreed, except for > > hash -p path -t name > which is valid, if ambiguous. `hash' prioritizes -t over -p in this case. Why would you want to make an exception for -t? > > This adds name, using path, but ignores the other option: > > hash -p path -l name > Because it's not listing output, and -l is an output modifier. > > hash -p path -d name > This is, again, ambiguous, and hash prioritizes -p over -d. > > Using -t ignores -d: > > hash ls; hash -td ls > Ambiguous; hash prioritizes -t over -d. > > And -d ignores -l: > > hash ls; hash -ld ls > Because -l is an output format modifier. It is the current implementation doing the prioritizing. Nowhere is this prioritization mentioned. I don't think that any option should be ignored. Nor should ambiguous commands be executed. An immediate exit, signals the developer to, get coffee, read the manual, and fix their code. Which presupposes that the manual is correct ;) -- Regards, Mike Jonkmans
Re: hash -d 0/1 and other hash stuff
On Wed, Sep 13, 2023 at 11:32:36AM -0400, Chet Ramey wrote: > On 9/12/23 8:28 AM, Mike Jonkmans wrote: > > On Mon, Sep 11, 2023 at 03:25:01PM -0400, Chet Ramey wrote: > > > On 9/6/23 10:55 AM, Mike Jonkmans wrote: ... > > > > 3) These option combinations should error/exit 1: > > > Agreed, except for > > > > hash -p path -t name > > > which is valid, if ambiguous. `hash' prioritizes -t over -p in this case. > > Why would you want to make an exception for -t? > What exception? "Agreed, except for" :) > > It is the current implementation doing the prioritizing. > This is a meaningless statement: it's always the implementation that does > the prioritizing. Which means that is not designed/documented as such. As stated in the immediate next sentence: > > Nowhere is this prioritization mentioned. > > I don't think that any option should be ignored. > > Nor should ambiguous commands be executed. > I think it's better to do something useful. I offered to write a patch. So I needed to find out, what you would accept. But back to usefulness it is. -- Regards, Mike Jonkmans
build disabling alias (patch)
On devel (94bce520): ./configure --disable-alias; make chokes on pcomplete.c: pcomplete.c:371:9: error: ‘alias_list’ undeclared (first use in this function); did you mean ‘alias_t’? 371 | free (alias_list); Easy fix attached. This statement has been introduced 12 years ago in devel (10 years ago in master, bash 4.3). Since then nobody disabled aliases or took the trouble of reporting. Wouldn't it be opportune to remove the `--enable-alias' option? Saves a couple of #ifdef's too. -- Regards, Mike Jonkmans pcomplete.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git i/pcomplete.c w/pcomplete.c index e6c21600..aa6a0440 100644 --- i/pcomplete.c +++ w/pcomplete.c @@ -365,10 +365,10 @@ it_init_aliases (ITEMLIST *itp) sl->list[n] = (char *)NULL; sl->list_size = sl->list_len = n; itp->slist = sl; -#else - itp->slist = (STRINGLIST *)NULL; -#endif free (alias_list); +#else + itp->slist = (STRINGLIST *)NULL; +#endif return 1; }
times posix special builtin
Hi,
The 'times' utility is not listed as a posix special builtin
in doc/bashref.texi
It can be fixed by changing:
@w{shift trap unset}
to:
@w{shift times trap unset}
--
Regards, Mike Jonkmans
posix command search and execution
Hi,
I have some remarks/questions on POSIX Command Search and Execution,
related to bash and some to POSIX itself.
Introduction
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01
describes what to do when a simple command name needs to be resolved.
A rationale is in:
https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_09_02
POSIX Command Search
According to these docs (what I make of it), resolving is done in steps,
the first applicable step is used:
1) No slash in the name.
1a) Run the builtin if the name matches that of a Special Builtin.
1b) List several names that have unspecified results.
1c) Use a function, for functions not matching standard utilities.
1d) Lists 20 fixed utility names (like alias, cd etc.) that are
to be invoked at his point. No PATH search yet.
These are the `regular builtins'.
(These need not exist as builtin).
1e) Search via PATH.
1eI) Search is successful.
1eIa) Check for `regular builtins' and functions
and invoke that regular builtin/function.
Q: Shouldn't this specify an ordering for builtins/functions?
Q: Why check for regular builtins? That was already done in 1d.
1eIb) Run the utility.
(This is where ordinary builtins should run).
(It seems logical that a builtin takes precedence over PATH).
1eII) Nothing in PATH, exit with 127
2) Slash in name? Try to execute; prescribed exits: 127 or 126.
I hope I have understood POSIX correctly on these points.
Bash Command Search
When not in posix mode, bash does:
- Ignore 1a, 1b, 1d.
- 1c) Just use the function. (Especially masking standard utilities).
- 1e) With 1eIa & b use builtins even if utility is not found in PATH.
Which is has a quite logical order: function, builtin, PATH.
In posix mode, it seems that bash:
- 1a) Honors special builtins (but see 1c).
- 1b) With source as a special builtin - which is ok (as unspecified).
- 1c) Doesn't allow you to define a function with the name
of a special builtin.
A function defined before `set -o posix' will mask a
special builtin. (This seems to be ok).
Utilities are masked by a function of that name.
- 1d) All names in the list, except newgrp, are bash-builtins
and are used.
- 1eIa)
Functions don't need a successful path search - per 1c.
- 1eIa & b)
Builtins are also ran regardless of path search.
- *) ok.
Regarding source and 1c in bash, a mail from our beloved maintainer:
https://lists.gnu.org/archive/html/bug-bash/2014-03/msg00084.html
Thus in posix mode, bash does not follow this part of the standard.
But should it?
I would rather have POSIX modified to *also* accept the, more logical,
bash way (i.c. first matching functions, then builtins, then PATH).
Would that be a feasible modification to suggest to the Austingroup?
Remarks
- Regarding 1eIb.
The shells posh, dash, ksh and zsh
also run builtins, even when not found in PATH.
Checked with the `test' builtin (mv /usr/bin/test{,.sav})
on the versions found on Ubuntu 22.04.
- The 'newgrp' utility (mentioned in 1d) is not a builtin in bash.
This is ok. The regular builtins from 1d need not be provided. See:
https://lists.gnu.org/archive/html/bug-bash/2005-02/msg00129.html
Builtins are defined in:
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_83
Q: Isn't that incorrect in stating where regular builtins are defined?
- In bash's code builtins/builtins.c (made from builtins/mkbuiltins.c),
the regular builtins are flagged with POSIX_BUILTIN.
But `hash' is not. Omission or intention?
Also, I don't see any use of the POSIX_BUILTIN flag. Remove?
POSIX Definitions
- A utility is mostly a builtin or executable:
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_22
- Utilities:
https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html
Q: Where is `standard utilities' defined - as used in 1d.
- The POSIX definition of `regular builtin' is the same as those in 1d.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_06
--
Regards, Mike Jonkmans
Re: posix command search and execution
Thanks for the answers, Chet.
On Mon, Nov 06, 2023 at 02:28:24PM -0500, Chet Ramey wrote:
> On 11/6/23 10:48 AM, Mike Jonkmans wrote:
>
> > Q: Why check for regular builtins? That was already done in 1d.
> Implementations can provide other builtins. The check in 1d is only for
> those specific ones.
It was my earlier understanding that POSIX partitions the builtins into:
- special builtins
- regular builtins (listed in 1d).
- ordinary builtins (i.c. not the two others)
Because the first two are specific lists.
XBD definition of builtins:
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_83
(Note that this definition says that 'regular builtins are defined
in detail in XCU Command Search and Excecution'
which makes one think that the regulars are the list of 20 in 1d).
And the table of regular builtins:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tagtcjh_18
(which is mostly the same as the list in 1d - except for the XSI stuff).
Upon rereading those, I think it is more like:
- special builtins
- regular builtins listed in 1d
- regular builtins
> > 1eIb) Run the utility.
> > (This is where ordinary builtins should run).
> > (It seems logical that a builtin takes precedence over PATH).
> You'd be surprised. Note that this seems to require that you can only run
> a builtin if it exists (or something with that name exists) in $PATH.
Yes. that is weird.
> Look at https://www.austingroupbugs.net/view.php?id=854 for a discussion
> of this issue.
Thanks for the link, I find that very hard to read though.
> > 1eII) Nothing in PATH, exit with 127
> So if you have a builtin that doesn't exist in $PATH and isn't listed as
> one of the regular builtins, what do you do? Even the unspecified list
> doesn't give much help.
Interpretation depends on the definition of regular builtins.
Which IMHO is not well specified.
> > I hope I have understood POSIX correctly on these points.
False hope ;)
> > In posix mode, it seems that bash:
...
> >A function defined before `set -o posix' will mask a
> > special builtin. (This seems to be ok).
> It will not, at least not while posix mode is enabled. If you mean that a
> function will be executed before a special builtin when not in posix mode,
> you are correct, because there are no special builtins when not in posix
> mode.
I was mistaken. Guess I toggled the posix option too much.
...
> > Thus in posix mode, bash does not follow this part of the standard.
> Exactly which part of the standard are you saying bash is not following?
That would be PATH search failing and executing a builtin.
Though i think that standard utilities *must* be in PATH,
otherwise there is no conformance a priori.
> The requirements concerning PATH search and builtins are different in the
> next version of POSIX, the result of interp 854. The standard already
> says this about functions with the same name as a special builtin:
>
> "The function is named fname; the application shall ensure that it is a name
> (see XBD Name) and that it is not the name of a special built-in utility."
Nice.
> > But should it?
> > I would rather have POSIX modified to *also* accept the, more logical,
> > bash way (i.c. first matching functions, then builtins, then PATH).
> > Would that be a feasible modification to suggest to the Austingroup?
> I think the resolution to interpretation 854 addresses this. Shells
> who want this odering just declare all the builtins they implement as
> `intrinsic' so they're not subject to a PATH search. That way there's no
> difference between the regular builtins and the ones an implementation
> chooses to provide. It still leaves posix special builtins, but I think
> those are with us forever.
They should be named 'forever builtins', like 'forever chemicals'.
It's a real shame. There certainly are use cases for overriding the
special builtins (e.g. logging around `.' a.k.a. source).
> > - Regarding 1eIb.
> >The shells posh, dash, ksh and zsh
> >also run builtins, even when not found in PATH.
> >Checked with the `test' builtin (mv /usr/bin/test{,.sav})
> >on the versions found on Ubuntu 22.04.
> Yes, this is part of the discussion of interp 854. The business of running
> builtins other than the ones listed only after a PATH search was always
> ahistorical.
Hmm, my check maybe incorrect.
Removing the standard utility `test' is not conformant.
Then again, is there a requirement for the standard utilities to be
found in the current PATH? Or do they just need to be present somewhere.
> > - The 'newgrp' utility (mentioned in 1d) is not a builtin in bash.
Re: posix command search and execution
On Tue, Nov 07, 2023 at 09:39:33AM +0700, Robert Elz wrote: > Date:Mon, 6 Nov 2023 14:28:24 -0500 > From:Chet Ramey > Message-ID: <0ab6075e-22bf-43cd-992c-b2476f626...@case.edu> > > | On 11/6/23 10:48 AM, Mike Jonkmans wrote: > | > According to these docs (what I make of it), resolving is done > | > in steps, the first applicable step is used: > This is one of the most debated, and stupidest, parts of posix. Unneeded complexity, I would say. > | > 1b) List several names that have unspecified results. > | This is an ad-hoc list of builtins that shells implement, > | not necessarily common across all shells. > If it were just builtins it would not be important, the issue > is more that some shells implement some of that list as reserved > words, or aliases, and if that's done what applications can do > alters dramatically. So avoiding using those words as command > names, except when using the known features of a specific shell, > is the best way to remain portable. Since we are dealing with 'Simple Commands', I hadn't yet even considered reserved word and aliases. Makes sense though. > | > 1c) Use a function, for functions not matching standard utilities. > No, that's not what it says, it is except of standard utilities > implemented as functions. More on that below. I see the nuance. Thanks for pointing that out. So a function is called, unless it is provided by the implementation and matches a standard utility. In particular, a user function with the name of a standard utility, will be called at this point. > | > 1d) Lists 20 fixed utility names (like alias, cd etc.) that are > | > to be invoked at his point. No PATH search yet. > | > These are the `regular builtins'. > In the next standard the ones listed are the intrinsic builtins, > and includes only those that must be builtin to work. But > implementations can add more to the list. Chet mentioned that. But I find the Austin-discussion hard to read. It makes sense to partition the builtins in three categories with a separate name for each. > | > 1eI) Search is successful. > | > 1eIa) Check for `regular builtins' and functions > | > and invoke that regular builtin/function. > | > Q: Shouldn't this specify an ordering for builtins/functions? > | The text seems to imply that you can't have both, doesn't it? > While I suppose you could have both, it would be very unusual. Unusuality sketch: - the shell provides a builtin for a standard utility - the distributor provides a function for the same utility in /etc/profile (maybe to mitigate some security issue) Are scripts in /etc/profile considered part of the implementation? ... > | My feeling, without testing anything, is that most shells would allow > | functions to override builtins here. > Since I have never seen any shell implement any standard utility > as a function, it would be very hard to test. Further if the > did, also implementing the same thing as a builtin would be > even harder to imagine - why do it twice when one of the two > would never be used? So not just hard to test - probably > impossible. > > It is also unclear to me why anyone would ever implement a standard > builtin as a function - implementing builtins is simpler for the > implementation than functions (in my experience anyway) and in > any case, if the rules in the standard are followed, there is > no way (except possibly by using "command", and even that is not > clear to me) to tell if the implementation used a function or > a builtin (maybe the output from type might make it clear, but > not necessarily). It is all kind of theoretical. What wonders me is that the POSIX specifications and definitions sometimes are imprecise or lacking. > | This has been an area of significant disagreement. > It has indeed. Agreed to disagree. > | > 1eIb) Run the utility. > | > (This is where ordinary builtins should run). > | > (It seems logical that a builtin takes precedence over PATH). > | You'd be surprised. > Yes. But almost all shells implement it that way, so the > seemingly logical assumption is mostly backed by experience. Again this is not too precise for a standard. > | Note that this seems to require that you can only run > | a builtin if it exists (or something with that name exists) in $PATH. > A builtin for a standard utility, yes. Unless the implementation has > defined it as intrinsic (which the forthcoming standard allows, but > discourages). Applications (which includes users) who invoke non > standard utilities are stepping outside the standard, so get > unspecified res
Re: posix command search and execution
On Tue, Nov 07, 2023 at 11:49:25AM -0500, Chet Ramey wrote: > On 11/7/23 8:54 AM, Mike Jonkmans wrote: ... > > > Look at https://www.austingroupbugs.net/view.php?id=854 for a discussion > > > of this issue. > > Thanks for the link, I find that very hard to read though. > It's also incomplete; there was a lot of discussion on the mailing list. > I don't have a link to a usable public mailing list archive. So the discussion is hidden. Hmm. I already did not find it much of a discussion in terms of opposition. ... > > Then again, is there a requirement for the standard utilities to be > > found in the current PATH? Or do they just need to be present somewhere. > They have to be findable using the value returned by `getconf PATH'. If > the user modifies PATH to, say, prepend directories before that standard > PATH, then all bets are off. I see. Weirdly on Ubuntu 22.04, with /bin symlinked to /usr/bin, `getconf PATH' produces `/bin:/usr/bin'. That looks like a recipe for redundant `stats'. > > > > - The 'newgrp' utility (mentioned in 1d) is not a builtin in bash. > It's gone in the latest draft of the next version of the standard anyway. Good riddance. > > > > - Utilities: > > > > https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html > > > > Q: Where is `standard utilities' defined - as used in 1d. > > > These are the standard utilities. > > Some of these utilities are marked with optional `codes'. > > Are these also considered standard utilities - even when the option is > > not true? > Not really, no. If the implementation claims to support, for instance, XSI, > the XSI-shaded utilities have to be present and they have to behave as > specified. If the implementation doesn't, they don't. > https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap02.html#tag_02_01_04 > > So in 1d, if the system doesn't claim XSI conformance, the shell doesn't > have to include type or ulimit in this required invocation order. > (But wait! The list of intrinsics in the latest draft includes type and > ulimit, and isn't XSI-shaded. So that will change.) Isn't that described in `Note 0004803' in https://www.austingroupbugs.net/view.php?id=854 (not possible to add shade to type and ulimit in column order) -- Regards, Mike Jonkmans
Re: posix command search and execution
On Wed, Nov 08, 2023 at 10:38:19AM -0500, Chet Ramey wrote: > On 11/7/23 5:04 PM, Mike Jonkmans wrote: > > It depends on your requirements. If you want something that is easy to > explain to users, you want to reduce the distinction between `intrinsics' > and `regular builtins' to something as small as possible. That way the > search reduces to > > special builtins* > functions > intrinsics > executables > > in that order, with ways to override each one (except special builtins > in posix mode). That is a good point. The POSIX standard is intended for two classes of readers, implementors and users (application writers). If the users had anything to say, they would probably vote for the bash way. > > >| > 1eI) Search is successful. > > >| > 1eIa) Check for `regular builtins' and functions > > >| > and invoke that regular builtin/function. > > >| > Q: Shouldn't this specify an ordering for builtins/functions? > > >| The text seems to imply that you can't have both, doesn't it? > > > While I suppose you could have both, it would be very unusual. > > Unusuality sketch: > > - the shell provides a builtin for a standard utility > > - the distributor provides a function for the same utility in /etc/profile > >(maybe to mitigate some security issue) > > Are scripts in /etc/profile considered part of the implementation? > Yes. But there's no way to make a distinction between this type of function > and any other user-supplied function, so the distinction isn't useful. A shell could flag a function as belonging to the implementation. Then mark all new functions as such while not done with /etc/profile. Not that I really would like to see that happen, just for argument's sake. > > I already can't find the definition of standard utilities. > The standard utilities are the ones specified in the standard. An implementor will probably know this. But a user might not. The standard should define that. > > >| I think the resolution to interpretation 854 addresses this. Shells > > >| who want this ordering just declare all the builtins they implement > > > as > > >| `intrinsic' so they're not subject to a PATH search. > > > Yes. > > Can this intrinsic list be amended with any user loaded builtins? > You mean enable -f? It's up to the shell implementation. Bash treats > dynamically-loaded builtins the same as any other builtin. That was indeed what I meant. Does the new POSIX version with intrinsics also allow for this? -- Regards, Mike Jonkmans
Re: posix command search and execution
On Wed, Nov 08, 2023 at 11:52:19PM +0700, Robert Elz wrote: > Date:Tue, 7 Nov 2023 23:04:10 +0100 > From: Mike Jonkmans > Message-ID: <20231107220410.gc27...@jonkmans.nl> > > | It makes sense to partition the builtins in three categories with > | a separate name for each. > > That's more or less what has been done now, with the specials, intrinsics, > and regular builtins. A step in the right direction. In 30 years or so we may lose the special builtins ;) > | Unusuality sketch: > | - the shell provides a builtin for a standard utility > | - the distributor provides a function for the same utility > > What counts as "the implementation" is always (apparently) an unspecified > issue, splitting things into 2 camps (as almost every standards org > tends to do, there's "us" and "the rest of them") never works well > in practice, yet continues to be done for all kinds of purposes, as it > always seems to provide a nice clean answer - yet as that shows, never > really does, as what is called what always depends upon from what point > of view you're looking, and there tend to be many. > > But: > | in /etc/profile > in that case it would be not be "provided by the implementation" - anything > loaded from profile files is a user add on (even if it is forced upon the > user by their admins) - if the local user doesn't want that function, they > can simply remove it from /etc/profile (or not use that at all). > > | Are scripts in /etc/profile considered part of the implementation? > So, no, I don't think so. I don't want to start a flame war, but Chet thought it was. :) >From a user/application writer's perspective I would say that it is. Having a work around is not really convincing, it is still part of the implementation. Code -> Distribution -> Admin -> User The `implementation' could be including any up to the first three. > | It is all kind of theoretical. What wonders me is that the POSIX > | specifications and definitions sometimes are imprecise or lacking. > > They sometimes are, often because no-one considered some weird case, > so didn't think to actually write down what happens then - in the POSIX > world that's called implicitly unspecified. Occasionally there are > simple errors (minor ones, like spelling, grammar, punctuation or > formatting issues - which are easy to overlook, though annoying) to > major ones where the standard says what someone thought was correct, > but they were mistaken. > > But it isn't always the case, even when it seems so - to really > understand it, you really have to read and comprehend (almost) the > whole damn thing (all 4000 approx pages of it). The text in one > place often says nothing about something which is specified in some > entirely different context. Writing specifications is not an easy task. Let's try to make things better (for whoever is defining `better'). Seems there are at least a couple of goals for the POSIX standard. - providing direction to implementers - converging implementations - providing an api to users An anti goal would be to keep supporting historical burdens, like special builtins. > | > Yes. But almost all shells implement it that way, so the > | > seemingly logical assumption is mostly backed by experience. > | Again this is not too precise for a standard. > Yes, but the point is that this isn't what the standard says, more like > what it really should say. This PATH search for builtins stuff is an > invention by people trying to force something upon shells that none of > them implemented, because they thought it would be better. Ouch. > It is actually messier that it first seems, if you have a builtin command, > it isn't simply finding that command's name in a PATH search that is > required, but that the directory (from PATH) in which it was found be the > one "associated" with the builtin command (how exactly that is determined, > or what it really even means is not specified anywhere). The effect is > that if a shell with a builtin "test" believes that the associated > directory is /bin and someone's path is /usr/bin:/bin - and there is a > test command in /usr/bin (even if /usr/bin and /bin are the same directory, > or /bin/name and /usr/bin/name are linked and so invoke the same thing) > then the filesystem command is supposed to be invoked rather than the > builtin.There are reasons for that, but they're really fairly stupid. Oh dear. I thought it could not get any worse. > [Finding the standard utilities] > | Good to know. I was wondering about that. > | But where i
Re: posix command search and execution
On Wed, Nov 08, 2023 at 10:42:20AM -0500, Chet Ramey wrote: > On 11/7/23 5:37 PM, Mike Jonkmans wrote: > > On Tue, Nov 07, 2023 at 11:49:25AM -0500, Chet Ramey wrote: > > > On 11/7/23 8:54 AM, Mike Jonkmans wrote: > > So the discussion is hidden. Hmm. > Not hidden; if you look at the open group's mailing list archives, you > can find the messages. I just don't consider the interface usable. It's > barely searchable. Ah, thanks. I'll have a look later. > > > > Then again, is there a requirement for the standard utilities to be > > > > found in the current PATH? Or do they just need to be present somewhere. > > > They have to be findable using the value returned by `getconf PATH'. If > > > the user modifies PATH to, say, prepend directories before that standard > > > PATH, then all bets are off. > > I see. Weirdly on Ubuntu 22.04, with /bin symlinked to /usr/bin, > > `getconf PATH' produces `/bin:/usr/bin'. > > That looks like a recipe for redundant `stats'. > Does that matter? The value getconf returns is static, and is guaranteed > to find all the standard utilities regardless of what the file system > looks like. Maybe it matters. If I am not mistaken, for POSIX compliance, both /bin and /usr/bin have to be in PATH (see quote from Robert). A naïve implementation, doing a PATH search, might be calling `stat' for both directories, whilst they are symlinked. This might cost some performance. [ Just checked: bash stats both. ] Symlinking also meddles with the description below. Relevant quote from Robert's mail: It is actually messier that it first seems, if you have a builtin command, it isn't simply finding that command's name in a PATH search that is required, but that the directory (from PATH) in which it was found be the one "associated" with the builtin command (how exactly that is determined, or what it really even means is not specified anywhere). The effect is that if a shell with a builtin "test" believes that the associated directory is /bin and someone's path is /usr/bin:/bin - and there is a test command in /usr/bin (even if /usr/bin and /bin are the same directory, or /bin/name and /usr/bin/name are linked and so invoke the same thing) then the filesystem command is supposed to be invoked rather than the builtin.There are reasons for that, but they're really fairly stupid. This is more complex than it needs to be (and should be). For a user it is very hard to have any confidence in the posix-ness of their scripts. PS. Remember this `nn'/`Pnews' message? "Your message will cost the net hundreds if not thousands of dollars to send everywhere." What would the cost of all these extra `stat' calls be? ;) -- Regards, Mike Jonkmans
Re: posix command search and execution
On Thu, Nov 09, 2023 at 10:12:06PM +0700, Robert Elz wrote: > Date:Thu, 9 Nov 2023 14:21:35 +0100 > From: Mike Jonkmans > Message-ID: <20231109132135.ga208...@jonkmans.nl> > > | If I am not mistaken, for POSIX compliance, both /bin and /usr/bin have > | to be in PATH (see quote from Robert). > > No, I didn't say that, there are no particular required directory > names, just that PATH needs to include whatever directories contain > all the standard utilities ... getconf() returns that path. > > I used those names just as an example. Ah, that is clear then. On Ubuntu 22.04 `getconf path' returns /bin:/usr/bin and these are symlinked. -- Regards, Mike Jonkmans
Re: posix command search and execution
On Fri, Nov 10, 2023 at 08:07:31PM -0500, Chet Ramey wrote: > On 11/9/23 11:17 AM, Mike Jonkmans wrote: > > On Thu, Nov 09, 2023 at 10:12:06PM +0700, Robert Elz wrote: > > > > On Ubuntu 22.04 `getconf path' returns /bin:/usr/bin > > and these are symlinked. > Then that's an issue to take up with Debian/Ubuntu, right? If they're > symlinked by the distro, then getconf PATH should only return one of them. Yes, that would be more efficient than a shell checking for symlinks in PATH. I'll try to convince them. -- Regards, Mike Jonkmans
Re: posix command search and execution
On Sat, Nov 11, 2023 at 08:31:14AM -0500, Greg Wooledge wrote: > On Sat, Nov 11, 2023 at 09:14:41AM +0100, Mike Jonkmans wrote: > > On Fri, Nov 10, 2023 at 08:07:31PM -0500, Chet Ramey wrote: > > > On 11/9/23 11:17 AM, Mike Jonkmans wrote: > > > > On Thu, Nov 09, 2023 at 10:12:06PM +0700, Robert Elz wrote: > > > > > > > > On Ubuntu 22.04 `getconf path' returns /bin:/usr/bin > > > > and these are symlinked. > > > Then that's an issue to take up with Debian/Ubuntu, right? If they're > > > symlinked by the distro, then getconf PATH should only return one of them. > > > > Yes, that would be more efficient than a shell checking for symlinks in > > PATH. > > > > I'll try to convince them. > > Good luck with that. The problem with "usrmerge" in Debian is that > it's still optional. There are Debian infrastructure systems (virtual > machines especially) that don't use it. Therefore, simply changing > the static answer in confstr(3) across all Debian systems would be > incorrect. They would need to make it check the actual /bin to see > whether it's a symbolic link to usr/bin, and give an answer accordingly. > > It's a lot less work to just leave it as is. Thanks for the warning. I'll leave them alone then. -- Regards, Mike Jonkmans
Re: About `M-C-e` expand result `'` failed
On Wed, Jan 31, 2024 at 04:25:18PM -0500, Chet Ramey wrote: > On 1/30/24 10:59 AM, Zachary Santer wrote: > > On Tue, Jan 30, 2024 at 10:04 AM Andreas Schwab wrote: > > > > > The command is doing exactly what it is documented to do, that is do all > > > of the shell word expansions. > > > > Quote Removal shows up as a subsection of the Shell Expansion section in > > the manual, but it doesn't show up in the list of expansions at the > > beginning of that section.[1] Additionally, we see the following: > > > After these expansions are performed, quote characters present in the > > original word are removed unless they have been quoted themselves (quote > > removal). > > Quote removal is part of the shell word expansion process. POSIX lists it > as one of the word expansions, and it is. The first sentence under EXPANSION may make you think otherwise: Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion. The third sentence, after this, corrects this: After these expansions are performed, quote characters present in the original word are removed unless they have been quoted themselves (quote removal). That sentence could also be placed right after the first. Returning to `shell-expand-line'. As end user I would expect that, running shell-expand-line then accept-line, would do the same as just an accept-line. Quote removal is absolutely needed, otherwise expansions may mess up the result when they introduce quotes. I think that escaping is needed after quote removal in shell-expand-line. -- Regards, Mike Jonkmans
Re: About `M-C-e` expand result `'` failed
On Thu, Feb 01, 2024 at 08:14:34PM -0500, Chet Ramey wrote: > On 1/31/24 6:05 PM, Mike Jonkmans wrote: > > > The first sentence under EXPANSION may make you think otherwise: > >Expansion is performed on the command line after it has been split into > > words. > >There are seven kinds of expansion performed: brace expansion, > >tilde expansion, parameter and variable expansion, command substitution, > >arithmetic expansion, word splitting, and pathname expansion. > > The third sentence, after this, corrects this: > >After these expansions are performed, quote characters present in the > > original > >word are removed unless they have been quoted themselves (quote removal). > > That sentence could also be placed right after the first. > I can rearrange things, but it seems clear already. "There are seven kinds of expansion..." then three sentences later comes quote removal. The reader may wonder whether quote removal is an expansion. That question pops up in the description of shell-expand-line: Expand the line as the shell does. This performs alias and history expansion as well as all of the shell word expansions. Now "... all of the shell word expansions", could be read as the seven kinds of expansion. Maybe it is confusing that a (quote) removal is an expansion. Oh well. > > Returning to `shell-expand-line'. > > As end user I would expect that, running shell-expand-line then accept-line, > > would do the same as just an accept-line. > Why would you expect this? You have deliberately introduced an extra > expansion step, similar to using `eval'. I might expect that, because it would be useful to see what is going to be executed. Then after review of the expansion, I could accept it. But that is not possible, in general, with shell-expand-line, as OP has shown. > > I think that escaping is needed after quote removal in shell-expand-line. > No, that would be another function. Maybe less is more ;) PS. Just found out that `shell-expand-line' does not do pathname expansion. That was unexpected. In the `shell-expand-line' description: '... word expansions.' Does that mean every expansion but pathname expansion? -- Regards, Mike Jonkmans
glob-expand-word and vi-command mode
>From the manual, glob-expand-word: The word before point is treated as a pattern for pathname expansion, and the list of matching filenames is inserted, replacing the word. If a numeric argument is supplied, an asterisk is appended before pathname expansion. [ With a block cursor, `before point' would be left of that. ] [ mkdir test; cd test; touch file1 file2 ] Going into `vi-command' mode on the line `ls *' puts the cursor on the `*'. Then `glob-expand-word' does nothing with the `*', it just inserts a space. Resulting in `ls *' (cursor still on `*'). Expected: nothing happens. Also in `vi-command' mode, line is `ls "*"', cursor on the last `"'. Then `glob-expand-word' produces `ls file1 file2 "'. Munching a quote and adding a space. Expected: `ls "file1 file2". And again in `vi-command' mode with line `ls *1*', cursor on last `*'. Then `glob-expand-word' produces `ls file1 *'. Not sure why this is useful. Note that `ls "f"* ' with cursor on the last space, doesn't expand. Would be a `nice to have' to do quote removal and maybe other expansions. Wow, tilde expansion is done: `ls ~/b*' expands. Cool, but undocumented. I would argue that `glob-expand-word' should expand the word the cursor is in. Then only if the cursor is on whitespace, the `word before point' is used. For `vi-command' mode it might be nice if the cursor didn't move onto the last character. vi-insert mode: `ls *' cursur is after the `*' then going to vi-command mode, the cursor moves onto the `*'. This prevents the `glob-expand-word' from doing its thing. -- Regards, Mike Jonkmans
Re: About `M-C-e` expand result `'` failed
On Fri, Feb 02, 2024 at 10:09:53AM -0500, Chet Ramey wrote: > On 2/2/24 8:12 AM, Mike Jonkmans wrote: > > On Thu, Feb 01, 2024 at 08:14:34PM -0500, Chet Ramey wrote: > > "There are seven kinds of expansion..." then three sentences later comes > > quote removal. The reader may wonder whether quote removal is an expansion. > OK, then how about we enumerate the word expansions without numbering them. Maybe: Several kinds of expansion are performed in order: first brace expansion; then tilde expansion, parameter and variable expansion, command substitution and arithmetic expansion (done in a left-to-right fashion); word splitting; pathname expansion and finally quote removal. Then the alinea about the order and the alinea on quote removal could be removed. > shell-expand-line (M-C-e) > Expand the line by performing shell word expansions. This per- > forms alias and history expansion, $'string' and $"string" quot- > ing, tilde expansion, parameter and variable expansion, arith- > metic expansion, word splitting, and quote removal. See HISTORY > EXPANSION below for a description of history expansion. That form is clearer indeed. But isn't that missing `, process substitution (if available)'? > > Maybe it is confusing that a (quote) removal is an expansion. Oh well. > What is the logic here? Expansion is making things bigger. Removal is more like contraction. > > Then after review of the expansion, I could accept it. > Then undo the editing operation and let the expansions happen all over > again as part of command execution. That is a nice tip. I didn't realize that it was possible to undo. So normally I would avoid using expansions. :) -- Regards, Mike Jonkmans
Re: glob-expand-word and vi-command mode
On Fri, Feb 02, 2024 at 09:50:46AM -0500, Greg Wooledge wrote: > On Fri, Feb 02, 2024 at 03:39:54PM +0100, Mike Jonkmans wrote: > > [ mkdir test; cd test; touch file1 file2 ] > > > > Going into `vi-command' mode on the line `ls *' puts the cursor on the `*'. > > Then `glob-expand-word' does nothing with the `*', it just inserts a space. > > Resulting in `ls *' (cursor still on `*'). > > Expected: nothing happens. > > I'm not sure what keystrokes you're actually using, or what bind calls > you've done leading up to this, but in a vanilla instance of bash with > nothing done except 'set -o vi', typing > > l s space * esc * > > will replace the * with file1 file2 and another space, and also puts > you in insert mode for some reason. Probably historical. esc * is bound to insert-completions (which may be better than glob-expand-word, as it doesn't need a glob) What I did (bash 5.1): INPUTRC=/dev/null bash --norc --noprofile set -o vi bind 'set show-mode-in-prompt on' bind -m vi-command "\C-f": glob-expand-word Then type: l s space * esc ^F Result: ls * (space inserted before the *) Otoh, if I type: l s space * space esc ^F Result: ls file1 file2 space (both stay in command mode) -- Regards, Mike Jonkmans
Re: glob-expand-word and vi-command mode
On Sat, Feb 03, 2024 at 03:43:45PM -0500, Chet Ramey wrote: > On 2/2/24 9:39 AM, Mike Jonkmans wrote: > > From the manual, glob-expand-word: > > glob-expand-word doesn't work that great in vi command mode, mostly for > the reasons you suspect. What made you use it over the standard vi mode > binding for `*' (vi-complete)? That behaves like POSIX says it's supposed > to in vi command mode. I made some mappings on all *-expand-* variations for the discussion on shell-expand-case. The '*' mapping seams more appropiate. But the vi-command mode puts the cursor on the last character, instead of one past it (as it is in vi-insert-mode). That has the effect that '*' does not work easily. If you type, in vi-insert-mode: l s space * files starting with 'ls' will be completed. Expected all files to be inserted. Not sure if that is fixable. -- Regards, Mike Jonkmans
Re: glob-expand-word and vi-command mode
On Sat, Feb 03, 2024 at 04:59:08PM -0500, Chet Ramey wrote: > On 2/2/24 5:15 PM, Mike Jonkmans wrote: > > On Fri, Feb 02, 2024 at 09:50:46AM -0500, Greg Wooledge wrote: > > > On Fri, Feb 02, 2024 at 03:39:54PM +0100, Mike Jonkmans wrote: > > > > [ mkdir test; cd test; touch file1 file2 ] > > > > Going into `vi-command' mode on the line `ls *' puts the cursor on the > > > > `*'. > > > > Then `glob-expand-word' does nothing with the `*', it just inserts a > > > > space. > > > > Resulting in `ls *' (cursor still on `*'). > > > > Expected: nothing happens. > > > I'm not sure what keystrokes you're actually using, or what bind calls > > > you've done leading up to this, but in a vanilla instance of bash with > > > nothing done except 'set -o vi', typing > > > l s space * esc * > > > will replace the * with file1 file2 and another space, and also puts > > > you in insert mode for some reason. Probably historical. > > esc * is bound to insert-completions > > It's bound to vi-complete, which bash replaces with something that does > the pathname expansion that POSIX requires. How can I find this out? INPUTRC=/dev/null bash --norc --noprofile set -o vi bind -m vi-insert -p bind -m vi-command -p Both show no bindings for "*". (`man 3 readline' mentions "*" on vi-complete in command mode) > > (which may be better than glob-expand-word, as it doesn't need a glob) > Well, it appends a `*' if the word doesn't have any globbing chars. Seems logical. > If your goal is to be in vi command mode when the command completes, why > not use a macro? > bind -m vi-command '"\C-f":"*\e" > does that. Clever. -- Regards, Mike Jonkmans
Re: About `M-C-e` expand result `'` failed
On Tue, Feb 06, 2024 at 02:59:21PM -0500, Chet Ramey wrote: > On 2/4/24 2:35 AM, Martin D Kealey wrote: > > > PS: Sadly M-C-r seems to be already taken, so I can't just hop one key over. > > You can rebind it, you know. Anyway, let's assume the existence of a new > bindable command that (shell) quotes the (shell? readline?) words on a > line. Let's assume that this new command is bound to some sequence like > \C-xq. Then you can use a macro to create something like what you want: > > "\M-\C-w":"\M-\C-e\C-xq" > > with each command being separately undoable. Assuming you want M-C-e, C-xq, Return to be a equivalent to a verbose Return. I don't think it is doable that way, please consider: $ printf "<%s>\n" x\ y $ printf "<%s>\n" x\ y ## M-C-e $ printf <%s>\n x y ## C-xq (let's assume it quotes the redirections) $ printf "<%s>\n" x y The original words before quote removal should be requoted. Also when expansion introduces quotes: $ x=\" $ printf "$x\n" ## M-C-e-new (shell-expand-quoted?) $ printf "\"\n" $ printf $x\\n ## M-C-e-new $ printf \"\\n Oh well, I am happy with the undo :) -- Regards, Mike Jonkmans
Re: nameref and referenced variable scope, setting other attributes (was "local -g" declaration references local var in enclosing scope)
On Mon, Mar 18, 2024 at 04:19:55PM -0400, Chet Ramey wrote:
> On 3/14/24 8:57 PM, Zachary Santer wrote:
> > On Thu, Mar 14, 2024 at 3:43 PM Chet Ramey wrote:
> > >
> > > In fact, before 2020, local -p with no name arguments behaved the same as
> > > local without arguments, which just printed all the local variable names
> > > at
> > > the current scope in the form of assignment statements. That was certainly
> > > not usable to reproduce the current state.
> >
> > While we're kind of on the subject, I find it really odd that the
> > ${var@A} parameter expansion will expand to either an assignment
> > statement or a 'declare' command, depending on whether or not the
> > variable has an attribute set.
>
> Yes. There is one thing missing: the transformation should expand to a
> `declare' command when applied to a local variable at the current scope,
> even if there are no attributes to be displayed. Agreed?
>
> I am less convinced about outputting a `-g' for a global variable when
> called from a function scope, but I could be persuaded.
This would be logical.
Forgot the use case, but it was about a year ago that I needed this.
--
Regards, Mike Jonkmans
Re: nameref and referenced variable scope, setting other attributes (was "local -g" declaration references local var in enclosing scope)
On Tue, Mar 19, 2024 at 02:24:34PM -0400, Chet Ramey wrote: > On 3/19/24 11:50 AM, Mike Jonkmans wrote: > > > > Yes. There is one thing missing: the transformation should expand to a > > > `declare' command when applied to a local variable at the current scope, > > > even if there are no attributes to be displayed. Agreed? > > > > > > I am less convinced about outputting a `-g' for a global variable when > > > called from a function scope, but I could be persuaded. > > > > This would be logical. > > Forgot the use case, but it was about a year ago that I needed this. > > The former, the latter, or both? I can also agree to the former. My use case was with needing the '-g'. -- Regards, Mike Jonkmans
Re: Bash-4.2-rc2 available for FTP
On Wednesday, February 02, 2011 08:56:24 Chet Ramey wrote: > The second release candidate of bash-4.2 is now available with the URL > > ftp://ftp.cwru.edu/pub/bash/bash-4.2-rc2.tar.gz - braces.c:mkseq() is using an intmax_t type for the length in the asprintf call when it needs to be an int. a quick check of a 32bit system shows that sizeof(intmax_t) is 8 bytes which means the output most likely will get screwed up (since it'll be interpreted in the C library as 2 arguments). - lib/glob/smatch.c needs externs.h for mbsmbchar. seems like externs.h could do with including bashtypes.h/command.h/general.h too since it needs basic types from all of those. - lib/glob/smatch.c seems its STR defines could be unified with stuff in general.h - seems like lib/sh/snprintf.c should be including some header for isnan and isinf (maybe math.h ?) otherwise, some quick smoke tests show it seems to be working OK so far ... -mike signature.asc Description: This is a digitally signed message part.
Re: Bash-4.2-rc2 available for FTP
On Wednesday, February 02, 2011 21:49:38 Chet Ramey wrote: > On 2/2/11 6:27 PM, Mike Frysinger wrote: > > - lib/glob/smatch.c needs externs.h for mbsmbchar. seems like externs.h > > could do with including bashtypes.h/command.h/general.h too since it > > needs basic types from all of those. > > Or an extern declaration for mbsmbchar, to avoid having to include other > files. that defeats the whole point of having a single extern line. changing the header and matching func definition wouldnt automatically catch random externs sprinkled over the tree and could result in an arbitrarily crashing binary that showed no build errors or warnings. -mike signature.asc Description: This is a digitally signed message part.
Re: Do more testing before a release?
On Wednesday, February 16, 2011 23:51:16 Clark J. Wang wrote: > I know little about open source development process (and control?). I just > don't know where to get the bash code (like CVS, SVN respository) before > it's released. I think it's better to make it open to more people so > everyone can help review and test before a stable release. the 4.2 rc1 was announced on the list and garnered testing/feedback -mike signature.asc Description: This is a digitally signed message part.
empty quotes break pattern replacements in bash-4.2
this simple code no longer works in bash-4.2:
$ f=abc; echo ${f##""a}
abc
same goes for ${f//""a} and ${f%%""c}, and perhaps more operations
removing the quotes, or quoting the single char in question, makes it work:
$ f=abc; echo ${f##a} ${f##"a"}
bc bc
the original bug report uses variables in the pattern and quotes them to avoid
expansion of globs and such. but if the variable happened to be empty, things
no longer worked correctly.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: empty quotes break pattern replacements in bash-4.2
On Friday, February 18, 2011 23:17:11 Chet Ramey wrote:
> On 2/18/11 9:06 PM, Mike Frysinger wrote:
> > this simple code no longer works in bash-4.2:
> > $ f=abc; echo ${f##""a}
> > abc
> > same goes for ${f//""a} and ${f%%""c}, and perhaps more operations
>
> One more: everything that calls getpattern().
>
> > removing the quotes, or quoting the single char in question, makes it
> > work: $ f=abc; echo ${f##a} ${f##"a"}
> > bc bc
> >
> > the original bug report uses variables in the pattern and quotes them to
> > avoid expansion of globs and such. but if the variable happened to be
> > empty, things no longer worked correctly.
>
> Try this patch.
seems to do the trick. i'll have the original reporter (Ulrich Müller) verify
on his end too.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: configure fails with gcc 4.6.0 LTO
On Saturday, March 19, 2011 17:33:36 Chet Ramey wrote: > On 3/18/11 9:42 PM, Zeev Tarantov wrote: > > x86_64-pc-linux-gnu-gcc: error: \: No such file or directory > > x86_64-pc-linux-gnu-gcc: error: \: No such file or directory > > x86_64-pc-linux-gnu-gcc: error: \: No such file or directory > > x86_64-pc-linux-gnu-gcc: error: \: No such file or directory > > lto-wrapper: > > /usr/x86_64-pc-linux-gnu/gcc-bin/4.6.0-pre/x86_64-pc-linux-gnu-gcc > > returned 1 exit status > > /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.0-pre/../../../../x86_64-pc-linu > > x-gnu/bin/ld: lto-wrapper failed > > It would be helpful to see the exact command that caused this error, in > its unexpanded state. it's right above (this is a snippet from config.log). however, i'm not sure looking into this bug report is useful. we already told this guy to stop using gcc-4.6 and using ridiculous CFLAGS/LDFLAGS if he wasnt going to assist in figuring out the bugs. -mike signature.asc Description: This is a digitally signed message part.
Re: configure fails with gcc 4.6.0 LTO
On Mon, Mar 21, 2011 at 8:32 AM, Greg Wooledge wrote: > On Sat, Mar 19, 2011 at 09:52:05PM +, Zeev Tarantov wrote: >> configure:3122: checking for C compiler default output file name >> configure:3144: x86_64-pc-linux-gnu-gcc -g -O2 -flto >> -DDEFAULT_PATH_VALUE='"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"' >> -DSTANDARD_UTILS_PATH='"/bin:/usr/bin:/sbin:/usr/sbin"' >> -DSYS_BASHRC='"/etc/bash/bashrc"' >> -DSYS_BASH_LOGOUT='"/etc/bash/bash_logout"' >> -DNON_INTERACTIVE_LOGIN_SHELLS -DSSH_SOURCE_BASHRC -Wl,-flto >> conftest.c >&5 >> x86_64-pc-linux-gnu-gcc: error: \: No such file or directory >> x86_64-pc-linux-gnu-gcc: error: \: No such file or directory >> x86_64-pc-linux-gnu-gcc: error: \: No such file or directory >> x86_64-pc-linux-gnu-gcc: error: \: No such file or directory >> lto-wrapper: >> /usr/x86_64-pc-linux-gnu/gcc-bin/4.6.0-pre/x86_64-pc-linux-gnu-gcc >> returned 1 exit status > > I'm not sure how much of the distortion we're seeing here is being > caused by a mail user agent, versus how much is caused by the experimental > gcc he's using, etc. > > My guess, without knowing anything about this version of gcc, is that > the command that ./configure is supposed to execute is really something > like: > > x86_64-pc-linux-gnu-gcc -g -O2 -flto \ > -DDEFAULT_PATH_VALUE='"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"' > \ > -DSTANDARD_UTILS_PATH='"/bin:/usr/bin:/sbin:/usr/sbin"' \ > > and so on, where the backslashes are supposed to be at the ends of the > lines, to indicate continuation. And somewhere along the way, they're > being doubled so that they're becoming literal words. > > imadev:/tmp$ /net/appl/gcc-3.3/bin/gcc -g -O2 -DFOO=BAR \\ -DQWERTY=UIOP \\ > hello.c > gcc: \: No such file or directory > gcc: \: No such file or directory > > Kinda like that. > > Maybe it's gcc 4.6-prewhatever that's doing it. Maybe he's actually using > some sort of "build system wrapper" that's broken. I don't know. I just > recognize the symptom, not the cause. the Gentoo ebuild double quotes the flags in env CPPFLAGS so that gcc sets the strings correctly when compiling. the failure only happens when LDFLAGS contains -flto which says to me that gcc doesnt parse arguments the same as when executing other helpers. LDFLAGS=-flto CPPFLAGS=-DD=\'\"\"\' ./configure but still not a bash issue -mike
Re: [Bash-announce] german misspelling
On Wed, Mar 23, 2011 at 2:51 PM, Ralf Thielow wrote: > i've found a german misspelling in command "cd". > Where can i fix it? please use bug-bash, not bash-announce a bit odd announce isnt moderated so only people like Chet can announce ... -mike
Re: RFE: make [[ compatible with [ 'Option'?
On Mon, Mar 28, 2011 at 2:25 PM, Greg Wooledge wrote: > In any case, I see no benefit to changing how [[ works. A change would > just cause more confusion. and probably break many existing scripts -mike
heredocs incorrectly printed in for ((...)) loop
seems to be just like the bug fixed in bash41-006, but with a diff for loop
style. simple example:
f() {
for (( :; :; )) ; do
cat <
signature.asc
Description: This is a digitally signed message part.
Re: heredocs incorrectly printed in for ((...)) loop
On Tuesday, April 19, 2011 16:38:57 Chet Ramey wrote: > > seems to be just like the bug fixed in bash41-006, but with a diff for > > loop style. simple example: > > Try the attached patch and let me know the results. It fixes this > case and a couple of others. sorry for the delay ... that patch does indeed seem to fix the issue. thanks! -mike signature.asc Description: This is a digitally signed message part.
Re: Bash source repository
On Sunday, May 29, 2011 22:18:33 Bradley M. Kuhn wrote: > It's been two years since this discussion began and there have been requests older than that. you just found the most "recent". -mike signature.asc Description: This is a digitally signed message part.
Re: Permission denied to execute script that is world executable
On Saturday, June 18, 2011 16:37:18 John Williams wrote: > Is this a bash bug, or intentional behavior? it's coming from the kernel, not bash post the output of `mount` and make sure that it doesnt have the "noexec" flag -mike signature.asc Description: This is a digitally signed message part.
Re: bug: return doesn't accept negative numbers
On Monday, August 08, 2011 21:20:29 Chet Ramey wrote: > On 8/8/11 8:53 AM, Eric Blake wrote: > > However, you are on to something - since bash allows 'exit -1' as an > > extension, it should similarly allow 'return -1' as the same sort of > > extension. The fact that bash accepts 'exit -1' and 'exit -- -1', but > > only 'return -- -1', is the real point that you are complaining about. > > That's a reasonable extension to consider for the next release of bash. i posted a patch for this quite a while ago. not that it's hard to code. -mike signature.asc Description: This is a digitally signed message part.
Re: Purge History of rm commands
On Monday, September 19, 2011 01:18:02 Roger wrote:
> I'm stumped on this as my history is in the format of:
>
> $ tail ~/.bash_history
> #1316296633
> man bash
> #1316296664
> bash -xv
> #1316372056
> screen -rd
> #1316375930
> exit
> #1316392889
> exit
>
> Is there a method of purging the history off all rm commands with such a
> file format? I've tried using history | find | grep | sed, but the
> history doesn't accept more then one history command line number.
so put it into a for loop ?
> I'm guessing the next easiest method is to learn awk/gawk so I can edit the
> above .bash_history file.
gawk '{ c = $0; getline; if ($1 != "rm") { print c; print; } }' .bash_history
> The easiest method, just open the history file with VI/VIM and start
> deleting the 20 or so lines... which I'll likely start doing now. ;-)
>
> ... or did I miss something?
i rarely use `history`, so i cant suggest any improvements there
-mike
signature.asc
Description: This is a digitally signed message part.
[patch] fix parallel build issues with parse.y
the current yacc rules allow multiple runs to generate the same files. usually
this doesn't come up as the generated files are shipped in the tarball, but
when you modify parse.y (applying a patch or developing or whatever), you can
hit this problem.
simple way of showing this:
make -j y.tab.{c,h}
a correct system would not show the yacc parser running twice :)
simple patch is to have the .h file depend on the .c file, and have the .h file
itself issue a dummy rule (to avoid make thinking things changed).
-mike
--- a/Makefile.in
+++ b/Makefile.in
@@ -579,16 +579,17 @@
# old rules
GRAM_H = parser-built
-y.tab.o: y.tab.c ${GRAM_H} command.h ${BASHINCDIR}/stdc.h input.h
+y.tab.o: y.tab.h y.tab.c ${GRAM_H} command.h ${BASHINCDIR}/stdc.h input.h
${GRAM_H}: y.tab.h
@-if test -f y.tab.h ; then \
cmp -s $@ y.tab.h 2>/dev/null || cp -p y.tab.h $@; \
fi
-y.tab.c y.tab.h: parse.y
+y.tab.c: parse.y
# -if test -f y.tab.h; then mv -f y.tab.h old-y.tab.h; fi
$(YACC) -d $(srcdir)/parse.y
touch parser-built
# -if cmp -s old-y.tab.h y.tab.h; then mv old-y.tab.h y.tab.h; else cp -p
y.tab.h ${GRAM_H}; fi
+y.tab.h: y.tab.c ; @true
# experimental new rules - work with GNU make but not BSD (or OSF) make
#y.tab.o: y.tab.c y.tab.h
[patch] fix parallel build issues with syntax.c
the current code generates a bunch of local libraries in subdirs and then
links bash against that. those subdirs sometimes need version.h. so they
have a rule to change back up to the parent dir and build version.h (which is
fine). the trouble is that the top level objects and the subdirs are allowed
to build in parallel, so it's possible for multiple children to see that
version.h is not available and that it needs to be created, so they all do.
there is even more trouble is that version.h depends on all the top level
sources, some of which are compiled (like syntax.c). so these parallel
children all kick off a job to generate syntax.c which in turn requires the
mksyntax helper executable. obviously multiple processes rm-ing, compiling,
and linking the same files quickly falls apart.
so tweak the subdirs to all depend on the .build target which in turn depends
on all of these top level files being generated. now the subdirs won't try and
recursively enter the top level.
(noticed by David James)
-mike
--- a/Makefile.in
+++ b/Makefile.in
@@ -597,6 +598,11 @@
# $(YACC) -d $(srcdir)/parse.y
# -if cmp -s old-y.tab.h y.tab.h; then mv old-y.tab.h y.tab.h; fi
+# Subdirs will often times want version.h, so they'll change back up to
+# the top level and try to create it. This causes parallel build issues
+# so just force top level sanity before we descend.
+$(LIBDEP): .build
+
$(READLINE_LIBRARY): config.h $(READLINE_SOURCE)
@echo making $@ in ${RL_LIBDIR}
@( { test "${RL_LIBDIR}" = "${libdir}" && exit 0; } || \
[patch] builtins: fix parallel build between top level targets
the top level Makefile will recurse into the defdir for multiple targets
(libbuiltins.a, common.o, bashgetopt.o, builtext.h), and since these do not
have any declared interdependencies, parallel makes will recurse into the
subdir and build the respective targets.
nothing depends on common.o or bashgetopt.o, so those targets don't get used
normally. this leaves libbuiltins.a and builtext.h. at a glance, this
shouldn't be a big deal, but when we look closer, there's a subtle failure
lurking.
most of the objects in the defdir need to be generated which means they need
to build+link the local mkbuiltins helper. the builtext.h header also needs
to be generated by the mkbuiltins helper. so when the top level launches a
child for libbuiltins.a and a child for builtext.h, we can hit a race
condition where the two try to generate mkbuiltins, and the build randomly
fails.
so update libbuiltins.a to depend on builtext.h. this should be fairly simple
since it's only a single target.
--- a/Makefile.in
+++ b/Makefile.in
@@ -674,7 +674,7 @@
$(RM) $@
./mksyntax$(EXEEXT) -o $@
-$(BUILTINS_LIBRARY): $(BUILTIN_DEFS) $(BUILTIN_C_SRC) config.h
${BASHINCDIR}/memalloc.h version.h
+$(BUILTINS_LIBRARY): $(BUILTIN_DEFS) $(BUILTIN_C_SRC) config.h
${BASHINCDIR}/memalloc.h ${DEFDIR}/builtext.h version.h
@(cd $(DEFDIR) && $(MAKE) $(MFLAGS) DEBUG=${DEBUG} libbuiltins.a ) ||
exit 1
# these require special rules to circumvent make builtin rules
signature.asc
Description: This is a digitally signed message part.
Re: Encrypted bashrc?
On Friday 11 November 2011 00:48:59 Clark J. Wang wrote: > In my company all the people share a few of Solaris servers which use NIS > to manage user accounts. The bad thing is that some servers' root passwords > are well known so anybody can easily su to my account to access my files. > To protect some private info in my bashrc I want to encrypt it. Any one has > a good solution for that? if they have root, they have access to all memory and devices. including your terminal where you enter the passphrase/key, or the memory where the file is decrypted/read. encrypting the files will make things harder, but won't make it inaccessible to people who really want it. if you want to protect private information, don't put it on a remote server. -mike signature.asc Description: This is a digitally signed message part.
Re: How to directly modify $@?
On Sunday 20 November 2011 11:54:42 Pierre Gaston wrote:
> On Sun, Nov 20, 2011 at 6:43 PM, Peng Yu wrote:
> > Hi,
> >
> > I don't see if there is a way to directly modify $@. I know 'shift'.
> > But I'm wondering if there is any other way to modify $@.
> >
> > ~$ 1=x
> > -bash: 1=x: command not found
> > ~$ @=(a b c)
> > -bash: syntax error near unexpected token `a'
>
> you need to use the set builtin:
> set -- a b c
yep
to pop items off the end:
shift [n]
to add items to the end:
set -- "$@" a b c
to add items to the start:
set -- a b c "$@"
to extract slices:
set -- "${@:[:]}"
e.g.
set -- a b c
set -- "${@:2:1}" # this sets $@ to (b)
with those basics, you should be able to fully manipulate $@
-mike
signature.asc
Description: This is a digitally signed message part.
Re: Bash git repository on savannah
On Wednesday 23 November 2011 23:23:43 Chet Ramey wrote: > I spent a little while messing around with git over the past couple of > days, and ended up updating the bash git repository on savannah > (http://git.savannah.gnu.org/cgit/bash.git to browse the sources). > Bash-4.2 patch 20 is the head of the tree, and there's a branch > containing the `direxpand' patches that I've posted here. Each > bash-4.2 patch is in there as a separate commit. i thought you were maintaining a private CVS somewhere ? if so, would it be possible to create a git tree from that ? if you want someone else to take care of the details, i can help ... i've converted a bunch of projects in the past over to git from CVS/SVN. -mike signature.asc Description: This is a digitally signed message part.
Re: Bash git repository on savannah
On Friday 25 November 2011 22:28:49 Chet Ramey wrote: > On 11/24/11 12:36 PM, Mike Frysinger wrote: > > On Wednesday 23 November 2011 23:23:43 Chet Ramey wrote: > >> I spent a little while messing around with git over the past couple of > >> days, and ended up updating the bash git repository on savannah > >> (http://git.savannah.gnu.org/cgit/bash.git to browse the sources). > >> Bash-4.2 patch 20 is the head of the tree, and there's a branch > >> containing the `direxpand' patches that I've posted here. Each > >> bash-4.2 patch is in there as a separate commit. > > > > i thought you were maintaining a private CVS somewhere ? if so, would it > > be possible to create a git tree from that ? if you want someone else > > to take care of the details, i can help ... i've converted a bunch of > > projects in the past over to git from CVS/SVN. > > Thanks. I have bash sources going back a number of years, and I can get > them into git. It will just take me a while. > > The question is what to do with that tree once it's assembled. push it to the git repo on svannah you cited above ? do you plan on continuing development on the internal tree, or would you be switching to git fulltime ? -mike signature.asc Description: This is a digitally signed message part.
Re: Bash git repository on savannah
On Sunday 27 November 2011 21:31:16 Chet Ramey wrote: > On 11/26/11 2:56 AM, Mike Frysinger wrote: > >> Thanks. I have bash sources going back a number of years, and I can get > >> them into git. It will just take me a while. > >> > >> The question is what to do with that tree once it's assembled. > > > > push it to the git repo on svannah you cited above ? > > Probably. I just have to figure out how to get git to do what I want. if you have questions, feel free to post and i'll try to follow up. i know learning curve can be high at first. > > do you plan on continuing development on the internal tree, or would you > > be switching to git fulltime ? > > I don't think I'll push every change to git as soon as it happens, but > I'm thinking about fairly frequent commits to a `bash-devel' sort of > tree. The question is whether or not enough people would be interested > in that to make the frequency worth it. i would ;) -mike signature.asc Description: This is a digitally signed message part.
Re: popd always has return status 0
On Thursday 01 December 2011 19:01:50 james.cuze...@lyraphase.com wrote: > Description: > popd does not appear to return a nonzero exit status when the directory > stack is empty anymore. works for me: $ echo $BASH_VERSION ; popd ; echo $? 4.2.20(1)-release bash: popd: directory stack empty 1 as does your popall func -mike signature.asc Description: This is a digitally signed message part.
Re: return values of bash scripts
On Tuesday 20 December 2011 17:18:16 kc123 wrote:
> For example, my script below called crond.sh:
> ...
> content=`ps auxw | grep [c]rond| awk '{print $11}'`
> ...
> and output is:
> CONTENT: /bin/bash /bin/bash crond
>
> Why are there 2 extra arguments printed (/bin/bash) ?
because you grepped your own script named "crond.sh"
make the awk script smarter, or use pgrep
-mike
signature.asc
Description: This is a digitally signed message part.
Re: bash, echo or openssl bug?
On Tuesday 03 January 2012 08:48:27 nick humphrey wrote:
> Description:
> i dont know if the bug is a bash bug or openssl or echo, but when i
> echo a string and pipe it to openssl, the
> output comes on the same line as the prompt instead of a new line. it makes
> the output hard to read because it is prepended
> to the prompt text, e.g. mySecretPasswordtcadmin@buildserver: ~$
>
> Repeat-By:
> 1. run the following code in bash terminal:
> echo OHBjcWNLNGlQaVF5 | openssl base64 -d
>
> 2. the output in the bash terminal looks like this:
> mySecretPasswordtcadmin@buildserver: ~$
there is no bug in any of these packages. openssl doesn't include a trailing
new line.
> 3. the output SHOULD look like this:
> mySecretPassword
> tcadmin@buildserver: ~$
then add it yourself:
$ echo OHBjcWNLNGlQaVF5 | openssl base64 -d; echo
$ out=$(echo OHBjcWNLNGlQaVF5 | openssl base64 -d); echo "${out}"
... many other ways ...
-mike
signature.asc
Description: This is a digitally signed message part.
excess braces ignored: bug or feature ?
can't tell if this is a bug or a feature.
FOO= BAR=bar
: ${FOO:=${BAR}
echo $FOO
i'd expect an error, or FOO to contain those excess braces. instead, FOO is
just "bar".
-mike
signature.asc
Description: This is a digitally signed message part.
Re: RFE: allow bash to have libraries (was bash 4.2 breaks source finding libs in lib/filename...)
On Wednesday 29 February 2012 17:53:21 Linda Walsh wrote: > Eric Blake wrote: > > On 02/29/2012 12:26 PM, Linda Walsh wrote: > >>> Any pathname that contains a / should not be subject to PATH searching. > > > > Agreed - as this behavior is _mandated_ by POSIX, for both sh(1) and for > > execlp(2) and friends. > > Is it that you don't read english as a first language, or are you just > trying to be argumentative?' i'm guessing irony is lost on you ad hominem attacks have no business on this list or any other project. if you can't handle that, then please go away. -mike signature.asc Description: This is a digitally signed message part.
Re: I think I may have found a possible dos attack vector within bash.
On Tuesday 20 March 2012 15:55:18 Chet Ramey wrote:
> or the even simpler
>
> f()
> {
> f | f &
>
> }
> f
i like the variant that uses ":" instead of "f":
:(){ :|:& };:
-mike
signature.asc
Description: This is a digitally signed message part.
Re: UTF-8 regression in bash version 4.2
On Tuesday 27 March 2012 08:08:33 Pierre Gaston wrote: > On Tue, Mar 27, 2012 at 3:00 PM, Joachim Schmitz wrote: > > dennis.birkh...@rwth-aachen.de wrote: > > > > > > >> Bash Version: 4.2 > >> Patch Level: 24 > >> Release Status: release > > > > Interesting, seems the announcements dor patches 21-24 have gotten lost? > > they were posted on the mailing list, maybe the relay to the group failed i recall seeing them, as does the archive: http://lists.gnu.org/archive/html/bug-bash/2012-03/threads.html Bash-4.2 Official Patch 24, Chet Ramey, 2012/03/12 Bash-4.2 Official Patch 23, Chet Ramey, 2012/03/12 Bash-4.2 Official Patch 22, Chet Ramey, 2012/03/12 Bash-4.2 Official Patch 21, Chet Ramey, 2012/03/12 -mike signature.asc Description: This is a digitally signed message part.
Re: status on $[arith] for eval arith vsl $((arith))??
On Saturday 07 April 2012 16:45:55 Linda Walsh wrote: > Is it an accidental omission from the bash manpage? it's in the man page. read the "Arithmetic Expansion" section. -mike signature.asc Description: This is a digitally signed message part.
string replace with multibyte chars and extglob fails with bash-4.2
first set your locale to something unicode based:
export LC_ALL=en_US.UTF-8
then try the simple script (from Ulrich Müller):
$ cat test.sh
shopt -s extglob
text="aaaäöü"
echo "${text} ${text//?aa} ${text//\aaa}"
with bash-4.1_p2, i get:
aaaäöü äöü äöü
but with bash-4.2_p8 ... 4.2_p24 (just what i have locally):
aaaäöü aaaäöü aaaäöü
seems like a bug to me
-mike
signature.asc
Description: This is a digitally signed message part.
[patch] fix building when readline is disabled
if you disable readline, the complete.def code fails to build. simple patch below (not sure if it's correct, but at least gets the conversation going). -mike --- a/builtins/complete.def +++ b/builtins/complete.def @@ -49,6 +49,8 @@ $END #include +#ifdef READLINE + #include #include "../bashtypes.h" @@ -867,3 +869,5 @@ compopt_builtin (list) return (ret); } + +#endif signature.asc Description: This is a digitally signed message part.
Re: [patch] fix building when readline is disabled
On Monday 23 April 2012 18:57:05 Chet Ramey wrote:
> On 4/23/12 12:22 AM, Mike Frysinger wrote:
> > if you disable readline, the complete.def code fails to build. simple
> > patch below (not sure if it's correct, but at least gets the
> > conversation going).
>
> How did you disable readline? Running configure --disable-readline and
> building as usual works for me. You might want to run `make clean' before
> rebuilding.
i suspect you have readline files still being included and it appears to work.
let's look at vanilla bash-4.2:
$ tar xf bash-4.2.tar.gz
$ cd bash-4.2
$ ./configure --disable-readline
$ make
...
rm -f complete.o
./mkbuiltins -D . complete.def
gcc -c -DHAVE_CONFIG_H -DSHELL -I. -I.. -I.. -I../include -I../lib -I.-g
-O2 complete.c || ( rm -f complete.c ; exit 1 )
rm -f complete.c
...
so, let's go into that dir and run it by hand:
$ cd builtins
$ ./mkbuiltins -D . complete.def
$ strace -f -eopen gcc -c -DHAVE_CONFIG_H -DSHELL -I. -I.. -I.. -I../include
-I../lib -I.-g -O2 complete.c |& grep readline.h
...
[pid 12453] open("../lib/readline/readline.h", O_RDONLY|O_NOCTTY) = 4
...
if you were to clean out your readline code first, you'd see the build error
i'm seeing instead of the local readline code getting implicitly used even
though it was explicitly disabled.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [patch] fix building when readline is disabled
On Monday 23 April 2012 20:08:26 Chet Ramey wrote:
> On 4/23/12 7:40 PM, Mike Frysinger wrote:
> > On Monday 23 April 2012 18:57:05 Chet Ramey wrote:
> >> On 4/23/12 12:22 AM, Mike Frysinger wrote:
> >>> if you disable readline, the complete.def code fails to build. simple
> >>> patch below (not sure if it's correct, but at least gets the
> >>> conversation going).
> >>
> >> How did you disable readline? Running configure --disable-readline and
> >> building as usual works for me. You might want to run `make clean'
> >> before rebuilding.
> >
> > i suspect you have readline files still being included and it appears to
> > work.
>
> OK, so you've stripped the local readline copy out of the source tree?
yes
> Then configured it to build with a system readline library installation
> that you remove?
the system doesn't have readline at all
> > let's look at vanilla bash-4.2:
> > $ tar xf bash-4.2.tar.gz
> > $ cd bash-4.2
> > $ ./configure --disable-readline
> > $ make
> > ...
> > rm -f complete.o
> > ./mkbuiltins -D . complete.def
> > gcc -c -DHAVE_CONFIG_H -DSHELL -I. -I.. -I.. -I../include -I../lib -I.
> >-g -O2 complete.c || ( rm -f complete.c ; exit 1 ) rm -f complete.c
> > ...
> >
> > so, let's go into that dir and run it by hand:
> > $ cd builtins
> > $ ./mkbuiltins -D . complete.def
> > $ strace -f -eopen gcc -c -DHAVE_CONFIG_H -DSHELL -I. -I.. -I..
> > -I../include -I../lib -I.-g -O2 complete.c |& grep readline.h ...
> > [pid 12453] open("../lib/readline/readline.h", O_RDONLY|O_NOCTTY) = 4
> > ...
> >
> > if you were to clean out your readline code first, you'd see the build
> > error i'm seeing instead of the local readline code getting implicitly
> > used even though it was explicitly disabled.
>
> What does "clean out your readline code" mean? Disabled means that it
> doesn't end up in the resulting binary, and the bash binary is not linked
> against readline. There aren't any link errors because the builtins are
> in a library, and no other bash code calls any function in complete.c, so
> complete.o is not linked out of libbuiltins.a. What kind of "build error"
> are you seeing?
without a readline.h header available (system or local copy), the build fails.
there's plenty of READLINE and HISTORY ifdefs in the other files, so this looks
like one place that got missed. imo, there should be no attempts to include a
readline header if support is disabled.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [patch] fix building when readline is disabled
On Tuesday 24 April 2012 08:23:04 Chet Ramey wrote: > On 4/24/12 12:00 AM, Mike Frysinger wrote: > >> OK, so you've stripped the local readline copy out of the source tree? > > > > yes > > > >> Then configured it to build with a system readline library installation > >> that you remove? > > > > the system doesn't have readline at all > > Why? because it's a small system which has no need for things like readline. i don't think this is a terribly unusual use case. -mike signature.asc Description: This is a digitally signed message part.
