Unexpected delay in using arguments.
Try this script:
#!/bin/bash
TIMEFORMAT='%R'
n=1000
m=2
f1 () { :; }
f2 () { i=0; time while [ "$((i+=1))" -lt "$n" ]; do :; done
i=0; time while [ "$((i+=1))" -lt "$n" ]; do f1; done
}
test1() { set -- $(seq $m)
f2 ""
f2 "$@"
}
test1
To get:
0.019
0.028
0.019
19.204
Which is a thousand times slower.
Bash 5 is even worse, try:
time b50sh -c 'f(){ :;};for i do f; done' {0..500}
real0m20.709s
user0m19.856s
sys 0m0.024s
Read more detail here:
https://unix.stackexchange.com/questions/462084/bash-has-troubles-using-argument-lists
Re: Rational Range Interpretation for bash-5.0?
On 08/06/2018 03:07 PM, Chet Ramey wrote: Hi. I am considering making bash glob expansion implement rational range interpretation starting with bash-5.0 -- basically making globasciiranges the default. It looks like glibc is going to do this for version 2.28 (at least for a-z, A-Z, and 0-9), and other GNU utilities have done it for some time. What do folks think? I am in favor of the idea provided there is a way to change back to what is now the default. Also, the character order may be extended to use C.utf-8 instead of ASCII to have an stable order for all UNICODE characters. Read https://manpages.debian.org/unstable/open-infrastructure-locales-c.utf-8/locales-c.utf-8.7.en.html
Re: Unexpected delay in using arguments.
Yes Greg, Please read the expanded source question, bash 5 with (and without) bash-malloc has been tested. https://unix.stackexchange.com/q/462084/265604 The problem with arguments is still present. 2018-08-13 9:37 GMT-04:00 Greg Wooledge : > On Sun, Aug 12, 2018 at 03:16:34AM -0400, Bize Ma wrote: > > Which is a thousand times slower. > > Bash 5 is even worse > > Pre-release bash sources use a debugging-friendly (slow) malloc. Or > something. Damn, google is not helping me out here. >
Assignment of $* to a var removes spaces on unset IFS.
Executing this code: set -- " foo "" bar baz " " quux " unset IFS a=$* b="$*" printf '[%s]' "$a" "$b"; echo Leads to this results in several shells: ash : [ foo bar baz quux ][ foo bar baz quux ] dash: [ foo bar baz quux ][ foo bar baz quux ] b205sh : [ foo bar baz quux ][ foo bar baz quux ] b30sh : [ foo bar baz quux ][ foo bar baz quux ] b32sh : [ foo bar baz quux ][ foo bar baz quux ] b41sh : [ foo bar baz quux ][ foo bar baz quux ] b42sh : [ foo bar baz quux ][ foo bar baz quux ] b43sh : [foo bar baz quux][ foo bar baz quux ] b44sh : [foo bar baz quux][ foo bar baz quux ] bash: [foo bar baz quux][ foo bar baz quux ] posixbash : [foo bar baz quux][ foo bar baz quux ] lksh: [ foo bar baz quux ][ foo bar baz quux ] mksh: [ foo bar baz quux ][ foo bar baz quux ] ksh93 : [ foo bar baz quux ][ foo bar baz quux ] attsh : [ foo bar baz quux ][ foo bar baz quux ] zsh : [ foo bar baz quux ][ foo bar baz quux ] Bash since 4.3 fails to follow what the documentation describes as that on an assignment values do not undergo splitting or globing.
Re: Assignment of $* to a var removes spaces on unset IFS.
2018-08-13 21:16 GMT-04:00 Eduardo Bustamante : > On Mon, Aug 13, 2018 at 5:25 PM Robert Elz wrote: > [...] > > What's more, reporting bugs in bash 4.3 is just a waste of everyone's > time. > > Upgrade to (at least) the current released version, and use that instead. > > Adding to this point: > > * bash 4.3 was released 4 years ago (2014-02-26) > * bash 4.4 is the current release of bash (2016-09-15) > * bash 5.0 is in alpha state (2018-05-22) > > You both are confused. The original table included > b43sh : [foo bar baz quux][ foo bar baz quux ] > b44sh : [foo bar baz quux][ foo bar baz quux ] That's 4.3.48(2)-release and4.4.19(2)-release Both are failing. Note that 4.4.19 is newer than what is available in https://www.gnu.org/software/bash/ The report is valid for present releases. The bug is still present (since 4.3). Please, do not waste our time in incorrect claims. Do your homework and test ! > I know that some stable Linux distributions might still be > distributing 4.3, but that just means that you have to report the bug > against these distros, not upstream. That is not the problem, don't jump to conclusions.
Re: Assignment of $* to a var removes spaces on unset IFS.
2018-08-13 22:09 GMT-04:00 Eduardo A. Bustamante López : > On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote: > (...) > > Please, do not waste our time in incorrect claims. > > > > Do your homework and test ! > (...) > > > dualbus@ubuntu:~/src/gnu/bash$ bash /tmp/script > 4.4.19(1)-release > [foo bar baz quux][ foo bar baz quux ] > > So, it is confirmed that the bug exists in the present release, Is it not? > Did you test Bash 5.0? Because that's where the current bug fixes are going to. So, is it a "wont fix" for 4.4 (present release) ? > I don't see Chet releasing a new version of 4.4 to fix something that's already > fixed there. That version is not even beta, it is still alpha, are you asking that everyone should use non-released (and not yet tested as beta) alpha release ? > dualbus@ubuntu:~/src/gnu/bash$ cat /tmp/script > echo $BASH_VERSION > set -- " foo "" bar baz " " quux " > unset IFS > a=$* > b="$*" > printf '[%s]' "$a" "$b"; echo Thanks for copying the supplied script. > dualbus@ubuntu:~/src/gnu/bash$ ./bash /tmp/script > 5.0.0(1)-alpha > [ foo bar baz quux ][ foo bar baz quux ] > > Thanks for testing. !! > Please read this thread: > > * http://lists.nongnu.org/archive/html/bug-bash/2017-09/msg00058.html > So? It is not solved in that thread, it is even said that: > This is a bug in bash and it should be fixed, not excused. To which I agree. After a year, nothing else have been said about it. It seems about time to get it solved. Or say that it won't be.
Re: Assignment of $* to a var removes spaces on unset IFS.
2018-08-13 23:17 GMT-04:00 Eduardo A. Bustamante López : > On Mon, Aug 13, 2018 at 10:52:20PM -0400, Bize Ma wrote: > (...) My point is that, in the context of bug reports, it's > important that you always test against the *unreleased* version of the > code, > Not if the version to test is the released one. > (...)Here's the response from Chet: > > http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html > > The fix for this issue is already in the Git repository (devel branch) and > in > Bash 5.0. > Excellent, that is good to know. There is a solution. Chet explains it clearly. Just that it is not applied to the present release version 4.4. Maybe it should be put into a "stack of pending patches" that all will be applied when a new number release (4.5) has to be done for whatever other reason. When that happens, we can say that it "is solved for current release". I cannot answer for Chet, but please consider this: > > The fix for this bug breaks backwards compatibility In what sense? Other shells don't have this issue and the change got introduced in 4.3, so, this change is backward compatible with several versions up to 4.3 (backward compatible). > so that means it cannot be > a patch-level fix for 4.4. It should either be a minor version increase > (4.5) or > go into the next major release, which is 5.0 (which was already declared > alpha > state, and includes the fix). > I am not who must make that call. Someone else must. If this is not going to be solved in 4.4, then it is a "wontfix" for that release. > It seems about time to get it solved. Or say that it won't be. > > Again, it's fixed already No, it is not "fixed already" on 4.4 release. Let's call things as they are, not as anyone "hope" they can be. and the fix is scheduled to go out in the next > release (5.0). > Excellent, great, version 5.0 has one issue less to solve. Yay!! > Given that, do you think this bug is severe enough to have to issue a new > minor > version just for it? (4.5), > Don't ask me, it is not a call that I can make, I am just a random user. > or can we just wait for 5.0 to come out which fixes this and a bunch of > other > bugs? (see the CHANGES file, in particular, entry `oo', > http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h= > bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b) > Yes, there it says that some changes were made to match POSIX interpretation of $* $@. I believe that the issue presented in the OP is one of those changes for 5.0.
Re: Unexpected delay in using arguments.
2018-08-14 11:25 GMT-04:00 Chet Ramey : > On 8/12/18 3:16 AM, Bize Ma wrote: > > Try this script: > (...) > Which is a thousand times slower. > > If you build a profiling version of bash, you'll find that about 75% of > that time is spent copying the list of arguments around I just don't see why there is any need to copy arguments. Most other shells seem that they don't do that. > , since you have > to save and restore it each time you call f1. There is no need to "copy", that just waste additional memory. Just point to a new list of arguments (it is a change of only a memory pointer). Release the memory when the argument list is not being used any more. > > Bash 5 is even worse, try: > > Bash-5.0-alpha is not a released version, so it uses the debugging malloc. > If you profile that, you'll find that about 99% of the time is spent > marking allocations as active and free in the table the bash malloc uses > to keep track of active memory. > Yes, tests with "--without-bash-malloc" and "RELSTATUS=release" were done after I posted the OP. Yes, version 5.0 is not corrected, and looks that it is not "worse". Sorry for the misrepresentation.
Re: Unexpected delay in using arguments.
> Of course I assume this is only a proxy simpler reproducer
> for the actual problem program
Of course this is a "reproducer" of the issue.
> but just the same it is almost always possible
> to refactor a program into a different algorithm that avoids the need
> to enumerate so many arguments in memory.
As you say: "almost".
Take a look at the Stéphane Chazelas example to convince yourself.
2018-08-14 17:50 GMT-04:00 Bob Proulx :
> I followed along through the script:
>
> > m=2
> ...
> > test1() { ...
>
> Eventually I got to this line:
>
> > set -- $(seq $m)
>
> At that line I see that bash must allocate enough memory to hold all
> of the numbers from 1 through 2 in memory all at one time. That
> is very inefficient. That is at least 100K of memory.
>
> $ seq 2 | wc -c
> 108894
>
> Of course I assume this is only a proxy simpler reproducer for the
> actual problem program but just the same it is almost always possible
> to refactor a program into a different algorithm that avoids the need
> to enumerate so many arguments in memory. I suggest refactoring the
> program to avoid the need for this memory stress.
>
> Bob
>
Re: Unexpected delay in using arguments.
2018-08-15 3:36 GMT-04:00 Bob Proulx : > Bize Ma wrote: > > I still believe that to be true. You are entitled to have an opinion, even if incorrect. > Since you haven't shared what the > actual task is there is no way for us to propose any counter example > improvements. So "almost" is as far as I can go. Instead should I > say 99.44% of the time? Since I have never come across a counter > example yet? > Give it time. > > Take a look at the Stéphane Chazelas example to convince yourself. > > Nothing Stéphane said changed my statement at all. > How do you process a «list of files» without the «list of files» ? > It does look like bash can be more efficient with argument handling. > Since, for example, dash does it. > That is true.
Re: Assignment of $* to a var removes spaces on unset IFS.
2018-08-15 8:44 GMT-04:00 Greg Wooledge :
> This reply was sent to me without Cc-ing the list. I have added the Cc.
>
> On Tue, Aug 14, 2018 at 11:39:20PM -0400, Bize Ma wrote:
> > On Tue, 14 Aug 2018 12:34:31, Greg Wooledge said:
> >
> > > I will also repeat, once more, my advice that one should NEVER write
> > > a script containing an unquoted $* or $@ expansion.
> >
> > That is plainly INCORRECT, Greg.
>
> You are incorrect.
I am always incorrect, I like to be so.
But we are discussing an issue, not my personal problems.
> > > It breaks in all kinds of ways in more than one shell.
> >
> > That several shells do different things is a bug on those shells, not
> bash.
>
> Agreed.
Excellent, we agree.
> And the fact that IN REAL LIFE, THOSE SHELLS EXIST AND HAVE
> THOSE BUGS, which are triggered by incorrect code
No, sorry, but bugs are triggered by perfectly good code, that's why they
are called bugs.
> , is a reason to write
> code correctly so as not to trigger those bugs.
>
Writing code to *work around* bugs is *not* the correct solution.
It is only a way to *perpetuate* those bugs.
The correct solution is to resolve the bugs.
But this is just a smoke curtain. There are no bugs about $@ and $* in Bash.
Or maybe you're one of those people who doesn't care about reality.
>
Sometimes I don't , most of the time when I sleep, sometimes when I day
dream.
> > > Just don't do it, and these problems go away.
> >
> > If I want the split+glob to take effect I can do:
> >
> >echo $*
> >
> >
> > There is nothing wrong with that (don't claim that it change in different
> > shells, that is a different issue than using split+glob in bash, go back
> to
> > the point above about other shells if you wish).
>
> There is absolutely definitely positively 100% certainly something wrong
> with that.
>
Ah, yes, that's correct, thanks, I should have used printf, not echo.
But you corrected it, thanks again.
Let's break your script, shall we?
>
> Here's your script, except I'm going to represent it as a function. Doing
> it as a script would have the same effect.
>
> glob() {
> # "Return" (write to stdout, one per line) the expansions of all
> # arguments as globs against the current working directory.
> printf %s\\n $*
> }
>
> Will you at least agree that this is your intent,
No, that is *not* my intent.
You can not implement a glob function when *both: split and glob* are in
effect.
If you want a glob function, stop the split, set IFS to null:
$ cat script
#!/bin/bash
glob() ( IFS=; printf %s\\n $* )
glob '*Web Service*'
$ ./script
Epic Web Service - PatientLookup.pdf
and a fair
> representation of your proposed solution? I'll take that as a "yes".
>
No it is not a fair representation of anything I said.
Only of a twist that you want to present here.
So now, you can pass SOME globs to it (properly quoted), and it will
> appear to work for those globs:
>
But it will fail to work on the next example as you want to demonstrate,
therefore you have not coded correctly to workaround the shells bugs
(your own words).
Some of us care about reality.
>
Yes, when I am awake.
Other languages have no problem with this. Tcl,
(...)
But we are in bash, are we not?
Now, you tried to implement Tcl's [glob] in bash, but you did it naively
> and the naive version does not work. You took a shortcut. That shortcut
> falls off a cliff.
>
No, *you* did. You implemented a flawed glob function.
> I'll leave the non-broken implementation as an exercise for the reader.
>
You are not able to do it?
> Now, back to the original points:
>
> 1) Certain shells have bugs in them. These shells are in widespread use
>on real systems in real life.
>
> 2) One of those shells is bash, which makes it relevant to bug-bash.
>
> 3) Some of those bugs involve the unquoted expansions of $* and $@.
>
Not in bash.
> 4) Even in the ABSENCE of such bugs, the use of an unquoted $* or $@
>expansion does not actually solve the problems you claim it solves.
>
I claimed nothing. I presented a valid use. Don't put words in my mouth.
> 5) Therefore, for TWO reasons, you should not use unquoted $* or $@
>in your shell scripts.
>
>Reason 1: it doesn't solve the problem.
>Reason 2: it sometimes breaks even worse due to shell bugs.
>
There is no therefore if the premises are wrong.
Bash removes unrequested characters in bracket expressions (not a range).
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: Linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib
-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2
-fdebug-prefix-map=/build/bash-7fckc0/bash-4.4=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall
-no-pie -Wno-parentheses -Wno-format-security
uname output: Linux io 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2
(2018-10-27) x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu
Bash Version: 4.4
Patch Level: 12
Release Status: release
Description:
Bash is removing characters not explicitly listed in a bracket
expression (character range).
In this example, it is removing digits from other languages.
Also tested (and it fails) in bash 3.{0,1,3} 4.{1,2,3} and 5.0
Not a problem in bash 2.{0,1}
Repeat-By:
If the characters are a problem: please visit:
https://unix.stackexchange.com/q/483743/265604
$ a='0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९'
$ echo "${a//[0123456789]}"
۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९
Re: Bash removes unrequested characters in bracket expressions (not a range).
Chet Ramey () wrote: > On 11/23/18 6:09 PM, Bize Ma wrote: > > > Bash Version: 4.4 > > Patch Level: 12 > > Release Status: release > > > Description: > > > > Bash is removing characters not explicitly listed in a bracket > > expression (character range). > > In this example, it is removing digits from other languages. > > What is your locale? > > The locale used was en_US.utf-8 but also happens with 459 locales out of 868 available under Debian (not in C, for example). Also in all locales affected (except one), setting either LC_ALL=$loc or LC_COLLATE=$loc did the same. Except in zh_CN.gb18030 But IMO locale collation should not be used for an explicit list. I have been made aware that there is a cstart = cend = FOLD (cstart); inside the `sm_loop.c` file that will convert into a range many individual character. If that understanding is correct that is the source of the difference with other shells. I have the perception that a collation table *must have a "total order"*, in fact, an strict total order. If two characters `a` and `b` could sort as equal the order will fail to provide a confirmation that a character is absent from the list. Consider characters `a`, `b` and `c`, if a and b sort as equal, a sorted list in which we find `a` followed by `c` doesn't confirm that `b` is absent as the order could well be `b a c`. In this case, there must not be any other character than `a` in the range `a-a` and using a range `a-a` is equivalent (just slower and more complex) to the single character `a`. If this is not the case, the error is in the collation table, not in using single (faster) characters. And what should be updated is such collation table IMO.
Re: Bash removes unrequested characters in bracket expressions (not a range).
Chet Ramey () wrote: > I am going to forward the rest of our exchange to bug-bash; Great! > you left the mailing list off your replies for some reason. > > Chet > Since it was *you* who directed the answer only to me, I assumed (now seemingly incorrectly) that that was what you wanted to do.
Re: Bash removes unrequested characters in bracket expressions (not a range).
Chet Ramey () wrote: > On 11/24/18 4:32 PM, Bize Ma wrote: [...] > > I have been made aware that there is a > > cstart = cend = FOLD (cstart); > > inside the `sm_loop.c` file that will convert into a range many > > individual character. If that understanding is correct that is the > > source of the difference with other shells. > > I'm not sure what you mean by "convert into a range." If cstart and cend > were treated as a range, the start end and end characters would be the > same. If cstart == cend, a character that collates >= cstart and <= cend > would have to collate equal to cstart and cend. > Yes, exactly, a range where the start and the end are the same. Try: $ touch 0 1 ٠ ١ ۰ ۱ ߀ ߁ ० १ $ echo [1] 1 ١ It is converted to the same range as this $ echo [1-1] 1 ١ That happens because up to glibc 2.27 this has been the collation order of those characters (search in /usr/share/i18n/locales/iso14651_t1_common) : <0>IGNORE <0>IGNORE Collate to exactly the same values. This breaks the capacity to detect that a character is absent in a list ordered by the collation order.
Re: Bash removes unrequested characters in bracket expressions (not a range).
Chet Ramey () wrote: > On 11/24/18 2:32 PM, Chet Ramey wrote: > > >> But IMO locale collation should not be used for an explicit list. > > > > Collation order is used for each individual character in a bracket > > expression when compared against the string, as posix specifies. > Yes, values resulting from a glob expansion should be compared with strcoll. How many characters should there be in a range like [0-0] ? Or to be more precise: in a [0] bracket expression? one? If I were you, I would file a bug report with Debian against wcscoll. > And I would be told that wcscoll is doing what the collation file 14651 is telling it to do. And, that in any case, that file has been updated in glib2.8 anyway. > It returns 0 (equal) for L"٠" and L"0" without setting errno. That's > clearly a problem with wcscoll (if the character isn't valid in the current > locale) or the locale definition. > Both characters collate to the same position as I have already explained. I don't follow you about what you mean with: *(if the character isn't valid in the current locale).*
Re: Bash removes unrequested characters in bracket expressions (not a range).
Chet Ramey () wrote: > I can't reproduce this: > If you could take a look at https://unix.stackexchange.com/a/483835/265604 you will see that it has been confirmed on "Ubuntu 17.10 (glibc 2.26) and on Ubuntu 18.04 (glibc 2.27), but it seems to be fixed on Ubuntu 18.10 (glibc 2.28)" It is interesting that (finally) glibc 2.28 has added a fourth sort key equal to the Unicode code point. That forces the order of all characters to be unique.
Re: write() not retried after EINTR in printf and echo
In here: https://lists.gnu.org/archive/html/bug-bash/2018-01/msg00032.html You promised: The next version of bash will install its SIGWINCH handler with SA_RESTART. Yet: Still a bug in bash version 5
Re: write() not retried after EINTR in printf and echo
Chet Ramey () wrote:
> And yet, when you look at the source, the signal handlers are installed
> with SA_RESTART. For example,
>
> # if defined (SIGWINCH)
> act.sa_flags = (sig == SIGWINCH) ? SA_RESTART : 0;
> # else
> act.sa_flags = 0;
> # endif /* SIGWINCH */
>
> in lib/readline/signals.c:rl_set_sighandler().
>
>
True for SIGWINCH, but "write error: Interrupted system call"
is still exposed to users (which it shouldn't) for other signals:
{ pid=$BASHPID; trap : USR1;
(sleep 1; kill -USR1 "$pid") &
printf %010d 1;
} | sleep 20
Error on arithmetic evaluation of `~0`.
This is the third time I am reporting this issue.
This fails:
var=(hello); echo "${var[~0]}"
syntax error: operand expected ...
While this works:
var=(hello); echo "${var[ ~0]}"
hello
It is also interesting that this fails:
var=hello; echo "${var[ ~0]}"
bash: var: bad array subscript
Isn't `var[0]` valid and equivalent to `var` ?
This was "supposed" to be resolved in a dev version,
but is still present on bash 5.
Terminating background bash kills the parent?
Open a new terminal like xterm, change prompt to something like: PS1=' $SHLVL $ ' Start a couple of bash subshells: 1 $ bash 2 $ bash 3 $ echo $$ 9371 Suspend the last one: 3 $ suspend [1]+ Stopped bash 2 $ echo $$ 9313 2 $ Make the suspended shell continue: 2 $ kill -CONT 9371 The parent gets killed 1 $ echo $$ 9034 1 $ Take a look at: https://unix.stackexchange.com/questions/379507/weird-terminating-background-bash-kills-the-parent
readline fails to understand options.
This works: $ bind 'set enable-bracketed-paste on'; bind -v | grep 'bracketed' set enable-bracketed-paste on However, almost any variation of the parameter "on" is not understood by readline: $ bind 'set enable-bracketed-paste on .'; bind -v | grep 'bracketed' set enable-bracketed-paste off $ bind 'set enable-bracketed-paste "on"'; bind -v | grep 'bracketed' set enable-bracketed-paste off Without any error or warning !! It appears to fall to default value, while this: $ bind 'set bell-style on' ; bind -v | grep 'bell-style' set bell-style audible $ bind 'set bell-style onf' ; bind -v | grep 'bell-style' set bell-style audible Stays at its last value. It seems that the parser for this options is unable to process anything after a valid parameter. And it doesn't emit an error or warning either. Since the parser needs some improvement: Could it be possible to add the capability to detect a `#` after the options as a comment ? Please read: https://unix.stackexchange.com/questions/487037/why-is-my-readline-inputrc-configuration-being-ignored/487285#487285
Re: Error on arithmetic evaluation of `~0`.
Chet Ramey () wrote:
> >
> > While this works:
> >
> > var=(hello); echo "${var[ ~0]}"
> > hello
>
> Because negative array subscripts count backwards from the end of the
> array.
>
Doh!, yes. And, because of that: "${var[-1]}"
should give the *last* element of array "var", shouldn't it?
Consequently, this happens:
$ unset var; var[0]=77; echo "${var[0]}"; echo "${var[-1]}"
77
77
The only value in var is at index 0, which means it is also the *last*
value.
> > It is also interesting that this fails:
> >
> > var=hello; echo "${var[ ~0]}"
> > bash: var: bad array subscript
> >
> > Isn't `var[0]` valid and equivalent to `var` ?
>
> Yes, but ~0 (-1) is not the same as 0.
>
Doh!, yes, of course, "0" is not equal to "~0" (-1). But if you were to
compare the two last command lines *as posted:*
> var=(hello); echo "${var[ ~0]}"
> var=hello ; echo "${var[ ~0]}"
You will notice that the only difference is that one explicitly creates an
array (and works) while the other creates an scalar (and fails).
The point being that a variable which has an scalar value "var=hello"
should act (for most practical cases) as an array for which only the
value at address 0 has been defined.
Both command line above should have printed "hello".
But it should be more clear to you written as this:
unset var; var=(hello); echo "${var[0]}:${var[ -1]}"
unset var; var=hello ; echo "${var[0]}:${var[ -1]}"
The first line works, the second fails on the negative index.
It seems that bash asserts that the variable is an array when the
index is negative (and emits an error if var is not an array).
> > This was "supposed" to be resolved in a dev version,
> > but is still present on bash 5.
>
> The other arithmetic contexts you reported (the "pure" arithmetic contexts
> the comment above references) were changed; this was left for backwards
> compatibility. Like I said above, it looks like it's time to deemphasize
> that.
>
In other words: you solved this for *some* arithmetic contexts, but not all.
Thanks for the confirmation.
Re: Unexpected delay in using arguments.
Chet Ramey () wrote: > On 8/15/18 3:36 AM, Bob Proulx wrote: > > > It does look like bash can be more efficient with argument handling. > > Since, for example, dash does it. > > Yes, it just needs new primitives to do it. The existing code for managing > the saved positional parameters has been in bash since the pre-1.0 days and > is pretty much unchanged since then. I'll take a look. > Have you been able to "take a look" ?
Re: write() not retried after EINTR in printf and echo
Chet Ramey () wrote: > On 12/19/18 10:15 PM, Bize Ma wrote: > > Ah, now we're moving the goalposts. No, that is not the intent. I merely failed to correctly convey the perception that others have of your code: https://unix.stackexchange.com/a/487260/265604 And, having failed, I am trying again. But, probably, I should just not inform you of what others think given your upfront rejection to any (constructive) criticism. There's a case to be made for doing > that, I suppose, but I'd rather defer to the system's default handling > for a particular signal. > Do as you wish, we will still be able to form our own opinion about buggy code.
Re: readline fails to understand options.
> > Sure, it's a candidate for inclusion in a future version. I wouldn't object > if someone wanted to do a sample implementation, since it changes how the > variable value is parsed. > I only wish I knew enough C to make any (even small) positive recommendation.
Re: write() not retried after EINTR in printf and echo
Eduardo Bustamante () wrote: > > (...) > What if instead of > complaining you do something about it, like, fixing the problem (send > a patch)? > You are assuming that if I take a look at the c code I will be able to read it, then understand it, and then make meaningful, or even reasonable changes to it. My experience tells me I can not. Not even read C correctly. So, sadly I can not send a patch, even if I were to try … Chet doesn't owe you anything. > Of course he doesn't owe me anything. As much as I don't owe *you* anything. But as I am reading your complaint to me and I am giving you an answer, I expect that Chet could be able to read a complaint, take the good parts, if any, and answer accordingly.
Re: Unexpected delay in using arguments.
Chet Ramey () wrote:
> On 12/23/18 12:07 PM, Bize Ma wrote:
{...}
> > Have you been able to "take a look" ?
>
> Yes, as a matter of fact. Look at the changelog for the day or two around
> the date of the referenced messages.
>
I am pretty sure that I will not understand the finer details of the c code
even
if I could take a look at it. But, for the learning: could you be so kind
as to
provide some pointer as to where to find the change log ? Thanks!.
Re: Error on arithmetic evaluation of `~0`.
Chet Ramey () wrote:
> On 12/23/18 12:01 PM, Bize Ma wrote:
>
{…}
> > Both command line above should have printed "hello".
>
> No. 0 is the only valid subscript for a non-array variable. The difference
> between bash and other shells that implement this feature is that bash
> warns about negative subscripts.
>
If you say so: fine for me.
It still irks me a little that a `${var[-1]}` isn't the "last value"
(sometimes!, consistency?).
I haven't seen that documented anywhere, though.
Re: Unexpected delay in using arguments.
Chet Ramey () wrote:
> It's in CWRU/CWRU.chlog in the development distributions
>
I am sorry but I have been unable to find either the source code
or the change log, either at the university site or by looking with google.
However, I have made some tests. Yes, the deeper internal delay
with arguments have been removed. Running this code
(from https://unix.stackexchange.com/q/462084/265604):
#!/bin/bash --
TIMEFORMAT='real: %R'
f1 () { :; }
f2 () { echo 'args = '"$#";
printf '1 function no args yes '; time for ((i=1;i<$n;i++));
do : ; done
printf '2 function yes args yes '; time for ((i=1;i<$n;i++));
do f1 ; done
set --
printf '3 function yes args no '; time for ((i=1;i<$n;i++));
do f1 ; done
echo
}
main () { set -- $(seq $m)
f2 ""
f2 "$@"
}
n=1000; m=2; main
Which had a 20 seconds delay on the second call with arguments (old
results):
args = 1
1 function no args yes real: 0.012
2 function yes args yes real: 0.033
3 function yes args no real: 0.024
args = 2
1 function no args yes real: 0.011
2 function yes args yes real: 20.446
3 function yes args no real: 0.021
Has almost removed the *reported *delay (as ksh does):
b5b2sh ./script # bash beta 2
args = 1
1 function no args yes real: 0.026
2 function yes args yes real: 0.053
3 function yes args no real: 0.051
args = 2
1 function no args yes real: 0.022
2 function yes args yes real: 0.052
3 function yes args no real: 0.046
However, the delay is still there, just not reported.
Time the whole script to get:
time b5b2sh ./script
real0m32.242s
The delay has actually increased to 32 seconds, while ksh is able to get:
time ksh ./script
real0m0.137s
That's 32.242 / 0.137 = 235.343 times longer.
It seems that the delay has just moved from the deep internal
shuffling of arguments to the phases of creating
(and, now also, erasing) the arguments.
bash b5b2shksh
set arguments 0.110 7.4040.030
args = 1
1 function no args yes 0.014 0.0250.020
2 function yes args yes 0.027 0.0550.020
unset arguments 0.000 0.0000.000
3 function yes args no 0.024 0.0530.020
call function no args 0.080 5.2070.070
args = 2
1 function no args yes 0.011 0.0260.000
2 function yes args yes 20.430 0.0550.010
unset arguments 0.003 4.9730.000
3 function yes args no 0.020 0.0480.010
call function with args20.578 23.5300.040
total20.788s 41.128s0.151
It seems odd that now calling the function without arguments
takes 5.207 seconds and that the un-setting (set --) the arguments
takes 4.973 seconds (setting a pointer to zero should be immediate).
While the call to the function is still taking (externally) ~20 seconds.
This test seems simpler, easier to run:
time b5a0sh -c '
TIMEFORMAT="real: %R";
a=$(seq 2);
f2(){ :; };
echo start>&2;
time set -- $a;
time f2 "$@";
time set --'
Which yields:
bash b5b2sh ksh
0.097 8.1540.020
0.107 39.3210.020
0.003 5.4700.000
0.239s52.975s 0.085s
Re: "return" should not continue script execution, even if used inappropriately
--
*From*: Greg Wooledge
*Subject*: Re: "return" should not continue script execution, even if used
inappropriately
*Date*: Mon, 21 Jan 2019 09:01:33 -0500
--
On Sun, Jan 20, 2019 at 05:43:04PM -0800, don fong wrote:
>* i don't see how this helps. the point is to have one file of code that*
>* behaves differently depending on whether it's dotted in or executed at the*
>* top level.*
https://mywiki.wooledge.org/BashFAQ/109
This seems to work:
[ "$BASH_SOURCE" = "$0" ] && echo "This file is meant to be sourced,
not executed" && exit 1
[ "$BASH_VERSION" ] || { echo "Please use bash" ; return 3; }
[[ "${BASH_VERSINFO[0]}" -le 2 ]] && echo 'No BASH_SOURCE array
variable' && return 2
echo "this file seems to be sourced"
Re: ${p+\"$p\"}
*From*: Dan Jacobson
*Subject*: ${p+\"$p\"}
*Date*: Mon, 21 Jan 2019 21:14:26 +0800
--
So how am I to get
"A"
with bash?
How about:
#!/bin/bash
p=A
q='"'
cat <<_EOF_
${p+$q$p$q}
_EOF_
Works on all shells I tested it.
Re: ${p+\"$p\"}
--
*From*: Robert Elz
*Subject*: Re: ${p+\"$p\"}
*Date*: Mon, 21 Jan 2019 23:38:22 +0700
--
(...)
> With the quotes, most other shells produce the output reported
> from dash (that includes ksh93, yash, ...) zsh just says it iis
> a parse error, which is probably a rational choice!
> Nothing else I tested produces what bash does, which does seem
> to be a kind of odd pair of results.
That is quite inexact, there are four groups:
heirloom sh: "A" \"A\" \\"A\\" \\\"A\\\"
ash: A \"A\" \A\ \\"A\\"
dash : A \"A\" \A\ \\"A\\"
b43sh : A \"A\" \A\ \\"A\\"
b44sh : A \"A\" \A\ \\"A\\"
posixbash : A \"A\" \A\ \\"A\\"
yash : A "A" \A\ \"A\"
y2sh : A "A" \A\ \"A\"
posh : A "A" \A\ \"A\"
lksh : A "A" \A\ \"A\"
mksh : A "A" \A\ \"A\"
ksh93 : A "A" \A\ \"A\"
attsh : A "A" \A\ \"A\"
zsh/sh : ./t.sh:7: parse error
zsh/ksh: ./t.sh:7: parse error
zsh: ./t.sh:7: parse error
But yes:
> But this is an area that is a monumental mess, and the only rational
> choice is to find some workaround which does not involve putting
> a " inside a var expansion inside a here doc.
Maybe:
p=A
q='"'
cat <<_EOF_
${p+$q$p$q} ${p+\\$q$p\\$q}
_EOF_
Which yields "A" \"A\" in all shells tested (except heirloom bourne shell).
Or, even simpler (depending on use):
echo ${p+"$p"} ${p+\"$p\"} ${p+\\"$p\\"} ${p+\\\"$p\\\"}
Re: Unexpected delay in using arguments.
Just to confirm that the delays were removed on the release version. Thanks Chet Ramey () wrote: In the development distributions, always available at > > http://git.savannah.gnu.org/cgit/bash.git/snapshot/bash-devel.tar.gz > > Or you could just clone the git tree. > Many thanks, found and used. I'm going to assume you're running the pre-release distributions on a > Linux system where the debugging code in the bash malloc is enabled. > Yes, a debian based linux system. I am not sure about the debugging code, but it seems like it was the source of the delay. Now, all the delays (using 5.0.2(2)-release) are gone. > So I'm comfortable saying there's no big performance issue here. > Yes, I am confirming this, thanks.
read -t 0 fails to detect input.
It seems that read -t 0 should detect if there is input from a pipe (and
others).
>From man bash:
>> If timeout is 0, read returns immediately, without trying to read any
data.
>> The exit status is 0 if input is available on the specified file
descriptor, non-zero otherwise.
So, it seems that this should print 1:
$ true | read -t 0 var; echo $?
1
And this should print 0 (input available), but it doesn't (most of the
time).
$ echo value | read -t 0 var ; echo $?
1
A little delay seems to get it working:
$ echo value | { read -t 0 var; } ; echo $?
0
Related: Comment to what is wrong with read -t 0:
https://unix.stackexchange.com/questions/33049/how-to-check-if-a-pipe-is-empty-and-run-a-command-on-the-data-if-it-isnt/498065?noredirect=1#comment916652_497121
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -g -O2 -fdebug-prefix-map=/build/bash-2bxm7h/bash-5.0=.
-fstack-protector-strong -Wformat -We$
uname output: Linux iodeb 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2
(2019-11-11) x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu
Bash Version: 5.0
Patch Level: 3
Release Status: release
Fwd: read -t 0 fails to detect input.
To: Chester Ramey On thu., dec. 19 of 2019 at 12:40, Chet Ramey () wrote: > On 12/18/19 6:40 PM, Bize Ma wrote: > > >>> The exit status is 0 if input is available on the specified file > > descriptor, non-zero otherwise. > > Bash-5.0 uses select/FIONREAD to determine whether or not there is input > available on the file descriptor. Those don't wait; they test whether or > not there is input on the specified file descriptor at the point they are > called. > Thanks Chet. Could you please comment about this assertions: 1.- bash will either do a select() or an ioctl(FIONREAD), or neither of them, but not both, as it should for it to work. read -t0 is broken. Do not use it – mosvy. <https://unix.stackexchange.com/questions/33049/how-to-check-if-a-pipe-is-empty-and-run-a-command-on-the-data-if-it-isnt/498064#comment969997_498064> 2.- Conclusion: read -t0 is *broken* in bash. Don't use it. – <https://unix.stackexchange.com/users/308316/mosvy> mosvy <https://unix.stackexchange.com/questions/33049/how-to-check-if-a-pipe-is-empty-and-run-a-command-on-the-data-if-it-isnt/498065?noredirect=1#comment916652_497121>
Re: Fwd: read -t 0 fails to detect input.
On Fri., 20 dec. 2019 at 3:57, Martin Schulte () wrote: > Hello! > > > Could you please comment about this assertions: > > > > 1.- bash will either do a select() or an ioctl(FIONREAD), or neither > > of them, but not both, as it should for it to work. read -t0 is broken. > > 2.- Conclusion: read -t0 is *broken* in bash. Don't use it. – > > No. It works as intended. It's not usable in a pipe in the way you try, > but this is caused by the principles of a pipe, not by a bug in read. > > Hello Martin! I am not the one making those comments. I don't have enough C expertise to neither confirm or deny them. But that also makes me unable to answer to the author of the comments in the proper way. I intended to receive something to correct his (incorect?) opinion. In any case, thanks. Best regards, Isaac
Not operator (~) fail on arithmetic expansion.
Configuration Information: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' -DCONF_VENDOR uname output: Linux zeus 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 GNU/Linux Machine Type: x86_64-pc-linux-gnu Bash Version: 4.4 Patch Level: 5 Release Status: release Description: The ~ operator is called not, and does a one's complement of the following value. That works correctly with $ echo $(( ~1 )) -2 Even with $ echo $(( ~0 )) -1 But fails with this: $ echo $((~0)) bash: /home/user: syntax error: operand expected (error token is "/home/user") Repeat-By: Use $((~0)) (without spaces) to generate the error.
