${1+"$@"} does not generate multiple words if IFS is empty

[martijn]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin11.4.2
Compiler: /usr/bin/clang
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='darwin11.4.2' -DCONF_MACHTYPE='x86_64-apple-darwin11.4.2' 
-DCONF_VENDOR='apple' -DLOCALEDIR='/opt/local/share/locale' -DPACKAGE='bash' 
-DSHELL -DHAVE_CONFIG_H -DMACOSX   -I.  -I. -I./include -I./lib  
-I/opt/local/include -pipe -Os -DSSH_SOURCE_BASHRC -arch x86_64
uname output: Darwin breedzicht.local 11.4.2 Darwin Kernel Version 11.4.2: Thu 
Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin11.4.2

Bash Version: 4.3
Patch Level: 42
Release Status: release

Description:
The substitution ${1+"$@"} should resolve to "$@" if there is at
least one parameter -- i.e. one word per parameter. This works fine
if IFS contains any character or is unset. If IFS is empty, it
instead resolves to the equivalent of "$*", i.e. a single word
concatenating all the parameters without a separator. IFS should
not influence the behaviour of "$@" under any circumstances.

(d)ash variants, AT&T ksh, mksh, zsh, yash all do this correctly.
bash 2.05b.13 also does this correctly (!). The bug is present in
bash 3.2.57 and bash 4.3.42.

(In bash, ${1+"$@"} is redundant, but in some old versions of ksh
and other shells, if 'set -u' is active, "$@" throws an error if
there are no parameters, so ${1+"$@"} is used as a workaround. As a
result, this is often found in cross-platform scripts. These break
on bash 3 & 4 if fieldsplitting is disabled by emptying IFS.)

Repeat-By:
$ IFS=''# bug only occurs on empty IFS
$ set -- this is a test
$ printf '|%s|\n' ${1+"$@"} # generates wrong output
|thisisatest|
$ printf '|%s|\n' "$@"  # generates correct output
|this|
|is|
|a|
|test|

${1+"$@"} does not generate multiple words if IFS is empty

[martijn]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin11.4.2
Compiler: /usr/bin/clang
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='darwin11.4.2' -DCONF_MACHTYPE='x86_64-apple-darwin11.4.2' 
-DCONF_VENDOR='apple' -DLOCALEDIR='/opt/local/share/locale' -DPACKAGE='bash' 
-DSHELL -DHAVE_CONFIG_H -DMACOSX   -I.  -I. -I./include -I./lib  
-I/opt/local/include -pipe -Os -DSSH_SOURCE_BASHRC -arch x86_64
uname output: Darwin breedzicht.local 11.4.2 Darwin Kernel Version 11.4.2: Thu 
Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin11.4.2

Bash Version: 4.3
Patch Level: 42
Release Status: release

Description:
The substitution ${1+"$@"} should resolve to "$@" if there is at
least one parameter -- i.e. one word per parameter. This works fine
if IFS contains any character or is unset. If IFS is empty, it
instead resolves to the equivalent of "$*", i.e. a single word
concatenating all the parameters without a separator. IFS should
not influence the behaviour of "$@" under any circumstances.

(d)ash variants, AT&T ksh, mksh, zsh, yash all do this correctly.
bash 2.05b.13 also does this correctly (!). The bug is present in
bash 3.2.57 and bash 4.3.42.

(In bash, ${1+"$@"} is redundant, but in some old versions of ksh
and other shells, if 'set -u' is active, "$@" throws an error if
there are no parameters, so ${1+"$@"} is used as a workaround. As a
result, this is often found in cross-platform scripts. These break
on bash 3 & 4 if fieldsplitting is disabled by emptying IFS.)

Repeat-By:
$ IFS=''# bug only occurs on empty IFS
$ set -- this is a test
$ printf '|%s|\n' ${1+"$@"} # generates wrong output
|thisisatest|
$ printf '|%s|\n' "$@"  # generates correct output
|this|
|is|
|a|
|test|

Re: [patch #10070] toggle invert flag when reading `!'

[Martijn Dekker]

Op 24-05-21 om 17:47 schreef Chet Ramey:

On 5/22/21 2:45 PM, Vincent Menegaux wrote:

Previously, these commands:

   [[ ! 1 -eq 1 ]]; echo $?
   [[ ! ! 1 -eq 1 ]]; echo $?

would both result in `1', since parsing `!' set CMD_INVERT_RETURN
instead of toggling it.


Interestingly, ksh93 produces the same result as bash. I agree that it's
more intuitive to toggle it.



Also interesting is that '!' as an argument to the simple 'test'/'[' 
command does work as expected (on both bash and ksh93): 'test ! ! 1 -eq 
1' and '[ ! ! 1 -eq 1 ]' return 0/true.


Even the man page for [[ is identical for bash and ksh93:

|   ! expression
|  True if expression is false.

This suggests it's supposed to be a logical negation operator, i.e. '!' 
is implicitly documented to negate another '!'. Bolsky & Korn's 1995 ksh 
book, p. 167, is slightly more explicit about it: "! test-expression. 
Logical negation of test-expression."


I also note that multiple '!' negators in '[[' work as expected on mksh, 
yash and zsh.


So, it seems clear that this is an actual bug in bash and ksh. I see 
it's been fixed in the bash devel tree now. I'll fix it in ksh 93u+m.


--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: Unclosed quotes on heredoc mode

[Martijn Dekker]

Op 20-11-21 om 23:54 schreef Robert Elz:

What the devel one does is unknown to me, I don't think I even have
the means to obtain it (I have nothing at all git related, and no interest
in changing that state of affairs).


Github allows downloading a gzipped tarball of any branch's current 
state via https://github.com///tarball/


There's a regularly updated mirror of the bash repo here:
https://github.com/bminor/bash/

So, the URL to the .tgz for the current bash devel branch is:
https://github.com/bminor/bash/tarball/devel

...and, since it came up in this thread, this one is for the current 
ksh93 development code (not a mirror):

https://github.com/ksh93/ksh/tarball/master

--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Incorrect alias expansion within command substitution

[Martijn Dekker]

On the latest code from the devel branch:
GNU bash, versie 5.2.0(35)-alpha (x86_64-apple-darwin18.7.0)

Reproducer script:

shopt -s expand_aliases
alias let='let --'
set -x
let '1 == 1'
: $(let '1 == 1')

Output:

+ let -- '1 == 1'
++ let -- -- '1 == 1'
foo: line 5: let: --: syntax error: operand expected (error token is "-")
+ :

The alias is incorrectly expanded in the command substitution, 
duplicating the '--' argument and causing a syntax error.


--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: Incorrect alias expansion within command substitution

[Martijn Dekker]

Op 02-02-22 om 13:25 schreef L A Walsh:

I can't say for sure, but it would be interesting if anyone else
has this result in a bash with aliases on by default:

I.e. My bash is posix compliant by default w/r/t aliases:

 env -i /bin/bash --noprofile --norc

bash-4.4$ shopt -p expand_aliases


Of course you're not seeing this bug if you're using bash 4.4. It was 
introduced in bash 5.2 alpha.



--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: Incorrect alias expansion within command substitution

[Martijn Dekker]

Op 01-02-22 om 15:23 schreef Chet Ramey:

Historically, bash (and ksh93) has favored the former. Just about all the
other shells claiming some sort of POSIX conformance favor the latter (all
the ash-based shells, yash, mksh).

What are your plans here?


I've no current plans. Any remotely plausible use of aliases is not 
going to be affected either way. I've done some pretty innovative stuff 
in modernish that involves aliases and that too would be unaffected. In 
my view, this difference is relevant to standards and regression test 
writers and probably no one else.


Having said that, I've never understood why ksh stores command 
substitutions as unparsed source code (including comments and all) in 
the parse tree and only parses that at execution time -- including in 
dot scripts (which are otherwise parsed in their entirety before 
execution) and shcomp bytecode output. That seems bizarre. It doesn't do 
that for regular subshells in parentheses or for process substitutions.


And ksh *still* has $(command substitution) parsing bugs -- a currently 
known one is https://github.com/ksh93/ksh/issues/269 and there are 
probably more yet to be discovered.


So, in a hypothetical future where I will finally have managed to 
understand the uncommented and undocumented dark magic that is the ksh 
parser, lexer, expansion and command substitution code, I would like to 
delete the lexer code that skips over a $(comsub) and instead parse 
comsubs the same way as regular subshells or process substitutions. 
Aliases would then be handled the same way as well.


Don't hold your breath though. The chances of it happening sometime this 
decade are low. Plus, there is still a lot of lower hanging fruit to 
pick in this incredibly buggy code base.


--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Process substitution as part of a word [was: Incorrect alias expansion ...]

[Martijn Dekker]

Op 02-02-22 om 20:59 schreef Chet Ramey:

[...] it makes no sense to join a process substitution to another word.


But it does. In arguments that look like assignments, and in GNU-style 
long options, file names can be part of a larger word.


Assuming tar is GNU tar, something like

tar -c -f txt.tar --files-from=<( ls *.txt )

works fine on bash and zsh, but not on ksh93; the process substitution 
becomes a separate word and the command fails.


This was reported as a bug to the abandoned AT&T ksh2020 development 
effort but never fixed there, and I've no clue how to fix it yet either.

https://github.com/att/ast/issues/1487
https://github.com/ksh93/ksh/issues/215

--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: nofork command substitution

[Martijn Dekker]

Op 22-05-2023 om 16:18 schreef Chet Ramey:

I'd call that a bug. It's not how mksh documents this type of command
substitution to work. ksh93 documents the parsing the same way.


So it does, yet ksh93 also accepts omitting the ;.

The only documentation of this is in the legacy changelog:


08-03-10  The new ${...} command substitution will treat the trailing }
  as a reserved word even if it is not at the beginning of a command,
  for example, ${ date }.


Which means something like 'echo }' breaks in such a comsub.

It is what it is. I will fix the documentation to match reality in ksh 
93u+m.


--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: shopt suggestion

[Martijn Dekker]

Op 31-08-17 om 16:43 schreef Nellis, Kenneth:
> At the keyboard, it is inconvenient to type (e.g.): 
> shopt -s dotglob; mycmd; shopt -u dotglob

Try with a subshell. All it takes is a couple of parentheses.

(shopt -s dotglob; mycmd)

> But, I wonder if shopt might not itself accept a command to execute 
> with one-time shopt options in effect.

The shopt command would have the same problem as your hypothetical
wrapper function: globbing is resolved before the command is even invoked.

- M.

'time' doesn't time subshells that exec

[Martijn Dekker]

The 'time' reserved word seems to be unable to time subshells that run
'exec'. Is this intentional? (ksh93, mksh and zsh all do manage this.)

$ time (sleep 1)

real0m1,003s
user0m0,001s
sys 0m0,002s
$ time (exec sleep 1)
$ time (echo hi; exec sleep 1)
hi
$ echo "$BASH_VERSION"
4.4.12(3)-release

- Martijn

bash-snapshot: 'unset IFS' ignored under very specific circumstances

[Martijn Dekker]

The bug test suite in my modernish shell library exposed a bug in the
current bash development snapshot. As far as I've been able to trace,
the following bug was introduced by bash-snap-20170620 (commit
d7d836dfc55b937f463f601ba5117d6442053089).

I've been able to distil the following test case:

set -o posix
fn() {
IFS= read junk 

Spaces trimmed from $* when assigned while IFS is unset [was: Unexpected word splitting on $* ...]

[Martijn Dekker]

Op 20-06-17 om 02:13 schreef Kevin Brodsky:
> When IFS is unset, unquoted $* undergoes word splitting as if IFS=' ',
> and not the expected IFS=$' \t\n'.

Possibly related are the following two issues:

1. When IFS is unset, assignments of unquoted ${var-$*}, ${var+$*}, etc.
to a variable removes leading and trailing spaces from $* (but not tabs
or newlines). (Modernish ID: BUG_PP_03B)
This issue exists on bash 4.3, 4.4 and current git.

Test script:

set "  abc  " " def  ghi " "jkl "
unset -v IFS var
var=${var-$*}/${var-$*}
printf '[%s]\n' "$var"

Actual output:
[abc def ghi jkl/abc def ghi jkl]

Expected output:
[  abcdef  ghi  jkl /  abcdef  ghi  jkl ]

2. When IFS is unset, conditional assignments of unquoted $* to a
variable within a parameter substitution, e.g. ${var=$*} or ${var:=$*},
removes leading and trailing spaces (but not tabs or newlines).
(Modernish ID: BUG_PP_04A)
This issue exists on all bash versions from current git down to at least
2.05b.

set "  abc  " " def  ghi " "jkl "
unset -v IFS var
: ${var:=$*/$*/${var-$*}}
printf '[%s]\n' "$var"

Actual output:
[abc def ghi jkl / abc def ghi jkl /abc def ghi jkl]

Expected output:
[  abcdef  ghi  jkl /  abcdef  ghi  jkl /  abcdef  ghi  jkl ]

Thanks,

- Martijn

Re: Spaces trimmed from $* when assigned while IFS is unset [was: Unexpected word splitting on $* ...]

[Martijn Dekker]

Op 27-09-17 om 14:44 schreef Greg Wooledge:
> I'm just going to chalk this up as yet another example of unquoted $*
> or $@ being Completely Wrong.

Nonsense. This is a bug in bash and it should be fixed, not excused.
Quoting expansions should never be necessary for assignments.

This strange behaviour with assignments is unique to bash. All other
shells act identically and correctly.

- M.

Re: Spaces trimmed from $* when assigned while IFS is unset [was: Unexpected word splitting on $* ...]

[Martijn Dekker]

Op 04-10-17 om 17:52 schreef Chet Ramey:
> It's interesting that other shells treat ${a:=b} as kind of like an
> assignment statement (word splitting) but not quite (tilde expansion).

Hmm...

v=~/bla
printf '%s\n' "$v"

outputs /Users/martijn/bla on all shells, so tilde expansion applies for
a simple assignment anyway.

And

unset -v v
v=${v:=~/bla}
printf '%s\n' "$v"

acts exactly the same.

This is all POSIXly correct, IIRC.

Of course, when ${v:=~/bla} is an unquoted expansion in a non-assignment
context, field splitting and pathname expansion apply to the expansion,
but not to the assignment itself, which is also perfectly logical:

[ -n "${ZSH_VERSION+s}" ] && emulate sh
unset -v v
touch /tmp/foo.bar /tmp/baz.bar
printf 'field: [%s]\n' ${v:=~/one ~/two /tmp/*.bar}
printf '  var: [%s]\n' "$v"

Output on bash, dash, FreeBSD sh, Busybox ash, yash, pdksh, mksh, ksh93,
zsh up to 5.0.8:

field: [/Users/martijn/one]
    field: [~/two]
field: [/tmp/baz.bar]
field: [/tmp/foo.bar]
  var: [/Users/martijn/one ~/two /tmp/*.bar]

Output on zsh 5.1 and later, it seems to do tilde expansion after word
splitting instead of before -- which looks like a bug[*2], I'll take
that up there:

field: [/Users/martijn/one]
field: [/Users/martijn/two]
field: [/tmp/baz.bar]
field: [/tmp/foo.bar]
  var: [/Users/martijn/one ~/two /tmp/*.bar]

Output on bosh (schilytools) and NetBSD sh -- clearly a bug, it doesn't
do tilde expansion at all:

field: [~/one]
field: [~/two]
    field: [/tmp/baz.bar]
field: [/tmp/foo.bar]
  var: [~/one ~/two /tmp/*.bar]

- Martijn

[*1]
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_01
[*2]
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06

command_not_found_handle documentation omission

[Martijn Dekker]

The bash manual and info pages state:

| If the search is unsuccessful, the shell searches for a
| defined shell function named 'command_not_found_handle'.  If that
| function exists, it is invoked with the original command and the
| original command's arguments as its arguments, and the function's
| exit status becomes the exit status of the shell.

This fails to mention that command_not_found_handle() is run in the
subshell forked to 'exec' the command, so even an explicit 'exit' will
not exit anything but that subshell. It also means a command handler
can't do anything that influences the main shell, except send it a
signal with 'kill'.

The phrasing "the function's exit status becomes the exit status of the
shell" is misleading to anyone without this knowledge, as it implies
that the shell will exit with that status, whereas in fact all that
happens is that $? is set to the function's exit status.

- Martijn

foo=$*: ^A and DEL are prefixed or removed

[Martijn Dekker]

Here's another corner-case bug with assigning $* to a variable (i.e.:
foo=$*). If IFS is empty, the $* expansion removes any $'\001' (^A) and
$'\177' (DEL) characters. If IFS contains a value, each ^A and DEL
character is prefixed by another $'\001'. If IFS is unset, the bug does
not show up at all.

This is another case where quoting the $* (i.e.: foo="$*") works around
the bug, yet it's still a bug.

Test script:

fn() {
foo=$*
printf '%s' "$foo" | od -c | awk 'NR==1 { $1=""; print; }'
}
teststring=$(printf '\001\002\003\177')
for IFS in '' ' ' 'X' ' X'; do
fn "$teststring"
done
unset -v IFS
fn "$teststring"

Expected output (and actual output from every non-bash shell):

 001 002 003 177
 001 002 003 177
 001 002 003 177
 001 002 003 177
 001 002 003 177

Actual output (bash 4.4.12, bash-20171110 snapshot):

 002 003
 001 001 002 003 001 177
 001 001 002 003 001 177
 001 001 002 003 001 177
 001 002 003 177

Actual output (bash 4.3.39, 4.2.53, 4.1.17, 3.2.57):

 001 002 003 177
 002 003
 001 002 003 177
 001 002 003 177
 001 002 003 177

Actual output (bash 2.05b):

 001 002 003
 001 002 003
 001 002 003
 001 002 003
 001 002 003

- Martijn

Re: foo=$*: ^A and DEL are prefixed or removed

[Martijn Dekker]

Op 16-12-17 om 17:06 schreef Chet Ramey:
> On 11/24/17 3:17 AM, Martijn Dekker wrote:
>> Here's another corner-case bug with assigning $* to a variable (i.e.:
>> foo=$*). If IFS is empty, the $* expansion removes any $'\001' (^A) and
>> $'\177' (DEL) characters. If IFS contains a value, each ^A and DEL
>> character is prefixed by another $'\001'. If IFS is unset, the bug does
>> not show up at all.
> 
> Thanks for the report. I'll fix this for the next version.

FWIW, here's a test script. It's POSIX compatible so you can compare
with other shells. Correct output is
soh stx etx del   / soh stx etx del
for all cases.

- M.

defaultIFS=$IFS
set -o errexit -o noglob
(set -o pipefail) 2>/dev/null && set -o pipefail
teststring=$(printf '\1\2\3\177')
n=0

trim_od() {
od -a | sed -n '1 { s/^0*[[:blank:]]*//; s/[[:blank:]]*$//; p; }'
}

doTest() {
set -- "$teststring"
eval "$testcmd"
case ${IFS+s}${IFS:+n} in
( sn )  i=$(printf %s "$IFS" | trim_od) ;;
( s )   i='(null)' ;;
( '' )  i='(unset)' ;;
( * )   echo 'internal error!' >&2; exit 125 ;;
esac
printf '\n%03d: IFS = %s: %s\n' "$((n+=1))" "$i" "$testcmd"
printf %s "$*${foo+/}${foo-}" | trim_od
}

doAllTests() {
for testcmd in \
'unset -v foo; set -- ${foo=$*}' \
'unset -v foo; set -- ${foo="$*"}' \
'unset -v foo; set -- "${foo=$*}"' \
\
'foo=; set -- ${foo:=$*}' \
'foo=; set -- ${foo:="$*"}' \
'foo=; set -- "${foo:=$*}"' \
\
'unset -v foo; set -- ${foo=$@}' \
'unset -v foo; set -- ${foo="$@"}' \
'unset -v foo; set -- "${foo=$@}"' \
\
'foo=; set -- ${foo:=$@}' \
'foo=; set -- ${foo:="$@"}' \
'foo=; set -- "${foo:=$@}"'
do
doTest "$testcmd"
done
}

unset -v IFS; doAllTests
IFS=''; doAllTests
IFS='x'; doAllTests
IFS=$defaultIFS; doAllTests

Re: foo=$*: ^A and DEL are prefixed or removed

[Martijn Dekker]

The latest development snapshot appears to have fixed this bug.

However, it introduced a new one: if the positional parameters are
empty, then

foo=$*

produces a segfault.

Thanks,

- Martijn

Re: make distclean breaks "later "configure && makes"", i.e., removes a file in distro that a build needs (cannot process parse.y)

[Martijn Dekker]

Op 14-02-18 om 14:29 schreef Greg Wooledge:
> On Wed, Feb 14, 2018 at 01:03:05PM +0100, Michael Felt wrote:
>> If after a successful build, I run "make distclean" - "./configure && make" 
>> no longer works.
> 
>> configure: WARNING: bison not available; needed to process parse.y
>> + /usr/bin/make > .buildaix/make.out
>> yacc: not found
> 
> You need to install the bison package.  The tarballs of bash source
> include an already-processed yacc file, but "make distclean" removes
> that, and you need bison (or yacc, but bison is preferred) to recreate
> it.

But isn't it the point of "make distclean" to delete everything *except*
the files that come with the tarball distribution?

- M.

[off-topic] ksh local vars [was: Unset array doesn't work]

[Martijn Dekker]

Op 05-03-18 om 13:57 schreef Greg Wooledge:
> On Sat, Mar 03, 2018 at 08:24:06PM +0900, Koichi Murase wrote:
>> - Note: ksh seems not support local variables. "typeset" in function
>> scopes defines a global variable, and "unset" removes the global
>> variable.
> 
> "Real ksh" has two different kinds of functions -- ones declared with
> "foo()" and ones declared with "function foo".  They have different
> treatment of local variables, and the ksh man page does not clearly
> (at least to me) describe the difference.

You have to distinguish between two kinds of ksh here.

In the old "real ksh", AT&T ksh88 (which is closed source and only in
current use as /usr/xpg4/bin/sh on Solaris), functions defined with
"foo()" and with "function foo" have identical behaviour with regards to
local variables. They have dynamic scoping, that is, if x calls y, then
y has access to x's local variables. This is the model that every other
current shell, including bash, emulates -- each with its own particular
set of quirks and gotchas, of course. That includes the "ksh clones"
(pdksh, mksh, et al) which are all ksh88 clones.

In the current "real ksh", AT&T ksh93, things changed in radical and
incompatible ways. Functions defined with "foo()" don't have local
variables at all. Functions defined with "function foo" have a new model
for local variables: static scoping. That is, if x calls y, then y does
*not* have access to x's local variables. In static scoping, "unset" in
a local scope can never influence the global scope. No other shell to
date has emulated the ksh93 model.

HTH,

- M.

[PATCH] allow file modes up to 7777 instead of 777

[Martijn Dekker]

This fixes two bugs:

1. The example 'mkdir' builtin, examples/loadables/mkdir.c, has a broken
'-m' option that doesn't accept sticky/setuid/setgid.

$ ./bash -c '(cd examples/loadables && make mkdir) &&
enable -f examples/loadables/mkdir mkdir &&
mkdir -m2775 foo'
['make' output skipped]
./bash: line 2: mkdir: invalid file mode: 2775

2. The 'umask' builtin doesn't accept modes > 0777.

$ ./bash -c 'umask '
./bash: line 0: umask: : octal number out of range

While POSIX says the effect of specifying sticky/setuid/setgid bits is
unspecified[*], the 'umask' builtin should still accept these bits, as
it does in every other shell.

It is also not consistent as bash will happily inherit an umask of 
from another process. For example:
$ ksh -c 'umask ; bash -c "umask"'

This is on MacOS and the BSDs; Linux seems to clear the first 8 bits at
the kernel level, so '0777' is output. That is a Linux-specific
constraint, though, and bash should still not accept through one route
what it rejects through another.

Patch:

diff --git a/builtins/common.c b/builtins/common.c
index a5f2584..0752f0d 100644
--- a/builtins/common.c
+++ b/builtins/common.c
@@ -537,7 +537,7 @@ read_octal (string)
 {
   digits++;
   result = (result * 8) + (*string++ - '0');
-  if (result > 0777)
+  if (result > 0)
return -1;
 }

By the way, why does that function repeatedly check the bounds for every
digit? Wouldn't this be more efficient:

diff --git a/builtins/common.c b/builtins/common.c
index a5f2584..6f1273f 100644
--- a/builtins/common.c
+++ b/builtins/common.c
@@ -537,11 +537,9 @@ read_octal (string)
 {
   digits++;
   result = (result * 8) + (*string++ - '0');
-  if (result > 0777)
-   return -1;
 }

-  if (digits == 0 || *string)
+  if (digits == 0 || *string || result > 0)
 result = -1;

   return (result);

Note: parse.y also uses read_octal() for $PS1, but it enforces a
three-character limit, so it's inherently limited to octal 777 and the
bounds check change in read_octal() doesn't affect it. I cannot find any
other uses of read_octal().

Thanks,

- M.

Re: [PATCH] allow file modes up to 7777 instead of 777

[Martijn Dekker]

Op 14-03-18 om 16:49 schreef Martijn Dekker:
> While POSIX says the effect of specifying sticky/setuid/setgid bits is
> unspecified[*], the 'umask' builtin should still accept these bits, as
> it does in every other shell.

Forgot the footnote link. Here it is:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/umask.html#tag_20_138_05

- M.

Re: [PATCH] allow file modes up to 7777 instead of 777

[Martijn Dekker]

Op 14-03-18 om 16:49 schreef Martijn Dekker:
[...]
>  {
>digits++;
>result = (result * 8) + (*string++ - '0');
> -  if (result > 0777)
> +  if (result > 0)
>   return -1;
>  }
> 
> By the way, why does that function repeatedly check the bounds for every
> digit?

It just occurred to me that, given a huge number argument, this
disallows the value to wrap around and return within the bounds.

- M.

[BUG] ERR trap triggered twice when using 'command'

[Martijn Dekker]

$ bash -c 'trap "echo WHOA" ERR; command false'
WHOA
WHOA

Expected output: just one WHOA (as on *ksh, zsh)

Thanks,

- M.

Re: [BUG] ERR trap triggered twice when using 'command'

[Martijn Dekker]

Op 02-04-18 om 16:47 schreef Nick Chambers:

You should still be providing this information :). How else would
you  expect people to help you.


I'm not looking for help. I'm here to report the bugs in bash that I 
find, and occasionally to help fix them. That's what this list is for.



 > NickChambers-iMac:~ Nick$ type command
 > command is a shell builtin
 >
 > This means that when you execute `command false`, false is executed
 > in a child subshell and command is executed in the current shell, and
 > both return 1.
 
 It means no such thing.


It means exactly that.

[...]

No use in arguing further. You are quite simply wrong.

- M.

Re: [BUG] ERR trap triggered twice when using 'command'

[Martijn Dekker]

Op 02-04-18 om 20:41 schreef Chet Ramey:

Thanks for the report. This clearly falls under this clause

"In every other respect, if command_name is not the name of a function, the
effect of command (with no options) shall be the same as omitting command."

in the Posix description of `command'. It should be a one-line fix.


Given that text, commands prefixed with 'command' maybe shouldn't appear 
twice in xtrace output either. (They don't in any other shell.)


$ bash -xc 'command true'
+ command true
+ true

$ dash -xc 'command true'
+ command true

$ ksh93 -xc 'command true'
+ command true

$ pdksh -xc 'command true'
+ command true

$ yash -xc 'command true'
+ command true

$ mksh -xc 'command true'
+ true

$ zsh -xc 'command true'
+zsh:1> true

Thanks,

- M.

[BUG] RETURN trap with -o functrace: infinite recursion on 'eval return'

[Martijn Dekker]

$ bash -o functrace -c 'fn() { eval "return"; }; trap "fn" RETURN; fn'

Result: segfault due to infinite recursion.

__

Even apart from this bug with 'eval return', the effect of '-o 
functrace' is a bit strange:


$ bash -o functrace -c 'fn() { printf foo; fn2; };
fn2() { printf bar; fn3; };
fn3() { printf baz\\n; };
trap fn RETURN; fn'
foobarbaz
foobarbaz
foobarbaz
foobarbaz

(I'd expect one 'foobarbaz', not four)

It seems odd that the RETURN trap would be triggered while a RETURN trap 
action is still being executed. Might it be better to temporarily 
deactivate the effect of '-o functrace' while a RETURN trap action is 
being executed?


Thanks,

- M.

Re: [BUG] RETURN trap with -o functrace: infinite recursion on 'eval return'

[Martijn Dekker]

Op 14-04-18 om 03:49 schreef Chet Ramey:

On 4/10/18 5:56 AM, Martijn Dekker wrote:

It seems odd that the RETURN trap would be triggered while a RETURN trap
action is still being executed. Might it be better to temporarily
deactivate the effect of '-o functrace' while a RETURN trap action is being
executed?


Well, trap handlers are recursive, in the sense that you can execute a trap
on signal X from a signal X trap handler.


I'm not sure how that would happen. Isn't a signal blocked while 
executing its trap handler?



 If the RETURN trap is inherited
by functions, and traps are recursive, wouldn't the bash behavior be the
logical thing to do?


I suppose. But if a signal is blocked while executing the respective 
trap, perhaps it would also be logical to deactivate the RETURN 
pseudosignal while executing a RETURN trap.


It seems like this is already done for the ERR pseudosignal, as "trap 
false ERR; false" does not cause an infinite loop.


- M.

[BUG] failure to push/restore closed file descriptor

[Martijn Dekker]

$ bash -c '{ exec 8Apparently, bash either fails to push the file descriptor onto the stack 
at '} 8<&-', or fails to restore it.


Same bug with loops ending in "done 8<&-".

Confirmed in all bash versions down to 2.05b.

Also note that pushing an open file descriptor works as expected:

$ bash -c '{ exec 8open"'

Output: Bad file descriptor (as expected)

FYI:
* bash and dash have this bug;
* zsh, yash, AT&T ksh93, pdksh, mksh, posh, schily's bosh, Busybox ash, 
FreeBSD sh, and NetBSD sh all do this correctly.


Thanks,

- Martijn

[BUG] persistently assigned variable cannot be unexported in POSIX mode

[Martijn Dekker]

POSIX allows assignments preceding special builtins such as ':', 'set', 
etc. not only to persist but also to be persistently exported. I think 
that exporting behaviour is inherently broken and it's unfortunate that 
bash does this in POSIX mode, as it's merely allowed and not actually 
mandated, but anyway...


What I'm reporting here is a bug I discovered with unexporting a 
variable that is so exported while bash is in POSIX mode. It cannot be 
unexported using 'typeset +x' if you try to do that in a shell function.


This works:

$ bash -o posix -c 'foo=abc : ; typeset +x foo; env|grep ^foo='
(no output, as expected: no longer exported)

But this doesn't:

$ bash -o posix -c 'fn() { foo=abc : ; typeset +x foo; env|grep ^foo=; 
}; fn'

foo=abc

Bug confirmed on all bash versions down to 2.05b.

I've also succeeded in making 'unset -v' fail silently for a variable 
that is so exported, but I've so far been unable to track down a 
specific reproducer that is simple enough to post here. It occurred as a 
failure in the modernish regression test suite, which is extensive. 
Chet, let me know if you want a reproducer for that and I'll email you a 
tarball plus instructions. It seems likely that fixing the bug reported 
above would fix that one as well, though.


- M.

Re: [BUG] persistently assigned variable cannot be unexported in POSIX mode

[Martijn Dekker]

Op 27-04-18 om 22:16 schreef Chet Ramey:

On 4/25/18 10:51 PM, Martijn Dekker wrote:


What I'm reporting here is a bug I discovered with unexporting a variable
that is so exported while bash is in POSIX mode. It cannot be unexported
using 'typeset +x' if you try to do that in a shell function.

This works:

$ bash -o posix -c 'foo=abc : ; typeset +x foo; env|grep ^foo='
(no output, as expected: no longer exported)

But this doesn't:

$ bash -o posix -c 'fn() { foo=abc : ; typeset +x foo; env|grep ^foo=; }; fn'
foo=abc


It seems like you're assuming that in posix mode, variable assignments that
precede special builtins executed in shell functions should create local
variables. Is that correct?


No. The issue is:

1. In POSIX mode, as ':' is a so-called special builtin, "foo=abc :" 
causes the assignment to 'foo' to persist past the ':' command, and 
'foo' also remains exported. This is all POSIX-compliant.


(I really wish it didn't remain exported, though. POSIX allows it, but 
doesn't mandate it; it's unspecified whether it remains exported or not. 
But it seems broken for it to remain exported. In fact I'm not sure why 
it's exported to begin with. As far as I know, there are no special 
builtins that run external binaries.)


2. The bug is: 'declare +x' a.k.a. 'typeset +x' then fails to unexport 
the variable in the second version above. The variable remains exported 
past 'typeset +x foo', as proven by grepping the output of 'env'.


It shouldn't matter if the variable is local to the function or not, as 
everything is done in the same scope. 'declare +x' should clear the 
export flag as it is documented to do. In the second invocation quoted 
above, it doesn't.


Having said that, I now cannot reproduce the issue in the 2018-04-27 
development snapshot... so thanks for fixing it anyhow :)


- Martijn

Re: [BUG] persistently assigned variable cannot be unexported in POSIX mode

[Martijn Dekker]

Op 30-04-18 om 21:12 schreef Chet Ramey:

2. The bug is: 'declare +x' a.k.a. 'typeset +x' then fails to unexport the
variable in the second version above. The variable remains exported past
'typeset +x foo', as proven by grepping the output of 'env'.

Now you've just created a local variable (typeset in a function creates
local variables, even in Posix mode) and made sure it doesn't have the
export attribute. You haven't given it a value, so it remains unset and
doesn't shadow the exported global you created in step 1.


Of course, that makes complete sense.

However, when adding the -g option, on bash 4.3.30, 4.4.19 and dev 
snapshot 2018-04-20:


$ bash -o posix -c 'fn() { foo=abc : ; typeset -g +x foo; env|grep 
^foo=; }; fn'

foo=abc

Causing 'typeset' to operate on the global scope had no effect. I had 
noticed that it made no difference, so I incorrectly figured the -g flag 
wasn't relevant and left it out of my bug report.


In any case, that still looks like a bug to me... particularly as bash 
4.2.45 works as expected, so it seems to break as of 4.3.


- M.

[BUG] 'unset' fails silently under specific conditions

[Martijn Dekker]

Op 26-04-18 om 04:51 schreef Martijn Dekker:
I've also succeeded in making 'unset -v' fail silently for a variable 
that is so exported, but I've so far been unable to track down a 
specific reproducer that is simple enough to post here.


Here is that reproducer. The following appears to be the minimum 
necessary to consistently make 'unset' silently fail on bash 4.2.45, 
4.3.30, 4.4.19 and current git. (I confirmed the bug is *not* present on 
bash 4.0.38, 3.2.48, 3.1.0, 3.0.16 and 2.05b.)


POSIXLY_CORRECT=y
func1() {
  var=1
  var=2 :   # or 'var=2 return', or another special builtin
}
func2() {
  func1
  unset -v var  # bug: fails silently
}
func2
echo ${var+"BUG: still set"}

Output: 'BUG: still set'
Expected output: none

It appears that it must have one function calling another, and that 
other function having an assignment preceding a special builtin, for 
'unset' to fail silently for the variable so assigned.


Thanks,

- M.

Re: [BUG] 'unset' fails silently under specific conditions

[Martijn Dekker]

Op 01-05-18 om 18:45 schreef Martijn Dekker:
It appears that it must have one function calling another, and that 
other function having an assignment preceding a special builtin, for 
'unset' to fail silently for the variable so assigned.


Actually it can be simplified further: the following also outputs "BUG: 
still set". I fact, with this version, the bug manifests all the way 
down to bash 2.05b!


POSIXLY_CORRECT=y
func() {
  var=1
  var=2 :   # or 'var=2 set foo', or another special builtin
  unset -v var  # bug: fails silently
}
func
echo ${var+"BUG: still set"}

Thanks,

- M.

Re: [BUG] 'unset' fails silently under specific conditions

[Martijn Dekker]

Op 01-05-18 om 19:36 schreef Chet Ramey:

OK, so which is it? Does an assignment statement preceding a special
builtin in a shell function create a local variable (as we discussed
just yesterday)


Huh? That was about 'typeset'/'declare'. How is that relevant here?


 or does it create a global variable because `Posix'?


Yes. POSIX does not specify local variables at all, and even in a bash 
context I see nothing there to indicate that it should be local.


The possibility that 'foo=bar :' might create a variable 'foo' that 
survives the ':' command, is exported, *and* is local to the function, 
hadn't occurred to me at all. POSIX is pretty strange sometimes but not 
*this* strange.


- M.

Re: [BUG] 'unset' fails silently under specific conditions

[Martijn Dekker]

Op 02-05-18 om 02:20 schreef Chet Ramey:

You complained that `typeset +x' didn't `unexport' a variable.  The reason > is 
that the variable assignment preceding the special builtin caused a
variable to be created at the global scope, and the `typeset' resulted in
a local variable being created.


I still can't see how that is relevant here: 'typeset'/'declare' is not 
involved in the current issue. But then, neither is 'unset'. It does 
appear that the current issue is not what I thought it was.


Instead, in POSIX mode, it looks like a variable assignment preceding a 
special builtin may create a variable at a function-local scope without 
'typeset'/'declare' being involved at all. But not always.


Let's see if I'm getting it right this time. In the following:
set -o posix
f() { foo=bar : ; }
f
the command 'foo=bar :'
1. makes the assignment 'foo=bar' survive the ':' command;
2. gives 'foo' a global scope;
3. permanently exports the global variable 'foo'.

However, in the following:
set -o posix
f() { foo=bar; foo=baz : ; }
f
the plain assignment 'foo=bar' creates an ordinary global variable named 
'foo' with value 'bar', and then the command 'foo=bar :'
1. makes the assignment 'foo=baz' survive the ':' command, but by 
creating *another* 'foo' instead of overwriting the first;

2. gives that other 'foo' a function-local scope;
3. exports the local variable 'foo' for the duration of its existence.

My testing confirms that this is what appears to happen, and I think 
it's a bug, because (1) POSIX has no notion of variables with a 
function-local scope; (2) even if it did, no command was issued that 
should make the variable local to the function; and (3) the behaviour in 
the second example is inconsistent with that in the first.


I think 'foo=baz :' should act the same in the second example as 
'foo=bar :' does in the first, i.e.: if there is already a variable by 
that name it should be overwritten, just like with any normal shell 
assignment.


- M.

Re: [BUG] 'unset' fails silently under specific conditions

[Martijn Dekker]

Op 02-05-18 om 15:38 schreef Greg Wooledge:

*NOW* there's evidence of a local variable, because unset removes it
and exposes the global.


Right. That's what triggered my initial report, in which I confused that 
phenomenon for a failure of the 'unset' command to unset the variable.



  But it's not "permanently exported" as far as  I can see.


Sorry for the inexact use of language -- I simply meant that the global 
scope of the variable acquired the export flag, as opposed to the 
function-local scope.


- M.

[BUG] assignments preceding 'read' persist past command in POSIX mode

[Martijn Dekker]

In bash-20180511 and bash-20180518 snapshots and in bash 5.0.0-alpha, in 
POSIX mode, assignments preceding 'read' incorrectly persist past the 
command. 'read' is a regular builtin so they should not persist.


$ ./bash -o posix -c 'v=ok; v=bug read x 

Re: $OPTIND varibale value is different from sh

[Martijn Dekker]

Op 20-06-18 om 13:39 schreef Greg Wooledge:

I really don't understand what you're doing here, either.  The only
use of OPTIND is after the final call to getopts, when there are no
more options to process.  At that point, OPTIND tells you how many
times you have to "shift" to get rid of all the options that were
processed.  (And you have to subtract 1, because legacy historical
reasons.)


In other words, because all the options have now been processed, OPTIND 
points to the first argument *after* the list of options. I don't see 
how that is "because legacy historical reasons". It seems both logical 
and useful to me.



Any attempt to use OPTIND in the middle of the loop seems pointless
and hazardous to me.


Agreed.

- M.

Re: $OPTIND varibale value is different from sh

[Martijn Dekker]

Op 20-06-18 om 17:45 schreef Ilkka Virta:
$ for sh in dash 'busybox sh' bash ksh93 zsh ; do printf "%-10s: " 
"$sh"; $sh -c 'while getopts abcd opt; do printf "$OPTIND  "; done; 
printf "$OPTIND  "; shift $(($OPTIND - 1)); echo "$1" ' sh -a -bcd 
hello; done

dash  : 2  3  3  3  3  hello
busybox sh: 2  3  3  3  3  hello
bash  : 2  2  2  3  3  hello
ksh93 : 2  2  2  3  3  hello
zsh   : 1  2  2  2  3  hello


yash has interesting behaviour here. While other shells use an extra 
internal (i.e. not exposed to the shell language) state variable to keep 
track of parsing multiple options combined in a single argument, yash 
just adds that extra state value to OPTIND.


yash  : 1:2  2:2  2:3  2:4  3  hello

Which demonstrates quite well that the value of OPTIND in the middle of 
processing really should be considered unspecified, and that only the 
initial value (1) and the final value can be relied upon.


Relevant POSIX text:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getopts.html#tag_20_54
| If the application sets OPTIND to the value 1, a new set of
| parameters can be used: either the current positional parameters or
| new arg values. Any other attempt to invoke getopts multiple times in
| a single shell execution environment with parameters (positional
| parameters or arg operands) that are not the same in all invocations,
| or with an OPTIND value modified to be a value other than 1, produces
| unspecified results.


- M.

Re: Directing into a variable doesn't work

[Martijn Dekker]

Op 24-06-18 om 05:08 schreef Peter Passchier:

With memory being abundant and filesystem access expensive, I want to
put stdout and stderr of a command into variables (without needing to
write to a file):

output=$($command 2>>>errors)


This would not work even if the feature is implemented. The $(command 
substitution) forks a subshell process to execute the command in, 
because the main shell process needs a process it can set up a pipe 
with. So the 'errors' variable would only exist in that subshell process 
and would be lost as soon as the command substitution completes.



Or:

$command >>>output 2>>>errors


This form seems conceivable to me.

However, note that here-documents and here-strings internally use 
temporary files, so they do involve file system access. I'm not Chet, 
but I don't think that would be different for your proposed feature. So 
while this might be some nice syntactic sugar, I'm afraid you would be 
disappointed about the performance.


I still kind of like the idea, though. As far as I know, there's 
currently no way to capture more than one output stream into separate 
variables without involving rather laborious handling of temporary 
files. Your proposal would probably still involve that, but the shell 
would do the hard work for you which seems like an improvement to me.


BTW, 'reverse here-document' doesn't sound quite right. You're not 
specifying any document or string containing input, you're specifying a 
variable in which to store output. So, here-variable?


- M.

Re: [minor] umask 400 causes here-{doc,string} failure

[Martijn Dekker]

Op 11-03-18 om 17:31 schreef Ilkka Virta:

On 11.3. 17:17, Stephane Chazelas wrote:

$ bash -c 'umask 400; cat <<< test'
bash: cannot create temp file for here-document: Permission denied



Those shells use temporary files to store the content of the
here-documents as the Bourne shell initially did, and open them
in read-only mode to make it cat's stdin.

When umask contains the 0400 bit, the file is created without
read permission to the user, hence the error upon that second
open().
I can think of several ways to address it:

1- do nothing and blame the user
2- open the file only once for both
writing the content and making it the command's stdin
3. use a pipe instead of a temp file
4. Reset the umask temporarily to 077


One more came to mind:

5. manually chmod() the tempfile to 0400 or 0600 if the open() for 
reading fails with EACCES, and then retry. Should be doable with a 
localized change to that particular error condition, without changing 
the overall behaviour.


Unless I'm missing something, there should be no reason for an internal 
temp file to have any permissions other than 0600 (user 
readable/writable), so it seems to me that an fchmod call straight after 
creating the file and before returning the fd is the simplest way of 
fixing the bug; this makes the permissions of internal temp files 
entirely independent of the umask.


diff --git a/lib/sh/tmpfile.c b/lib/sh/tmpfile.c
index e41e45b..1805cdf 100644
--- a/lib/sh/tmpfile.c
+++ b/lib/sh/tmpfile.c
@@ -203,7 +203,6 @@ sh_mktmpfd (nameroot, flags, namep)
 }
   if (namep)
 *namep = filename;
-  return fd;
 #else /* !USE_MKSTEMP */
   sh_seedrand ();
   do
@@ -224,8 +223,9 @@ sh_mktmpfd (nameroot, flags, namep)
   else
 free (filename);

-  return fd;
 #endif /* !USE_MKSTEMP */
+  fchmod(fd, S_IRUSR | S_IWUSR);
+  return fd;
 }

 FILE *

Another ^A bug

[Martijn Dekker]

That ^A / $'\1' character just keeps causing trouble...

This is a bug with IFS. Including a $'\1' character anywhere in IFS 
causes expansions in shell assignments to discard that character.


$ bash -c "c=$'Y\1Y';
for IFS in $'\1' $'\2' $'x\1' $'x\2' $'\1x' $'\2x'; do
v=X\${c}X;
printf %s \"\$v\" | od -a | awk 'NR==1 { \$1=\"\"; print }';
done"

Output on bash 4.4 and current git:
 X Y Y X
 X Y soh Y X
 X Y Y X
 X Y soh Y X
 X Y Y X
 X Y soh Y X

Expected output, on bash 4.3 and earlier, and all other shells:
 X Y soh Y X
 X Y soh Y X
 X Y soh Y X
 X Y soh Y X
 X Y soh Y X
 X Y soh Y X

Note that quoting the assigned value is an effective workaround. But in 
shell grammar, IFS should never have any influence on true shell 
assignments (as opposed to assignment-arguments) and quoting expansions 
should not be necessary. (This paragraph is included to pre-empt readers 
who would reply "just quote everything". Yes, I know. It's still a bug.)


Thanks,

- Martijn

Optimisation request

[Martijn Dekker]

I've noticed bash-5.0beta is quite a bit faster than bash-4.4 in running 
a lot of pure shell code. Good news.


One optimisation that isn't done yet: if a subshell function like

f() (foo)

is used in a command substitution, it forks twice. I would like it if it 
could be optimised to eliminate the fork if the subshell function is the 
last command in the command substitution.


Thanks,

- M.

[minor] for/select parsing inconsistency

[Martijn Dekker]

There is a minor inconsistency in grammatical parsing between 'for' and 
'select' loops. Since 'select' is basically a glorified 'for', it seems 
to me that they should parse the same way.


$ bash -c 'for x in; do :; done'
$ bash -c 'select x in; do :; done'
bash: -c: line 0: syntax error near unexpected token `;'
bash: -c: line 0: `select x in; do :; done'
$ bash -c 'foo=; select x in $foo; do :; done'
$

An empty iteration argument list is not accepted by 'select', unless it 
results from an expansion. 'for' does accept this.


On ksh93, mksh, and zsh, both 'for' and 'select' accept a literal empty 
list. On shells without 'select' (dash, yash), 'for' accepts it.


So it seems to me that 'select' on bash should change to match 'for'.

Thanks,

- M.

'eval' exit behaviour in posix mode

[Martijn Dekker]

In current git, 'bash -o posix' no longer exits the shell on a syntax 
error in the argument to the special builtin 'eval'. Bash 4.2-4.4 exit 
as POSIX specifies.


$ bash -o posix -c 'eval "(" || echo woops'
bash: eval: regel 1: syntaxfout: onverwacht bestandseinde
woops

On the other hand, note that 'command eval' should *not* exit. Bash 
4.2-4.4 exit on 'command eval "("' which is also a bug.


Thanks,

- M.

Re: 'eval' exit behaviour in posix mode

[Martijn Dekker]

Op 14-12-18 om 14:29 schreef Chet Ramey:

On 12/14/18 6:46 AM, Martijn Dekker wrote:

In current git, 'bash -o posix' no longer exits the shell on a syntax error
in the argument to the special builtin 'eval'. Bash 4.2-4.4 exit as POSIX
specifies.


Well, not quite. It's the specific case of an unexpected end of file. ksh93
doesn't exit, either, but it's probably the right thing to do, and the
right thing for backwards compatibility.


But now it exits on an unexpected EOF (or other syntax error) in 
'command eval' as well, which it shouldn't.


- M.

Re: Should [[ -v 1 ]] be supported?

[Martijn Dekker]

Op 27-12-18 om 19:22 schreef Chet Ramey:

On 12/26/18 10:49 PM, Peng Yu wrote:


Although [[ -z ${1+s} ]]  and (($#)) works for testing if $1 is set,
neither of them are uniformly better performance wise. In this case,
should [[ -v 1 ]] be supported?


So you're saying that neither of the existing options performs better
than the other, though they both perform well, so we should add some
new capability just because? That's a particularly poor argument.


Consistency might be a better argument. If [[ -v foo ]] is equivalent to 
[[ -n ${foo+s} ]] for variables (with the advantage that you don't need 
'eval' to handle arbitrary values of 'foo'), then perhaps it's not 
unreasonable to expect [[ -v 1 ]] to be equivalent to [[ -n ${1+s} ]].


FWIW, zsh and mksh do support this; ksh93 doesn't.

- M.

Re: Should [[ -v 1 ]] be supported?

[Martijn Dekker]

Op 28-12-18 om 01:39 schreef Peng Yu:

What I meant in my original email is that I want something for testing
if there is a command line argument (one or more, the exact number
does not matter). $# gives more than that info, because it tells not
only whether is any command line argument, but also how many. This
could lead to slower performance if the goal is to just test if there
is an argument.


I don't believe that at all. The number of positional parameters is kept 
anyway. It's not recalculated when you compare it to another number, so 
it's just as fast as a simple comparison of two integers.



[[ -z ${1+s} ]] does something also more than necessary too, because
it not only tests for whether $1 is set, it also replaced with a
string "s". This also does more than just testing whether $1 is set.


That's negligible.

And even if it weren't -- if performance is *that* important to you, 
you're using the wrong language altogether.


- M.

FIFO race condition on SunOS kernels

[Martijn Dekker]

You'd think that establishing a pipe between two processes is a very 
basic UNIX feature that should work reliably on all UNIX variants.


But the following script seems to break consistently on Solaris and 
variants (SunOS kernels) when executed by bash, ksh93, or dash. All it 
does is make 100 FIFOs and read a line from each -- it should be trivial.


And it does work fine on (recent versions of) Linux, macOS, and all the 
BSDs, on all shells.


#! /bin/sh
tmpdir=/tmp/FIFOs$$
trap "exec rm -rf $tmpdir" EXIT INT PIPE TERM
mkdir "$tmpdir" || exit
i=0; while test "$((i+=1))" -le 100; do
fifo=$tmpdir/FIFO$i
mkfifo "$fifo" || exit
echo "this is FIFO $i" >"$fifo" &
read foo <"$fifo" && echo "$foo"
done

Tested on Solaris 10.1, 11.3 and 11.4 and on OpenIndiana, all on 
VirtualBox. They all fail in identical ways.


ksh93 (which is /bin/sh on Solaris) doesn't cope with the script either 
but hangs slightly later. dash goes through all of them bust most fail 
with 'interrupted system call'. However, zsh, yash, and /usr/xpg4/bin/sh 
(ksh88) execute the script correctly on Solaris -- but about 20 times as 
slowly as on other OSs.


This makes me suspect the SunOS kernel must have some very bad race 
condition involving FIFOs, but some shells work around it.


Would it be possible to fix this on bash 5?

- M.

Re: FIFO race condition on SunOS kernels

[Martijn Dekker]

Op 01-01-19 om 23:47 schreef Vladimir Marek:
[...]

That said, I do use VirtualBox 5.1.22r115126 which is pretty old.


I recently upgraded to VirtualBox 6.0 on Mac OS X 10.11 and the results 
are identical.



Putting 0.5s delay anywhere in the loop makes the problem disappear.


Yes, I'd noticed as much. I am trying to get my portable shell library, 
modernish, to work on as many operating systems as possible. One of its 
recently added features involves creating and opening FIFOs on the fly, 
which breaks on Solaris on VirtualBox. My own experimentation told me 
that I need a full second to be safe.



Hmm, so we are waiting on a condition. That needs to be investigated
deeper. It would be great if you could open a case for this as we have
to prioritize our work ...


How would I go about it? I'm not an Oracle customer, nor could I justify 
the expense to become one. I use my test Solaris VMs only to test 
modernish, as the free-of-charge licence permits. I've previously been 
given to understand that only customers can report Solaris bugs. But I'd 
like to give some bug reports back if you give me a way to do it.



At the moment it looks like a bug in Solaris to me, but it shows only on
VirtualBox.


Thanks, that's good to have confirmed! It was hoping as much -- it would 
have been hard to believe that something this basic is broken on Solaris 
in general.


By the way, a side effect of developing modernish has been to put me on 
a crusade of sorts against shell bugs on all the shells, as I keep 
running into them. The "standards compliant" /usr/xpg4/bin/sh (ksh88) on 
Solaris has a LOT of bugs, including some really serious ones that I 
wrote about on the Austin Group list last year. Where/how could I reach 
a person who is interested in receiving bug reports?


Thanks,

- M.

Re: Sourcing a script from a pipe is not reliable

[Martijn Dekker]

Op 10-01-19 om 22:52 schreef Jeremy:
> We are trying to determine if the current shell supports passing positional
> arguments to a script sourced by dot (.), and we are trying to do it in a
> way that will work under pretty much any shell.  If you can show me how to
> do that with eval, then that would be great, even though in general I hate
> to ship a script that uses eval.

While not all shells support passing positional parameters to dot
scripts, it is very simple to make work for POSIX sh: shell functions
provide positional parameters, so wrap the dot command in one of those.

dot() {
_myscript=$1
shift
. "$_myscript"
}
# use like:
dot ./somescript.sh one two three

"Sourcing" a script read from standard input without relying on the
non-standard /dev/stdin is also very simple:

eval "$(cat)"

This does load the entire script in memory before beginning execution,
but I can't imagine that being a problem in 2019, unless the script is
generated by some infinite loop.

To get your local positional parameters for a script read from standard
input, you can combine both techniques:

dotstdin() {
eval "$(cat)"
}

The following then works as expected:

dotstdin one two three <<-'EOF'
printf "ok: %s\\n" "$@"
EOF

Note that double-quoting the "$(cat)" command substitution is essential
to avoid globally active split and glob; among other things, this makes
multiple-line scripts work.

This is all perfectly POSIX and should also work fine on bash 3.2.

Hope this helps.

- M.

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Martijn Dekker]

Op 16-01-19 om 02:21 schreef Quentin:
> If you really need some quality CSPRNG values, I'd suggest adding a
> $SECURE_RANDOM variable that just reads from /dev/urandom.

IMHO, this would clearly be the correct approach. I don't know of any
21st century Unix or Unix-like system that doesn't have /dev/urandom. I
would really like to see shells adopt this idea -- hopefully all with
the same variable name.

- M.

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Martijn Dekker]

Op 19-01-19 om 23:10 schreef Chet Ramey:
> On 1/19/19 2:45 PM, Martijn Dekker wrote:
>> Op 16-01-19 om 02:21 schreef Quentin:
>>> If you really need some quality CSPRNG values, I'd suggest adding a
>>> $SECURE_RANDOM variable that just reads from /dev/urandom.
>>
>> IMHO, this would clearly be the correct approach. I don't know of any
>> 21st century Unix or Unix-like system that doesn't have /dev/urandom. I
>> would really like to see shells adopt this idea -- hopefully all with
>> the same variable name.
> 
> OK, this is a reasonable approach. Since /dev/urandom just generates
> random bytes, there's a lot of flexibility and we're not subject to
> any kind of backwards compatibility constraints, especially not the
> 16-bit limit. What do you think would be the best way to present that
> to a user? As a 32-bit random number? A character string you can use to
> create filenames? Some other form?

I'd say numbers would be the most useful, as these are the easiest to
convert into anything else using shell arithmetic and parameter
expansions. E.g. to create a random character string for a temporary
file name, you could do

filename_suffix() {
chars=ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789
length=${#chars}
for ((i=0; i<10; i++)) do
printf '%s' "${chars:$(( SECURE_RANDOM % length + 1 )):1}"
done
}
tmpfile=/tmp/myfile.$(filename_suffix)

(which would of course already work with RANDOM but that would be
totally insecure, as in not effectively eliminating the risk of collisions).

- Martijn

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Martijn Dekker]

Op 21-01-19 om 20:12 schreef Chet Ramey:
> On 1/20/19 9:04 PM, Rawiri Blundell wrote:
>> For what it's worth I did consider suggesting URANDOM, however I
>> figured some users may confuse it like this:
>>
>> RANDOM -> /dev/random
>> URANDOM -> /dev/urandom
>>
>> Couple that with an established base of myths about /dev/urandom, I
>> thought it might be best to suggest something else to eliminate that
>> potential confusion.
> 
> I can see that, but I think RANDOM is established enough that nobody
> assumes it has anything to do with /dev/random.

Not every shell scripter has years of experience. If you pair a RANDOM
with a URANDOM in the shell, then I do think many people will
automatically associate these with /dev/random and /dev/urandom.

Also, I think the name should describe the functionality, not the
specific way it's obtained -- because that could change at some point in
the future, and/or become system-dependent.

So I think SRANDOM is the best name (or SECURE_RANDOM, though that is a
bit long).

> If we're converging on something like URANDOM (or some other name) for a
> better RNG, I don't see the need to change the RANDOM generator.

FWIW, I agree.

- M.

Re: bash-5.0: problem with variable scoping in posix-mode

[Martijn Dekker]

Op 27-01-19 om 22:59 schreef Chet Ramey:
> This is a consequence of a combination of two POSIX features. First, POSIX
> requires assignment statements preceding special builtins to create global
> variables (POSIX has no local variables) that persist in the shell context
> after the special builtin completes.

Exactly because POSIX has no notion of local variables (but bash does),
I don't think there is anything that requires you to ignore/override
bash local variables in this context. Any fully POSIX-compliant script
will not use local variables, so is not affected either way. I think
that leaves you free to do the logical thing and keep the scope local as
defined by the user.

>  Second, POSIX requires* that
> assignment statements preceding function calls have the same variable-
> assignment behavior as special builtins.

When combining POSIX mode with a bash local variable scope, I would take
that to mean the assignment should persist for as long as the local
scope does.

[...]
> (*) The most recent version of the standard has removed this requirement
> for shell functions, and I will change that behavior for the next release
> of bash.

Thank you for that. dash and yash recently got rid of that behaviour, I
hope other shell authors will also follow suit...

- M.

Re: Regression in --enable-minimal-config

[Martijn Dekker]

Op 11-05-19 om 13:08 schreef Miro Kropáček:

It seems --enable-minimal-config isn't tested very often as both
4.3.48 and 4.2.x required a few #ifdef's to be added to make it
compile (Linux).


I think you're probably hitting this bug:

https://lists.gnu.org/archive/html/bug-bash/2016-09/msg00083.html

The exit status of subshells is broken in bash-4.3 and bash-4.4 with 
--enable-minimal-config. This is a fatal bug that would easily break 
most substantial shell scripts, including configure.


The bug was fixed in bash-5.0.

As for a minimal bash-5.0 with full POSIX compliance, aliases are 
required by POSIX but turned off by the minimal config, so you want: 
--enable-minimal-config --enable-alias --enable-strict-posix-default


I now often test modernish (my shell language enhancement library) 
against that shell configuration (among many others), so 
--enable-minimal-config is now a little better tested than before.


--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: Regression in --enable-minimal-config

[Martijn Dekker]

Op 12-05-19 om 13:40 schreef Miro Kropáček:

On Sun, 12 May 2019 at 12:47, Martijn Dekker  wrote:

I think you're probably hitting this bug:

https://lists.gnu.org/archive/html/bug-bash/2016-09/msg00083.html

Thanks, that indeed looks like it.


The exit status of subshells is broken in bash-4.3 and bash-4.4 with
--enable-minimal-config. This is a fatal bug that would easily break
most substantial shell scripts, including configure.

The bug was fixed in bash-5.0.

Yes, 5.0 works. Do you know where the fix itself is?


No, but Chet would know.

Chet only commits snapshots to git and not individual fixes, so it's 
hard to isolate any one fix from the git repo.



 Commit
http://git.savannah.gnu.org/cgit/bash.git/commit/?id=61c476d20d32dfd389c79fd4f2161a780685e42e
mentions your report and fix but it definitely does not fix it.


Are you sure? I just did a test compile on that same commit with 
'--enable-minimal-config --enable-strict-posix-default --enable-alias' 
and the bug is definitely not there. Then when I back up one commit and 
run 'make' again, the bug appears.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: built-in printf %f parameter format depend on LC_NUMERIC

[Martijn Dekker]

Op 12-07-19 om 21:46 schreef Dennis Clarke:

Well the man page for XPG6 bc in Solaris 10 claims :

ENVIRONMENT VARIABLES
  See environ(5) for descriptions of the following environment
  variables  that  affect  the  execution of bc: LANG, LC_ALL,
  LC_CTYPE, LC_MESSAGES, and NLSPATH.

Really?


Note the absence of LC_NUMERIC in that paragraph.

- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: x[

[Martijn Dekker]

Op 29-07-19 om 19:09 schreef Eli Schwartz:

The initial workaround discovered, was to use

$ function _[ () { echo hello; }; <() _[
hello

The use of <() somehow suppresses the glitch in the same way that
quoting it does. If it were just glob expansion, then why should that be so?


As others pointed out, it's the start of an array assignment, and 
associative array indexes can contain newlines.


So we can de-obfuscate the issue by looking at a simple variable 
assignment 'x=foo' instead. Consider:


$ function x=foo { echo hello; }; <() x=foo
hello

However:

$ function x=foo { echo hello; }; In that case, 'x=foo' is interpreted as an assignment. Which is correct, 
because redirections can occur in any word of a simple command, 
including before and between assignments and arguments. They are 
performed and removed before further parsing and executing the command.


<(Process substitution) may look superficially like a fancy form of 
redirection[*], but it's actually more akin to command substitution: the 
command inside is run, and the substitution is replaced, not with that 
command's output, but with a file name from which to read that command's 
output (or, in the case of >(process substitution), a file name to which 
to write that command's input).


Because that command is empty in this instance, bash does not bother to 
substitute a file name, and the <() is substituted by nothing.


So, let's get rid of the function (because it is a distraction) and just 
see how 'x=foo' is parsed in substitutions and redirections:


$ <() x=foo
bash: x=foo: command not found
$ $() x=foo
bash: x=foo: command not found
$ `` x=foo
bash: x=foo: command not found
$ Here it is also important to understand that a "simple command" consists 
of either assignments, or command words, or both:


x=foo y=bar some command here

So, both process substitution and command substitution (both forms) 
signal to the parser that the preceding assignments in a simple command 
have ended, and we're now on to command words and thus trying to execute 
a command called 'x=foo'. This is also true on simple POSIX shells like 
dash (for command substitution -- they don't have process substitution). 
The fact that the substitutions are empty, and thus substituted by 
nothing, does not stop them from influencing the parser.


The same is not true for redirections, which are specified by POSIX to 
be parsed and removed at the earliest stage, before even distinguishing 
between assignments and command words.


- M.

[*] Pointless lament: It's unfortunate that process substitution starts 
with '<' or '>' and not with $ like every other kind of substitution and 
expansion (except obsolete `command substitutions`). It confuses people 
into thinking of it in terms of redirection, which is very misleading.


By the way, in modernish, I've re-implemented process substitution in a 
form usable by all POSIX shells (including simple ones like dash), as a 
form of command substitution:

$( % your command here )# like <(your command here)
$( % -o your command here ) # like >(your command here)

--
modernish -- harness the shell
https://github.com/modernish/modernish

[PATCH] allow process substitution in POSIX mode

[Martijn Dekker]

Process substitution is disabled in POSIX mode. A 'git blame' on parse.y 
suggests that this has been the case since bash prehistory.


To me, this seems:

- Inconsistent. Other bash extensions are not disabled when POSIX mode
  is active; as far as I can tell, this is the only one.

- Unnecessary. The <(...) and >(...) syntax is a straightforward syntax
  error in POSIX, so it's not as if this particular extension somehow
  conflicts with POSIX standard scripts.

- Annoying. When a bash-specific script uses functions written for POSIX
  (such as the modernish library), it needs to run in POSIX mode for
  compatibility with those, but can still use every bash extension
  except this one. It just seems pointless.

Would you consider the attached patch, which removes this limitation?

Thanks,

- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish
diff --git a/POSIX b/POSIX
index f3f4a0b7..d2836070 100644
--- a/POSIX
+++ b/POSIX
@@ -115,120 +115,118 @@ The following list is what's changed when 'POSIX mode' 
is in effect:
  read with the '.' or 'source' builtins, or in a string processed by
  the 'eval' builtin.
 
-  30. Process substitution is not available.
-
-  31. While variable indirection is available, it may not be applied to
+  30. While variable indirection is available, it may not be applied to
  the '#' and '?' special parameters.
 
-  32. When expanding the '*' special parameter in a pattern context
+  31. When expanding the '*' special parameter in a pattern context
  where the expansion is double-quoted does not treat the '$*' as if
  it were double-quoted.
 
-  33. Assignment statements preceding POSIX special builtins persist in
+  32. Assignment statements preceding POSIX special builtins persist in
  the shell environment after the builtin completes.
 
-  34. Assignment statements preceding shell function calls persist in
+  33. Assignment statements preceding shell function calls persist in
  the shell environment after the function returns, as if a POSIX
  special builtin command had been executed.
 
-  35. The 'command' builtin does not prevent builtins that take
+  34. The 'command' builtin does not prevent builtins that take
  assignment statements as arguments from expanding them as
  assignment statements; when not in POSIX mode, assignment builtins
  lose their assignment statement expansion properties when preceded
  by 'command'.
 
-  36. The 'bg' builtin uses the required format to describe each job
+  35. The 'bg' builtin uses the required format to describe each job
  placed in the background, which does not include an indication of
  whether the job is the current or previous job.
 
-  37. The output of 'kill -l' prints all the signal names on a single
+  36. The output of 'kill -l' prints all the signal names on a single
  line, separated by spaces, without the 'SIG' prefix.
 
-  38. The 'kill' builtin does not accept signal names with a 'SIG'
+  37. The 'kill' builtin does not accept signal names with a 'SIG'
  prefix.
 
-  39. The 'export' and 'readonly' builtin commands display their output
+  38. The 'export' and 'readonly' builtin commands display their output
  in the format required by POSIX.
 
-  40. The 'trap' builtin displays signal names without the leading
+  39. The 'trap' builtin displays signal names without the leading
  'SIG'.
 
-  41. The 'trap' builtin doesn't check the first argument for a possible
+  40. The 'trap' builtin doesn't check the first argument for a possible
  signal specification and revert the signal handling to the original
  disposition if it is, unless that argument consists solely of
  digits and is a valid signal number.  If users want to reset the
  handler for a given signal to the original disposition, they should
  use '-' as the first argument.
 
-  42. The '.' and 'source' builtins do not search the current directory
+  41. The '.' and 'source' builtins do not search the current directory
  for the filename argument if it is not found by searching 'PATH'.
 
-  43. Enabling POSIX mode has the effect of setting the
+  42. Enabling POSIX mode has the effect of setting the
  'inherit_errexit' option, so subshells spawned to execute command
  substitutions inherit the value of the '-e' option from the parent
  shell.  When the 'inherit_errexit' option is not enabled, Bash
  clears the '-e' option in such subshells.
 
-  44. Enabling POSIX mode has the effect of setting the 'shift_verbose'
+  43. Enabling POSIX mode has the effect of setting the 'shift_verbose'
  option, so numeric arguments to 'shift' that exceed the number of
  positional parameters will result in an error message.
 
-  45. When the 'alias' builtin displays alias definitions, it does not
+  44. When the 'alias' builtin displays alias definitions, it does not
  display them with a leading 'alias ' unless the '-p' option is
 

Re: [PATCH] allow process substitution in POSIX mode

[Martijn Dekker]

Op 08-12-19 om 10:29 schreef Oğuz:

I think otherwise. In POSIX mode Bash should at least print warnings
about features POSIX lets implementations provide but doesn't mandate,
like function, select, [[, etc.


That would be a far bigger change than just allowing process substitution.

Also, there is already a good shell that does this: yash 
. Its POSIX mode is deliberately strict and 
restrictive and does not allow any non-portable functionality, so it's a 
really good compatibility testing tool.


Then there are also shells that simply don't implement much more than 
POSIX, such as dash. They're useful for that as well.


So I don't think bash needs to fulfill that function. Bash's POSIX mode 
has always been simply a way to run POSIX scripts, with or without 
mixing them with bash scripts.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

Spurious recursive trap invocation warning

[Martijn Dekker]

The current git version of bash (at least on x86_64-apple-darwin18.7.0) 
gives a spurious warning about a recursive trap invocation when running 
the following more than once on an interactive shell. The trap unsets 
itself before resending the signal, so there should be no recursive 
invocation.


bash-5.0$ echo $BASH_VERSION
5.0.11(1)-maint
bash-5.0$ trap 'echo int; trap - INT; kill -s INT $$' INT; kill -s INT $$
int

bash-5.0$ trap 'echo int; trap - INT; kill -s INT $$' INT; kill -s INT $$
bash: warning: run_pending_traps: recursive invocation while running 
trap for signal 2

int

bash-5.0$ trap 'echo int; trap - INT; kill -s INT $$' INT; kill -s INT $$
bash: warning: run_pending_traps: recursive invocation while running 
trap for signal 2

int

Thanks and happy new year,

- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

trap: interactive bash exits on resending SIGINT twice

[Martijn Dekker]

Setting the following trap makes an interactive bash (any version) exit 
on issuing SIGINT:


trap 'trap "true" INT; kill -s INT $$; trap - INT; kill -s INT $$' INT
# ^^^ eat signal   ^^ unset trap

After setting that trap, pressing Ctrl+C, or otherwise causing SIGINT to 
be sent, causes an interactive bash to exit.


Expected behaviour: no exit.
Confirmed on macOS and Debian.

Ignoring the signal instead of merely eating it seems to be an effective 
workaround on more recent bash versions:

trap 'trap "" INT; kill -s INT $$; trap - INT; kill -s INT $$' INT
This avoids an interactive shell exiting on SIGINT on bash 4.4 and 5.0, 
but not 4.3 or earlier.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

temp setting POSIXLY_CORRECT turns alias expansion off

[Martijn Dekker]

When alias expansion is enabled for a script in bash native mode, 
prefixing POSIXLY_CORRECT=y to any command will turn alias expansion 
globally off. This is a bug because the assignment should only have 
effect on that command.


$ bash -c 'shopt -s expand_aliases; shopt|grep expand_a;
  POSIXLY_CORRECT=y true; shopt|grep expand_a'
expand_aliases  on
expand_aliases  off

Expected behaviour: alias expansion should stay on.

I've confirmed this bug on bash 4.2 and later; bash 4.1 and earlier seem 
fine.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: temp setting POSIXLY_CORRECT turns alias expansion off

[Martijn Dekker]

Op 16-01-20 om 17:02 schreef Chet Ramey:

On 1/15/20 10:24 PM, Martijn Dekker wrote:
When alias expansion is enabled for a script in bash native mode, 
prefixing POSIXLY_CORRECT=y to any command will turn alias expansion 
globally off. This is a bug because the assignment should only have 
effect on that command.


You're probably right, but it's an interesting question.

The idea is that

POSIXLY_CORRECT=y true

is essentially equivalent (assuming POSIXLY_CORRECT is not already set) to

POSIXLY_CORRECT=y ; true; unset POSIXLY_CORRECT

and turning off posix mode resets a default environment.


I think it *should* be essentially equivalent to
(POSIXLY_CORRECT=y; true)
minus the actual forked subshell of course. But the state after the 
command should be identical to the state before.


Another odd behaviour: 'unset POSIXLY_CORRECT' resets a default 
environment even if the variable was already unset. This is in all 
versions of bash.


If you'll forgive my frankness, I think the whole notion of coupling a 
shell variable (POSIXLY_CORRECT) to a shell option (-o posix) is broken, 
because:


  1. Shell options and variables are fundamentally different things,
 so implementing one as another causes behaviour that violates the
 principle of least astonishment.

  2. It's asking for corner case bugs, as there is plenty of weirdness
 associated with shell assignments, particularly when they precede
 commands.

  3. It makes it impossible to 'export POSIXLY_CORRECT=y' to use all
 external commands in POSIX mode while using bash in native mode
 -- something that would entirely make sense when writing bash
 scripts that should run on multiple different operating systems.

(Also, 4. this kills process substitution. You never responded to my 
patch last month, did you forget about it?)


So I think bash should only check at init time if POSIXLY_CORRECT exists 
in the environment and set -o posix if so (the same way it checks if it 
was invoked as sh).


Other than that, I think -o posix and POSIXLY_CORRECT should be 
decoupled completely and POSIXLY_CORRECT should just be a variable like 
any other.


Scripts should then simply set -o/+o posix in order to enable/disable 
POSIX mode for the shell where desired. I would bet that most already do 
that anyway, because that is how other shells work.


This might be a breaking change, but I think the incompatibility would 
be minor, and worth the improvement in consistent behaviour.


- Martijn

--
modernish -- harness the shell
https://github.com/modernish/modernish

[PATCH] efficient [[ -v 1 ]] [was: Should [[ -v 1 ]] be supported?]

[Martijn Dekker]

Op 29-12-18 om 01:19 schreef Chet Ramey:

On 12/27/18 3:11 PM, Martijn Dekker wrote:


Consistency might be a better argument. If [[ -v foo ]] is equivalent to [[
-n ${foo+s} ]] for variables (with the advantage that you don't need 'eval'
to handle arbitrary values of 'foo'), then perhaps it's not unreasonable to
expect [[ -v 1 ]] to be equivalent to [[ -n ${1+s} ]].


The completeness argument is more rigorous, and there's a case to add this
in a future version of bash. He didn't make that argument, though.


I noticed some deactivated code tagged bash-5.1 with my name on it. Cool. :)

Isn't the way below much more efficient, though? There's no need to 
retrieve and discard the value of a positional parameter -- we can just 
compare the number to $#, i.e. number_of_args(). Any number between 0 
and $#, inclusive, represents a set positional parameter by definition.


- M.

diff --git a/test.c b/test.c
index 9cbc0436..27efd9f7 100644
--- a/test.c
+++ b/test.c
@@ -640,12 +640,7 @@ unary_test (op, arg)
 #if 0  /* TAG:bash-5.1 from Martijn Dekker */
   else if (legal_number (arg, &r)) /* -v n == is $n set? */
{
- char *t;
- int ret;
- t = get_dollar_var_value (r);
- ret = t ? TRUE : FALSE;
- free (t);
- return ret;
+ return (r >= 0 && r <= number_of_args());
}
 #endif
   v = find_variable (arg);


--
modernish -- harness the shell
https://github.com/modernish/modernish

Bizarre interaction bug involving bash w/ lastpipe + Almquist 'wait'

[Martijn Dekker]

This is probably the strangest bug (or maybe pair of bugs) I've run into 
in nearly five years of breaking shells by developing modernish.


I've traced it to an interaction between bash >= 4.2 (i.e.: bash with 
shopt -s lastpipe) and variants of the Almquist shell, at least: dash, 
gwsh, Busybox ash, FreeBSD sh, and NetBSD 9.0rc2 sh.


Symptom: if 'return' is invoked on bash in the last element of a pipe 
executed in the main shell environment, then if you subsequently 'exec' 
an Almquist shell variant so that it has the same PID, its 'wait' 
builtin breaks.


I can consistently reproduce this on Linux, macOS, FreeBSD, NetBSD 
9.0rc2, OpenBSD, and Solaris.


To reproduce this, you need bash >= 4.2, some Almquist shell variant, 
and these two test scripts:


---begin test.bash---
fn() {
: | return
}
shopt -s lastpipe || exit
fn
exec "${1:-dash}" test.ash
---end test.bash---

---begin test.ash---
echo '*ash-begin'
: &
echo '*ash-middle'
wait "$!"
echo '*ash-end'
---end test.ash---

When executing test.bash with dash, gwsh, Busybox ash, or FreeBSD sh, 
then test.ash simply waits forever on executing 'wait "$!"'.


$ bash test.bash 
*ash-begin
*ash-middle
(nothing until ^C)

NetBSD sh behaves differently. NetBSD 8.1 sh (as installed on sdf.org 
and sdf-eu.org) seem to act completely normally, but NetBSD 9.0rc2 sh 
(on my VirtualBox test VM) segfaults. Output on NetBSD 9.0rc2:


$ bash test.bash /bin/sh
*ash-begin
*ash-middle
[1]   Segmentation fault   bash test.bash sh

I don't know if the different NetBSD sh behaviour is because the older 
NetBSD sh doesn't have the bug, or because some factor on the sdf*.org 
systems causes it to not be triggered.


To me, this smells like the use of some uninitialised value on various 
Almquist shells. Tracing that is beyond my expertise though.


Whether this also represents a bug in bash or not, I can't say. But no 
other shells trigger this that I've found, not even ksh93 and zsh which 
also execute the last element of a pipe in the main shell environment.


- Martijn

--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: AW: Bizarre interaction bug involving bash w/ lastpipe + Almquist 'wait'

[Martijn Dekker]

Op 07-02-20 om 12:19 schreef Walter Harms:

IMHO is the bug on bash side. ash can assume to get an "healthy"
environment from the caller. You can simply not fix everything that
can possible go wrong.


That is a rather fallacious argument. Of course you cannot fix 
*everything* that could possibly go wrong. You can certainly fix *this* 
thing, though. I know, because every non-Almquist shell does it.


These days, no program can realistically assume a "healthy" environment. 
Computers have become unimaginably complex machines, built on thousands 
of interdependent abstraction layers, each as fallible as the humans 
that designed and implemented them. So "unhealthy" environments happen 
all the time, due to all sorts of unforeseen causes.


It's well past time to accept that the 1980s are behind us. In 2020, 
systems have to be programmed robustly and defensively.



Obviously it should not segfault but so far i understand it is bsd as
that does, not busybox ash.


True. But instead, it simply gets stuck forever, with no message or 
other indicator of what went wrong. How is that better?


(Going slightly off-topic below...)

Segfaulting is actually a good thing: it's one form of failing reliably. 
And failing reliably is vastly better than what often happens instead, 
especially in shell scripts: subtle breakage, which can take a lot of 
detective work to trace, and in some cases can cause serious damage due 
to the program functioning inconsistently and incorrectly (instead of 
not at all).


Failing reliably is something the shell is ATROCIOUSLY bad at, and it's 
one of the first things modernish aims to fix.


- Martijn

--
modernish -- harness the shell
https://github.com/modernish/modernish

'=~' crash on NetBSD

[Martijn Dekker]

On NetBSD, bash (all versions, including current git) dumps core when 
using [[ to match against an ERE containing certain UTF-8 characters, 
for example, É ($'\303\211').


| $ bin/bash-5.0-debug -c "[[ c =~ $'\303\211' ]]"
|
| malloc: unknown:0: assertion botched
| malloc: 0x81ba08: allocated: last allocated from unknown:0
| free: start and end chunk sizes differ
| Aborting...Abort trap (core dumped)

If it helps, here's a backtrace generated from the core file by 'gdb'. 
It seems to suggest that only the first byte of the UTF-8 character is 
included in the pattern...


#0  0x7ae3a22fe48a in _lwp_kill () from /usr/lib/libc.so.12
#1  0x7ae3a22fdd32 in abort () from /usr/lib/libc.so.12
#2  0x0043492e in programming_error (
format=0x5213d8 "free: start and end chunk sizes differ")
at error.c:175
#3  0x004f3e78 in xbotch (mem=0x81ba08, e=8,
s=0x5213d8 "free: start and end chunk sizes differ", file=0x0, line=0)
at malloc.c:358
#4  0x004f4e40 in internal_free (mem=0x81ba08, file=0x0, line=0,
flags=0) at malloc.c:964
#5  0x004f5a1e in free (mem=0x81ba08) at malloc.c:1392
#6  0x004b3c02 in sh_regmatch (string=0x806358 "c",
pattern=0x819d28 "\303", flags=3) at shmatch.c:118
#7  0x00424c28 in execute_cond_node (cond=0x819a48)
at execute_cmd.c:3884
#8  0x00424e3f in execute_cond_command (cond_command=0x819a48)
at execute_cmd.c:3959
#9  0x0041fb34 in execute_command_internal (command=0x819a88,
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x8199e8)
at execute_cmd.c:1029
#10 0x00492b38 in parse_and_execute (
string=0x805188 "[[ c =~ \303 ]]", from_file=0x4fe190 "-c", flags=4)
at evalstring.c:460
#11 0x00406452 in run_one_command (
command=0x7f7fffb6ca46 "[[ c =~ \303 ]]") at shell.c:1436
#12 0x004055f8 in main (argc=3, argv=0x7f7fffb6c468,
env=0x7f7fffb6c488) at shell.c:738


--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: '=~' crash on NetBSD

[Martijn Dekker]

Op 07-05-20 om 14:20 schreef Chet Ramey:

On 5/6/20 2:08 PM, Martijn Dekker wrote:

On NetBSD, bash (all versions, including current git) dumps core when using
[[ to match against an ERE containing certain UTF-8 characters, for
example, É ($'\303\211').

| $ bin/bash-5.0-debug -c "[[ c =~ $'\303\211' ]]"
|
| malloc: unknown:0: assertion botched
| malloc: 0x81ba08: allocated: last allocated from unknown:0
| free: start and end chunk sizes differ
| Aborting...Abort trap (core dumped)


What version of NetBSD? I don't run NetBSD, and I can't reproduce this on
other platforms, so I'll have to go searching for something.


Sorry about that omission. I've tested NetBSD 8.0, 8.1 and 9.0 (all 
amd64) and the bug occurs on all of these.


If it helps, sdf.org will give you a free NetBSD 8.1 user account with 
access to gcc.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

[PATCH] fc: trim range instead of erroring out

[Martijn Dekker]

As I use/try/test many shells, I'm in the habit of using POSIX commands 
such as 'fc -l 1' to list the complete history.


If there have been more than $HISTSIZE command, the list is trimmed at 
the beginning without renumbering, and bash errors out:


$ fc -l 1
bash-5.0: fc: history specification out of range

This is different from every other shell, and also looks like it's 
contrary to the POSIX spec:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/fc.html#tag_20_44_05
| When a range of commands is used, it shall not be an error to specify
| first or last values that are not in the history list; fc shall
| substitute the value representing the oldest or newest command in the
| list, as appropriate. For example, if there are only ten commands in
| the history list, numbered 1 to 10:
|
| fc -l
| fc 1 99
|
| shall list and edit, respectively, all ten commands.

The attached patch removes the error and limits the range to available 
commands as specified.


- Martijn

--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh
diff --git a/builtins/fc.def b/builtins/fc.def
index c266ac2c..7f606c52 100644
--- a/builtins/fc.def
+++ b/builtins/fc.def
@@ -321,10 +321,7 @@ fc_builtin (list)
 while (last_hist >= 0 && hlist[last_hist] == 0)
   last_hist--;
   if (last_hist < 0)
-{
-  sh_erange ((char *)NULL, _("history specification"));
-  return (EXECUTION_FAILURE);
-}
+last_hist = 0;
 
   if (list)
 {
@@ -353,12 +350,10 @@ fc_builtin (list)
histbeg = histend = last_hist;
 }
 
-  /* We print error messages for line specifications out of range. */
-  if ((histbeg < 0) || (histend < 0))
-{
-  sh_erange ((char *)NULL, _("history specification"));
-  return (EXECUTION_FAILURE);
-}
+  if (histbeg < 0)
+histbeg = 0;
+  if (histend < 0)
+histend = 0;
 
   /* "When not listing, the fc command that caused the editing shall not be
  entered into the history list." */
@@ -382,12 +377,10 @@ fc_builtin (list)
histbeg = last_hist;
 }
 
-  /* We print error messages for line specifications out of range. */
-  if ((histbeg < 0) || (histend < 0))
-{
-  sh_erange ((char *)NULL, _("history specification"));
-  return (EXECUTION_FAILURE);
-}
+  if (histbeg < 0)
+histbeg = 0;
+  if (histend < 0)
+histend = 0;
 
   if (histend < histbeg)
 {

Re: [PATCH] fc: trim range instead of erroring out

[Martijn Dekker]

Op 12-08-20 om 16:40 schreef Chet Ramey:

What's your opinion about what the `as appropriate' means? An out-of-range
`first' gets substituted with the first command in the history, and an out-
of-range `last' gets the last history entry?


Yes, that was my intention.


 Bash does one thing, your
patch does another, but neither one does that.


Then I suppose my patch was wrong. It seemed to work for me.

- Martijn

--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: How to use PROMPT_COMMAND(S) without breaking other scripts

[Martijn Dekker]

Op 24-08-20 om 15:57 schreef Chet Ramey:

I sometimes think I should have stuck with converting PROMPT_COMMAND to
an array. Either way, there's going to be a transition, and maybe that
would have been the easiest.


Is it too late? I think that would actually be cleaner than adding a 
separate array, per Koichi's report.


One problem is that if a script does the obvious

PROMPT_COMMAND+=("some command here")

if PROMPT_COMMAND is not yet set, then it starts adding at array index 
0, so a subsequent traditional usage from some other script


PROMPT_COMMAND="some command here"

would overwrite it. So array usage should not use the 0 index. To avoid 
using the 0 index, one possibility is:


PROMPT_COMMAND[$(( ${#PROMPT_COMMAND[@]} + 1 ))]="some command here"

which, if PROMPT_COMMAND is unset, starts adding at index 1, not 0, and 
otherwise acts identically. However, 'set -u'/'set -o nounset' kills 
that. That option makes the ${#PROMPT_COMMAND[@]} expansion error out if 
there are no array elements (even though ${#@} works with no positional 
parameters). It's also an unwieldy command. So maybe that idea is not 
the best.


Another way to avoid using the 0 index, which is 'set -u' compatible, 
would be


PROMPT_COMMAND=${PROMPT_COMMAND-}
PROMPT_COMMAND+=("some command here")

The first command sets PROMPT_COMMAND[0] to the empty value if it 
doesn't exist yet, and otherwise leaves it unchanged. It's a bit of an 
ugly hack though.


But then, maybe it's best if bash itself just sets PROMPT_COMMAND[0] to 
the empty value on initialisation. IMO that would be a reasonably clean 
and reliable way to ensure a smooth transition.


Just my 2¢,

- M.

--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: How to use PROMPT_COMMAND(S) without breaking other scripts

[Martijn Dekker]

Op 24-08-20 om 20:37 schreef Chet Ramey:

On 8/24/20 12:58 PM, Martijn Dekker wrote:

Op 24-08-20 om 15:57 schreef Chet Ramey:

I sometimes think I should have stuck with converting PROMPT_COMMAND to
an array. Either way, there's going to be a transition, and maybe that
would have been the easiest.


Is it too late? I think that would actually be cleaner than adding a
separate array, per Koichi's report.


We're not "adding" anything. Bash just looks for this variable and reacts
if it finds it. It's not a special variable, and bash doesn't create it if
it doesn't exist.



So now you're adding code that looks for an array. Which is something 
other than nothing.


Granted, my language wasn't very precise, but I'm fairly confident that 
my intent was understandable.



What I mean is looking for PROMPT_COMMAND as happens now, and reacting a
different way if it's an array variable. That would resolve the existing
assignment issues, but open up the separate issues you describe.



But I also suggested a way of avoiding those issues: initialise 
$PROMPT_COMMAND a.k.a. ${PROMPT_COMMAND[0]} as empty.


- M.

--
||  modernish -- harness the shell
||  https://github.com/modernish/modernish
||
||  KornShell lives!
||  https://github.com/ksh93/ksh

Re: read Built-in Parameter Behavior -- Null Byte Delimiter

[Martijn Dekker]

Adam Danischewski schreef op 16-01-16 om 20:28:
> Yet if we look for the null byte:

bash, like most UNIX shells, cannot store or handle null bytes. (zsh is
the only exception I know of.)

> $> while IFS= read -r -d'' a; do echo "got $a"; done < <(find . -type f
> -print0)
>  * returns nothing *

This is because of how shell quoting works. The "read" command doesn't
even enter into it. It's all happening before "read" is even executed.

'' does not represent a null byte; it's an empty, as in zero bytes,
surrounded by single quotes.

Before passing the arguments to the command (such as 'read'), the shell
performs quote removal. This removes the quotes from the empty.

That leaves the 'read' command with no way to distinguish between

read -r -d'' a

and

read -r -d a

Meaning, you were actually executing "read" with "a" as the delimiter,
and no variable, so your "echo" command naturally returned nothing.
Meanwhile "read" would have stored whatever it got in the variable REPLY
by default.

Separating the quoted empty from the -d makes it possible to distinguish
it as an empty argument (which is still different from a null byte).

Hope this helps.

- M.

[PATCH] admit 'typeset' is here to stay

[Martijn Dekker]

In 'help typeset', the 'typeset' builtin is called obsolete and has been
so since at least bash 2.05b (2002) or possibly earlier. Perhaps it's
time to just call it a synonym, as indeed the texinfo documentation does.

diff -ur bash-4.4-beta.orig/builtins/declare.def
bash-4.4-beta/builtins/declare.def
--- bash-4.4-beta.orig/builtins/declare.def 2015-08-24
19:02:28.0 +0100
+++ bash-4.4-beta/builtins/declare.def  2016-01-28 22:32:16.0 +
@@ -65,7 +65,7 @@
 $SHORT_DOC typeset [-aAfFgilnrtux] [-p] name[=value] ...
 Set variable values and attributes.

-Obsolete.  See `help declare'.
+Synonym of `declare'.  See `help declare'.
 $END

 #include 

bash can't distinguish between empty and unset arrays

[Martijn Dekker]

bash treats an empty array as if it were an unset variable, which seems
very illogical as empty is quite distinct from unset:

$ myarray=()
$ [[ -v myarray ]] && echo set || echo unset
unset
$ set | grep ^myarray=  # yet, it's set:
myarray=()
$ set -u
$ for i in "${x[@]}"; do :; done
bash: x[@]: unbound variable

Note also that the "unbound variable" error is inconsistent with the
behaviour of "$@"; I would have thought that, logically, "$@" and
"${x[@]}" should behave the same way, since arrays are implemented as a
logical extension of the positional parameters concept.

zsh and ksh93 can distinguish between empty and unset arrays.

Thanks,

- Martijn

Re: Q: what is a fast way to see if an 'item' is present in an array?

[Martijn Dekker]

Linda Walsh schreef op 16-02-16 om 04:59:
> w/the slow func  being killed by a $() sub process, likely:

Yes, subshells are fatal for performance, particularly on bash.

> my fn2='() {my t="${2:?}[*]"
>my arRE="^($(IFS="|"; echo "${!t}"))$"
>[[ $1 =~ $arRE ]]
> }
> '

(What's "my"? An alias for "local"?)

Try this:

appears_in() {
local IFS="|" val="$1"
shift
[[ "$IFS$*$IFS" == *"$IFS$val$IFS"* ]]
}
if appears_in "somevalue" "${array[@]}"; then do stuff; fi

For anyone reading who may not know, the trick is that "$*" expands to a
single string containing all the positional parameters separated by the
first character of $IFS (or with no separator if IFS is empty, but
that's not useful here).

Of course this function is dependent on none of the elements containing
"|". But this lets you set any separator you want, so you could use a
really unlikely character such as IFS=$'\1'.

- M.

Re: [PATCH] admit 'typeset' is here to stay

[Martijn Dekker]

Dan Douglas schreef op 20-02-16 om 21:55:
> "typeset" vs. "declare" isn't always a straightforward decision
> depending on one's requirements. Bash's "typeset" has always
> been there for portability despite being a synonym for "declare",
> so I think just calling it a synonym would be counterproductive.
> http://thread.gmane.org/gmane.comp.standards.posix.austin.general/8371/focus=8377

Am I missing something? I thought they did exactly the same thing in
bash. If I'm not wrong about that, then as far as bash is concerned,
they are in fact synonyms and functionally equivalent.

> It's tricky because the big problems surrounding "typeset" lie in the
> details of variable scope and how each option is supported. There are
> some good reasons for choosing "typeset" if you're willing to work around
> all the subtleties but I think that's a bit beyond a concise description
> for the help output.

'typeset' and 'declare' in bash are both dynamically scoped, aren't
they? Where is the difference between them?

'typeset' in AT&T ksh93 is statically scoped, but ksh93 is the only
shell to have statically scoped local variables -- all the other shells
with 'typeset' (pdksh/mksh, bash, zsh, yash) have it as in ksh88,
dynamically scoped -- and it doesn't look like this is ever going to
change, or it would have changed sometime in the 23 years since ksh93
came out.

> For the most common requirement: localizing or applying an attribute to
> a variable within a single function; "local" and the posix-compatible
> declaration builtins are usually what people want. "local" is supported
> by every current implementation that also has "typeset" and "declare",

That would be bash and zsh only, so that's not very meaningful.

Perhaps you meant "every current implementation that also has
'typeset'", which would be ksh93, pdksh/mksh, bash, zsh, and yash.

However, ksh93, pdksh/mksh, and yash don't have "local".

The latter two have dynamically scoped 'typeset', so on those, you can
simply alias 'local' to 'typeset' to get the same functionality with the
familiar keyword.

But on ksh93, getting the equivalent of a dynamically scoped 'local' is
impossible. It has statically scoped local variables only. See Part III,
Q28 at http://kornshell.com/doc/faq.html

- M.

[Bug] kill -l 0 outputs T, not EXIT

[Martijn Dekker]

The command

kill -l 0

outputs T. I would expect EXIT as T is not a valid signal or
pseudosignal name. Since T is the last letter of EXIT, I suspect a typo
somewhere.

AT&T ksh93 outputs EXIT. pdksh/mksh and zsh output 0 (presumably EXIT is
not a real signal). dash and yash give a "no such signal number" error.
All of these seem sensible, but "T" does not.

Confirmed in bash 2.05b.13, 3.2.57 and 4.4.0.

Thanks,

- M.

Re: [Bug] kill -l 0 outputs T, not EXIT

[Martijn Dekker]

Op 13-09-16 om 14:29 schreef Greg Wooledge:
> On Tue, Sep 13, 2016 at 05:11:05AM +0200, Martijn Dekker wrote:
>> The command
>>
>>  kill -l 0
>>
>> outputs T. I would expect EXIT as T is not a valid signal or
>> pseudosignal name. Since T is the last letter of EXIT, I suspect a typo
>> somewhere.
> 
> My guess is it's just printing the string with the first 3 characters
> removed (SIGHUP -> HUP, SIGINT -> INT, EXIT -> T).

You're right. The fix is pretty simple, too. Patch attached.

- M.

diff --git a/builtins/common.c b/builtins/common.c
index b100ebe..2bfdf85 100644
--- a/builtins/common.c
+++ b/builtins/common.c
@@ -785,7 +785,7 @@ display_signal_list (list, forcecols)
 #if defined (JOB_CONTROL)
  /* POSIX.2 says that `kill -l signum' prints the signal name without
 the `SIG' prefix. */
- printf ("%s\n", (this_shell_builtin == kill_builtin) ? name + 3 : 
name);
+ printf ("%s\n", (this_shell_builtin == kill_builtin && STREQN (name, 
"SIG", 3)) ? name + 3 : name);
 #else
  printf ("%s\n", name);
 #endif

[BUG] incorrect exit status from subshell on bash with --enable-minimal-config

[Martijn Dekker]

On bash with --enable-minimal-config, as of version 4.3 (as far as I can
tell), there appears to be a bug with the exit status on returning from
subshells. Apparently the exit status of the subshell is not given to
the main shell.

On bash-4.4 with --enable-minimal-config:
$ (false);echo $?
0
$ (false);echo $?
0
$ (false);echo $?
0

On bash-4.4 with --enable-minimal-config --enable-alias:
$ (false);echo $?
1
$ (false);echo $?
0
$ (false);echo $?
0

Note the difference in behaviour: if --enable-alias is used it acts
correctly but only once.

On bash-4.3 the behaviour is identical. bash-4.2 doesn't compile on my
system so I was unable to test it. On bash-4.1 and bash-4.0 the bug is
not present. Testing was done on Mac OS X 10.11.6.

Thanks,

- Martijn

Re: [BUG] incorrect exit status from subshell on bash with --enable-minimal-config

[Martijn Dekker]

Op 22-09-16 om 00:06 schreef Martijn Dekker:
> On bash-4.4 with --enable-minimal-config:
> $ (false);echo $?
> 0
> $ (false);echo $?
> 0
> $ (false);echo $?
> 0

Another clue to the nature of the bug: it apparently happens before
exiting the subshell.

$ (false && echo true || echo false)
true
$ false && echo true || echo false
false

- M.

[BUG] false positive: [ ! -o optname ]

[Martijn Dekker]

The '!' operator in the legacy test/[ builtin does not invert the result
of the -o operator. Consequently the command

[ ! -o noclobber ]

amounts to a no-op, always returning exit status 0.

Proof:

$ set -o noclobber && [ -o noclobber ] && [ ! -o noclobber ] && echo bug
bug

Interestingly, mksh has this bug as well, though not the original pdksh.

Thanks,

- M.

Re: [BUG] false positive: [ ! -o optname ]

[Martijn Dekker]

Op 25-09-16 om 22:40 schreef isabella parakiss:
> On 9/25/16, Martijn Dekker  wrote:
>> The '!' operator in the legacy test/[ builtin does not invert the result
>> of the -o operator. Consequently the command
>>
>>  [ ! -o noclobber ]
>>
>> amounts to a no-op, always returning exit status 0.
[...]
> [ ! -o noclobber ] means
> is '!' a non empty string?  if not, is 'noclobber' a non empty string?

Wow. Yeah, I suppose that's one possible interpretation. I was going for
the POSIX one:

| These primaries can be combined with the following operators:
|
| !  expression
| True if expression is false. False if expression is true.
Ref:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html#tag_20_128

And indeed your interpretation does not apply to something like
"[ ! -e /tmp ]":

$ [ -e /tmp ]; echo $?
0
$ [ ! -e /tmp ]; echo $?
1

However, the supposed synonym -a acts differently:

$ [ -a /tmp ]; echo $?
0
$ [ ! -a /tmp ]; echo $?
0

Well, one more entry in the long list of arguments to avoid test/[...

Thanks,

- M.

Re: [BUG] false positive: [ ! -o optname ]

[Martijn Dekker]

Op 25-09-16 om 22:57 schreef Martijn Dekker:
> And indeed your interpretation does not apply to something like
> "[ ! -e /tmp ]":
> 
> $ [ -e /tmp ]; echo $?
> 0
> $ [ ! -e /tmp ]; echo $?
> 1
> 
> However, the supposed synonym -a acts differently:
> 
> $ [ -a /tmp ]; echo $?
> 0
> $ [ ! -a /tmp ]; echo $?
> 0

Which also makes sense now that I think about it, since -a means "and"
so it tests for the non-emptiness of both strings.

Bug report withdrawn.

Sorry for the noise,

- M.

Command substitution optimisation in dot scripts

[Martijn Dekker]

I detected an oddity (possible bug) in bash: the usual optimisation for
launching external processes in simple command substitutions is turned
off while executing a dot script.

Background: For reasons that would take too much space to explain here,
I need a cross-platform/POSIX way to get the process ID of the current
subshell. I can't use $BASHPID for this as it's only available on newer
bash (not including the default bash on Mac OS X).

As far as I know, the canonical cross-platform way of detecting the PID
of a subshell is:

my_subshell_pid=$(sh -c 'echo $PPID')

This works fine on every shell, except on bash when a dot script is
being executed.

This can be tested as follows:

(
my_subshell_pid=$(sh -c 'echo $PPID') &&
echo "PID of this subshell is: $my_subshell_pid" &&
echo "\$BASHPID is: $BASHPID" &&
trap 'echo "Correctly terminated"; exit 0' TERM &&
kill -s TERM "$my_subshell_pid"
)

This works fine on bash if executed normally or in a normal shell script:

$ bash test.sh
PID of this subshell is: 87562
$BASHPID is: 87562
Correctly terminated

But if you source it as a dot script (using '.' or 'source') from the
bash command line or another bash script, the cross-platform method
fails to get the correct subshell PID:

$ source test.sh
PID of this subshell is: 87551
$BASHPID is: 87550
bash: kill: (87551) - No such process

The PID obtained by the cross-platform method is off by one. My educated
(?) guess is that an extra bash subshell process is unnecessarily forked
just to launch the external 'sh' command. If that is the case, then this
adversely affects the performance of dot scripts in bash, which is why I
thought it seems worth reporting here.

Behaviour confirmed on bash-2.05b (Mac), bash-3.2.57 (Mac), bash-4.1.17
(Linux), bash-4.2.53 (Linux), and bash-4.4.0 (Mac).

Thanks,

- M.

BASH_SUBSHELL reset to 0 on EXIT trap in aubshell

[Martijn Dekker]

bash resets BASH_SUBSHELL to 0 when executing an EXIT trap, even if the
EXIT trap is executed in a subshell.

echo $(trap 'echo $BASH_SUBSHELL' EXIT; echo $BASH_SUBSHELL)
Actual output: 1 0
Expected output: 1 1

The same does not happen for a signal trap.

echo $(trap 'echo $BASH_SUBSHELL; exit' INT; echo $BASH_SUBSHELL;
   kill -s INT $BASHPID)
Actual output: 1 1
Expected output: 1 1

Thanks,

- Martijn

Re: BASH_SUBSHELL reset to 0 on EXIT trap in aubshell

[Martijn Dekker]

Also, perhaps BASH_SUBSHELL should be a readonly like BASHPID is.

- M.

Assigning to BASHPID fails silently

[Martijn Dekker]

bash 4.4.0 (I did not investigate other versions) does not produce an
error message if you try to assign something to the BASHPID readonly
using either arithmetic or normal assignment. Other readonlies produce a
message on an assignment attempt. BASHPID seems to be an exception.


Particularly annoying is that a non-interactive shell will exit silently
on a mistaken attempt to assign to BASHPID. I was making a bash function
to determine if we're in a subshell. Because of the lack of error
message, it took some doing to find my typo in
return "$((BASHPID = $$))"
(of course, comparison is '==' and not '='). Not even 'set -x' will give
a clue; it exits before producing relevant tracing output.


Proof below. (The exit status of the previous command is included in the
prompt in the output below.)

[0]$ UID=0
bash: UID: readonly variable
[1]$ BASHPID=0
[1]$ ((UID=0))
bash: UID: readonly variable
[1]$ ((BASHPID=0))
[1]$

Thanks,

- Martijn

Re: Assigning to BASHPID fails silently

[Martijn Dekker]

Op 19-10-16 om 15:18 schreef Chet Ramey:
> On 10/17/16 2:38 PM, Martijn Dekker wrote:
>> bash 4.4.0 (I did not investigate other versions) does not produce an
>> error message if you try to assign something to the BASHPID readonly
>> using either arithmetic or normal assignment. Other readonlies produce a
>> message on an assignment attempt. BASHPID seems to be an exception.
> 
> BASHPID is a dynamic variable.  There should be a sentence in the man
> page that says assignments to it have no effect (as it does for GROUPS
> and FUNCNAME, for example).

Assigning to BASHPID most certainly does have an effect. Since you
didn't quote that part, I think you might have missed my point that
attempting this will silently exit the shell without any error message,
causing the problem to be hard to track down. This is different from
GROUPS and FUNCNAME, where the shell silently continues (causing the
problem to be hard to track down in a completely different way -- if
anything, that's worse!).

> It probably should not be readonly, though
> it has been marked as such in previous versions of bash.

In what possible context would assigning to any of these variables make
sense, or be an indication of anything other than a fatal bug in the
script? I think they should all be readonly, and produce a proper
diagnostic error message upon exit if assigning is attempted.

If you're keeping them non-readonly for backwards compatibility reasons,
then IMHO that is misguided; assignments silently fail, which is far
worse than erroring out. Unsetting GROUPS and FUNCNAME first does work,
but it's a rare script that unsets variables before use. Maybe a
compromise would be to introduce a semi-readonly state, where unset
works but assignments fail.

Thanks,

- Martijn

Re: Assigning to BASHPID fails silently

[Martijn Dekker]

Op 20-10-16 om 14:22 schreef Greg Wooledge:
> On Wed, Oct 19, 2016 at 11:53:37PM +0200, Martijn Dekker wrote:
>> Assigning to BASHPID most certainly does have an effect. Since you
>> didn't quote that part, I think you might have missed my point that
>> attempting this will silently exit the shell without any error message,
>> causing the problem to be hard to track down.
> 
> Cannot reproduce:
> 
> imadev:~$ bash
> imadev:~$ echo $$
> 5659
> imadev:~$ BASHPID=923
> imadev:~$ echo $$
> 5659
> imadev:~$ exit
> imadev:~$ 

Urgh...

#! /usr/local/bin/bash
insubshell() {
return "$((BASHPID=$$))"
# ^ fatal typo
}
insubshell
echo continuing

The above script failed to produce output *once* (when run as 'bash
test.sh' with bash 4.4) and then output "continuing" ever since, so I
was suspecting a race condition. I can't reproduce it that way now for
the life of me, though.

However, a tweak proves I didn't lose my mind after all:

#! /usr/local/bin/bash
insubshell() {
return "$((BASHPID=$$))"
# ^ fatal typo
}
for ((i=0; i<1; i++)); do insubshell; done
echo $i
insubshell || echo ok
echo continuing

The output of this script on my copy of bash-4.4.0 is consistently:

| 0
| continuing

Clearly, that 0 should be 1. The 'insubshell' function somehow
manages to interrupt the 'for' loop.

So, in some contexts this bug causes a premature exit of the shell, in
others it causes a premature exit of a loop. This bug hunt could get
interesting.

This is on bash 4.4.0-release compiled by myself with default options on
Mac OS X 10.11.6 using "Apple LLVM version 7.3.0 (clang-703.0.31)",
architecture x86_64.

HTH,

- M.

Re: Race in bash-4.3 'typeset'?

[Martijn Dekker]

Op 25-10-16 om 00:42 schreef Stuart Shelton:
> Failing this, is there any alternative to ‘typeset’ to list a
> variable declared as local to a function but which has not yet been
> assigned a value?

Try simply testing the exit status of 'typeset -p' or 'declare -p'. If
the variable is not declared, it exits unsuccessfully.

if typeset -p "$var" >/dev/null 2>&1 && [[ ! -v $var ]]
then ...

As far as I can tell, this is not documented in 'help' or in the info
page, by the way. Perhaps it should be.

Also note that, while zsh and yash share this behaviour, ksh93 and
mksh/pdksh do not. If you want to be compatible with all the shells that
support 'typeset', you could do:

if [ -n "$(typeset -p "$var" 2>/dev/null)" ] &&
   eval "[ -z \"\${var+s}\" ]"
then ...

which is slower because it uses a subshell.

HTH,

- M.

Re: Race in bash-4.3 'typeset'?

[Martijn Dekker]

Op 25-10-16 om 18:19 schreef Stuart Shelton:
> However, it doesn’t appear to be able to detect local variables (is this 
> intentional or a bug?):

Strange. Works for me on bash 3.2.57 and bash 4.4.0 (Mac OS X) and bash
4.2.53 and 4.1.17 (Linux).

- M.

Re: Race in bash-4.3 'typeset'?

[Martijn Dekker]

Op 28-10-16 om 15:00 schreef Chet Ramey:
> On 10/25/16 12:40 AM, Martijn Dekker wrote:
>> Try simply testing the exit status of 'typeset -p' or 'declare -p'. If
>> the variable is not declared, it exits unsuccessfully.
>>
>> if typeset -p "$var" >/dev/null 2>&1 && [[ ! -v $var ]]
>> then ...
>>
>> As far as I can tell, this is not documented in 'help' or in the info
>> page, by the way. Perhaps it should be.
> 
> It's documented.  The man page, for instance, says:
> 
> "The return value is 0 unless ... one of the names is not a valid shell
> variable name"
> 
> which handles both the "not correctly formed" and "not found" cases.

To me that seems like a far-fetched interpretation. The way I read it,
if a variable is not declared, that does not imply its name is not
valid. An invalid name would mean it is impossible to declare it. I
think adding ", the variable is not declared" would prevent
misunderstandings.

- M.

Severe IFS glob bug needs patch [was: Setting IFS='*' causes bash to freeze on tab completion]

[Martijn Dekker]

Op 02-07-16 om 11:30 schreef Andreas Schwab:
> "Ibrahim M. Ghazal"  writes:
> 
>> After further investigation, the problem seems to be that when IFS is
>> set to '*', case *) doesn't get executed. Is this the intended
>> behavior?
>>
>> For example:
>>
>> IFS='*'; case "foo" in *) echo "got here";; esac
>>
>> doesn't print "got here".
> 
> The same issue exists for the other glob special characters ? [ ] .

Op 02-07-16 om 15:37 schreef Chet Ramey:
> Yep, that's a bug.  Thanks for the report and reproducer.  The problem is
> that the `*' ends up being matched literally instead of as a matching
> character.  This will be fixed in the next release of bash.

Unfortunately, this was only partially fixed in 4.4. The glob pattern
matching in 'case' and '[[' was fixed, but pathname expansion is still
broken:

$ IFS='?*[]'
$ ls *
ls: *: No such file or directory
$ touch a && ls ? [a]
ls: ?: No such file or directory
ls: [a]: No such file or directory

(Needless to say, this works fine on every non-bash shell.)

While this bug is relatively rarely triggered (since glob characters are
rarely put in IFS), it's actually REALLY SEVERE in its effects. Many if
not most scripts use 'case'/'[[' glob pattern matching or pathname
expansion at some point, so breaking either of these under any
circumstances is a big deal.

And when things do break, they break catastrophically. Not just your own
program breaks, but any third-party add-ons or libraries you're using
break as well.

While including any of the characters *, ?, [, ] in IFS might be rare,
strings do sometimes have values separated by one of these, so it's a
perfectly legitimate use case to split something on these characters.

So, in addition to fixing pathname expansion for 4.4, I hope you would
consider releasing a patch fixing both aspects of this bug for earlier
versions, at least down to 3.2 (which will forever be the default
version on Mac OS X, since Apple won't touch the GPL3).

Thanks,

- Martijn

Re: Severe IFS glob bug needs patch [was: Setting IFS='*' causes bash to freeze on tab completion]

[Martijn Dekker]

Op 04-11-16 om 01:27 schreef Eduardo Bustamante:
> I agree with everything, except calling it severe. This is
> self-inflicted harm, and easy to work around

It is not self-inflicted. The behaviour is a bug in bash. The user did
nothing to cause that bug.

Even if you think it's somehow reasonable to expect bash users to be
aware of this bug while using or coding for bash (though the bug has not
been publicised anywhere), bash is supposed to be a POSIX shell, and
POSIX is supposed to be a standard. You cannot assume that the code bash
executes is written with bash bugs in mind.

It is also not "easy" to work around. If your program for any reason
needs to split strings on any of the characters ?, *, [ and ], and uses
field splitting to achieve this (which in POSIX is the canonical
method), you will now need to program a workaround with something like
sed or parameter substitutions. Unless, of course, the easy workaround
is to simply use another shell.

Another big issue is that third-party library functions need to work for
arbitrary values of IFS. This bug makes that impossible for bash.

For instance, if you recall, the original bug reporter wanted to split
some string by "*" on an interactive shell. This caused a freeze upon
using auto-completion because the bash-completion package broke as a
result. Chet initially blamed it on bash-completion, but in fact the
fault was with bash itself. The resulting breakage was clearly severe.

- M.

Bug with && in piped functions in minimal-config bash

[Martijn Dekker]

My development with Modernish just exposed the following bug on bash
4.4.0(1)-maint, bash-20161028 snapshot, compiled with
--enable-minimal-config --enable-alias:

$ fn() { false && echo bug || echo ok; }
$ fn
ok
$ fn
ok
$ fn | cat
ok
$ fn | cat
bug
$ fn | cat
bug
$ (fn) | cat
ok

So apparently this only occurs if the function is piped into another
command, without explicitly declaring it to run in a subshell, and only
from the second time on.

Thanks,

- M.