Bash manual - possible minor formatting issue

[Greg]

Hello,

I think there may be a minor formatting issue in the set​ builtin section of 
the Bash manual 
(https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html). In 
the description of the "x​" option, the second "for" in the first sentence is 
formatted as inline code but I believe it should be formatted as regular text.

So instead of "arithmetic ​`for​​` commands" I believe it should be "arithmetic 
for commands" all in regular text. If I am mistaken then please ignore this 
email.

I have also attached a screenshot.

[image.png]

Regards,
Greg

FDs greater than 9 can't be closed

[greg]

Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: openbsd3.9
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i386' 
-DCONF_OSTYPE='openbsd3.9' -DCONF_MACHTYPE='i386-unknown-openbsd3.9' 
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/local/share/locale' -DPACKAGE='bash' 
-DSHELL  -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib  -I/usr/local/include 
-O2 -pipe
uname output: OpenBSD pegasus 4.0 GENERIC#1 i386
Machine Type: i386-unknown-openbsd3.9

Bash Version: 3.1
Patch Level: 1
Release Status: release

Description:
File descriptors greater than 9 seem to ignore move/close
redirections (e.g. 17<&-).  This problem does not occur in
earlier versions of bash, so it appears to be a regression.

Repeat-By:
Given this script:
#!/usr/local/bin/bash
exec 17& operators.

This problem was originally noted by "svizzero" in Freenode's
#bash channel.  I'm just reporting it so it can be fixed.

Fix:
The only workaround I've found is "don't use FDs greater than 9".


___
Bug-bash mailing list
[EMAIL PROTECTED]
http://lists.gnu.org/mailman/listinfo/bug-bash

Re: Behaviour of test -v with assoc array and quote character in key

[Greg Wooledge]

On Tue, Feb 23, 2021 at 12:17:10PM +0100, Alex fxmbsw7 Ratchev wrote:
> what, sorry, mailing stuff isnt much clear to me, ... its not possible to
> have a var=\'\] ; assoc[$var] ?

It should work for simple assignment and retrieval.

You need to add quotes for [[ -v 'assoc[$var]' ]] to work, and math
contexts are dodgy as hell.  No amount of quoting will make
(( 'assoc[$var]'++ )) work.  If you want to increment the value of
this array element, you'll need to retrieve it into a temporary string
variable, increment that, and then copy it back into the array.

Re: Behaviour of test -v with assoc array and quote character in key

[Greg Wooledge]

On Tue, Feb 23, 2021 at 02:03:30PM +0100, Léa Gris wrote:
> (
>   LANG=C
>   unset var assoc
>   var=\'\]
>   declare -Ai assoc
>   assoc[$var]=1
>   assoc[$var]+=1
>   ((assoc['$var']++))
>   typeset -p assoc
> )

This "works" in bash 5.0 but not in Debian's bash 5.1.  The -i flag is
irrelevant as well -- I get the same results with or without it (for
the command with (( )) in it).

Re: Behaviour of test -v with assoc array and quote character in key

[Greg Wooledge]

On Tue, Feb 23, 2021 at 02:03:55PM +0100, Andreas Schwab wrote:
> On Feb 23 2021, Greg Wooledge wrote:
> 
> > No amount of quoting will make (( 'assoc[$var]'++ )) work.
> 
> (( assoc[var]++ ))
> 
> Andreas.

This is not the same as assoc[$var].  Yours uses the literal string var
as the key, not the contents of the variable named var.

Re: Behaviour of test -v with assoc array and quote character in key

[Greg Wooledge]

Oğuz (oguzismailuy...@gmail.com) wrote:
> `(( assoc[\$var]++ ))' works fine as usual.

unicorn:~$ bash-5.1
unicorn:~$ declare -A hash
unicorn:~$ key=\'\]
unicorn:~$ hash[$key]=17
unicorn:~$ (( hash[\$key]++ ))
unicorn:~$ declare -p hash
declare -A hash=(["']"]="18" )
unicorn:~$ (( 'hash[$key]'++ ))
bash-5.1: ((: 'hash[']]'++ : syntax error: operand expected (error token is 
"'hash[']]'++ ")
unicorn:~$ (( hash['$key']++ ))
bash-5.1: ((: hash['']']++ : syntax error: invalid arithmetic operator (error 
token is "']++ ")
unicorn:~$ (( hash['$'key]++ ))
unicorn:~$ declare -p hash
declare -A hash=(["\$key"]="1" ["']"]="18" )

I understand absolutely none of this.  Quoting the $ works, but only
if you quote it with a backslash, not with single quotes?  And quoting
any other characters besides the $ fails?

Sorry, I'm still going with "this stuff's broken -- don't use it".

Re: Shell Grammar man page function definition

[Greg Wooledge]

Mike Jonkmans (bash...@jonkmans.nl) wrote:
> What does not work:
>   function x ( : )

The parser is looking for () after the function name.  Most likely, the
opening ( is confusing it.

unicorn:~$ bash
unicorn:~$ x() ( : )
unicorn:~$ function y() ( : )
unicorn:~$ type x
x is a function
x () 
{ 
( : )
}
unicorn:~$ type y
y is a function
y () 
{ 
( : )
}

Re: uname lead to embed code execution

[Greg Wooledge]

felix (fe...@f-hauri.ch) wrote:

> There it is:
> 
> $ declare -A map; key='foo$(uname >/dev/tty)bar'; map[$key]=
> $ echo map["$key"]
> map[foo$(uname >/dev/tty)bar]
> $ echo ${map["$key"]}
> 
> $ unset map["$key"]
> Linux

Yeah, it's sad but true.  I've just added a whole (short) section on
associative array index multiple expansions to my CodeInjection page.

https://mywiki.wooledge.org/CodeInjection

Re: doesnt interate the array anymore, what am i doing wrong

[Greg Wooledge]

Alex fxmbsw7 Ratchev (fxmb...@gmail.com) wrote:
> bash-5.1# cat int.fail
> int() {
>  declare -a int

You've declared a local array named "int" which has nothing in it.

This is inside a function named "int".

>  declare i=-1 now
> 
>  while [[ -v int[++i] ]] && now=${int[i]} ; do
>   printf %s\\n "$now"
>  done
> }

You're iterating on the empty local array.

> 
> int=( a b c )
> int

You also have a global array named "int".

I suggest that you use different names for your stuff.  This will help
you avoid a lot of problems.  Having a global variable, a local variable,
and a function all sharing the same name (and a name which is actively
*misleading* given that one of the arrays does not, in fact, contain
integer values) is not helping you.

Re: unsetting associative array executes commands

[Greg Wooledge]

On Thu, Mar 11, 2021 at 08:06:55AM -0700, Jason A. Donenfeld wrote:
> This behavior is quite surprising:
> 
> $ declare -A blah
> $ blah['$(DOESNOTEXIST)']=broken
> $ for i in "${!blah[@]}"; do echo "$i"; done
> $(DOESNOTEXIST)
> $ for i in "${!blah[@]}"; do unset blah["$i"]; done
> bash: DOESNOTEXIST: command not found
> bash: unset: [$(DOESNOTEXIST)]: bad array subscript

unicorn:~$ declare -A blah
unicorn:~$ blah['$(DOESNOTEXIST)']=broken
unicorn:~$ for i in "${!blah[@]}"; do unset 'blah[$i]'; done
unicorn:~$ declare -p blah
declare -A blah=()

Re: unsetting associative array executes commands

[Greg Wooledge]

On Thu, Mar 11, 2021 at 09:38:37AM -0700, Jason A. Donenfeld wrote:
> Single quotes with the nested double quote? That's nuts. But okay. I
> guess there really is some double expansion eval-like logic happening
> with the unset operator.

I removed the double quotes, as they serve no purpose there.

For an indexed array, use:  unset 'a[i]'
For an associative array, use:  unset 'a[$i]'

In both cases, use single quotes, and all will be well.

(In the indexed array case, there is another pitfall involving code
injection in the arithmetic context, so only use sanitized index values.)

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Tue, Mar 16, 2021 at 09:24:01AM +0100, Alex fxmbsw7 Ratchev wrote:
> it doesnt make me much sense that unset -v assoc[$key] gives syntax error
> or so..

Think of it this way: what would happen if you had a filename in a
variable -- say, $file -- and you ran this command:

rm -f $file

You'd expect that to fail, right?  Of course.

unset -v assoc[$key] fails for exactly the same reason.  unset does not
have any magical powers.  It's only a shell builtin, NOT a shell keyword.
Therefore, the standard shell rules apply.  Expansions occur first,
followed by word splitting, which cause a list of argument words to be
created and passed as arguments to the builtin.

key='two words'
unset -v assoc[$key]

==> 'unset' '-v' 'assoc[two' 'words]'

You can even use set -x to see this in action:

unicorn:~$ key='two words'
unicorn:~$ set -x
unicorn:~$ unset -v assoc[$key]
+ unset -v 'assoc[two' 'words]'
bash: unset: `assoc[two': not a valid identifier
bash: unset: `words]': not a valid identifier

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Tue, Mar 16, 2021 at 01:26:30PM +0100, Alex fxmbsw7 Ratchev wrote:
> no idea about your internal rules of keyword and builtin

unicorn:~$ type [
[ is a shell builtin
unicorn:~$ type [[
[[ is a shell keyword

You know how you're allowed to omit quotes inside [[ but not inside [ ?
That's because [[ has magical powers.  It's a "shell keyword".  It has
its own special rules.

> but clearly its
> gotta parse it at most 1. level otherwise its big security fault isnt it
> its a shell fuctionality keyword, unset, so ..

I would not complain if unset became a shell keyword, and had a complete
revamp of its behavior.  As long as we're willing to break backward
compatibility, which Chet seems like he's aiming to do anyway

Re: Likely Bash bug

[Greg Wooledge]

On Wed, Mar 17, 2021 at 10:59:42AM +0700, Robert Elz wrote:
>   | Operating system is BionicPup64 8.0.
> 
> That might.   More importantly is probably whatever package management
> system it uses.   I have no idea what the "ash" the bug report refers to
> is (there is an ancient shell of that name, but I cannot imagine any
> distribution including that, instead of one of its bug fixed and updated
> successors, like say, dash)

"ash" is often secret code for "busybox sh".

In this specific case, I'm not sure *what* it is.  I googled a few things
and came up with this page:

http://wikka.puppylinux.com/Ash

It says:

   Ash is a shell or command line interpreter. It acts as a cut down
   version of Bash with less commands. Puppy uses the Ash shell up to
   v 1.0.2

I'm not sure whether that's version 1.0.2 of this "ash" (whatever it
is), or of Puppy Linux.

> That is, not a bash problem at all.

Quite.

"Hi, I broke my OS, and I don't know how to fix it, and I don't know
how to contact my OS vendor's support, so I'll just file a bug against
a random shell instead!  Even though it's not the shell I'm using!"

Re: is it a bug that \e's dont get escaped in declare -p output

[Greg Wooledge]

On Wed, Mar 17, 2021 at 01:55:56PM -0400, Eli Schwartz wrote:
> On 3/17/21 12:43 PM, Alex fxmbsw7 Ratchev wrote:
> > it makes the output no more possible if interpreted by tty
> > the \e's get processed by terminal and no more .. just an example where it
> > is so:
> > 
> > var=$'1\e[G\e[K2' ; declare -p var
> > 
> > if G was H for other line then it completly messes the terminal up
> > 
> > isnt it better to escape by \e ? ..
> 
> Why is it the bash script interpreter's job to escape user-supplied
> strings based on how a terminal emulator chooses to *visually* display
> the results?

Call it a request for enhancement.

I thought, for a moment, that bash already used $'...' quoting for
newlines, but it turns out that's false.  At least for declare -p.
It would be nice if it did, though.  Newlines, carriage returns, escape
characters, etc.

Re: is it a bug that \e's dont get escaped in declare -p output

[Greg Wooledge]

On Wed, Mar 17, 2021 at 09:58:24PM +0200, Ilkka Virta wrote:
> On Wed, Mar 17, 2021 at 8:26 PM Greg Wooledge  wrote:
> 
> > I thought, for a moment, that bash already used $'...' quoting for
> > newlines, but it turns out that's false.  At least for declare -p.
> > It would be nice if it did, though.  Newlines, carriage returns, escape
> > characters, etc.
> >
> 
> It does in some cases:
> 
>  $ a=($'new \n line' $'and \e esc'); declare -p a
> declare -a a=([0]=$'new \n line' [1]=$'and \E esc')

But not for string variables, it seems.

unicorn:~$ unset a b; a=($'x\ny') b=$'c\nd'; declare -p a b
declare -a a=([0]=$'x\ny')
declare -- b="c
d"

It would be nice if the string variables were handled the same way as
the array elements.

Re: Probably not a bug but I was surprised: $' does not work inside "..." close.

[Greg Wooledge]

On Wed, Mar 17, 2021 at 09:11:48PM -0400, Dale R. Worley wrote:
> I have it encoded in my head that $ inside "..." is respected.  Subtly,
> the $'...' construction is not respected inside "...".

Bash has 5 types of quoting: '...', "...", $'...', $"..." and backslash.

$'...' is a form of quoting, not an expansion.  It won't "work" inside
of another type of quoting, just like '...' will not "work" inside "...".

echo "foo is '$foo'"

In this example, $foo is expanded, despite the fact that there are
single quotes around it, because the single quotes have no meaning in
this context.  They're inside a *different* type of quoting, so they
are just regular literal characters.

The only exception here is backslash, which retains its special powers
when inside "..." or $"...".

Re: missing way to extract data out of data

[Greg Wooledge]

On Thu, Mar 18, 2021 at 02:01:12PM +0100, Alex fxmbsw7 Ratchev wrote:
> there is way to crop data to wanted, by cropping the exclulsions away
> but what about a way to extract data, eg @( .. ) match
> not about using [[ =~ but only var internal stuff
> .. or do i miss there something

This is really a help-bash question, not a bug-bash question.  If you
show us your input, and your desired output, we might be able to give
you some suggestions on how to write your script.

Re: repost of alike 'missing a way to extract data'

[Greg Wooledge]

On Thu, Mar 18, 2021 at 02:15:19PM +0100, Alex fxmbsw7 Ratchev wrote:
> pseudo exmaple

Why not give a REAL example?

> declare -A big=( mess of data )
> var=$( declare -p big )
> 
> now to extract the elements, beware with mess of data i dont mean 'mess'
> 'of' 'data'

You don't write bash code to parse bash syntax.  You let bash do it
for you.

Apparently you are trying to serialize an associative array and transmit
it from one instance of bash to another.  That's totally fine, and
declare -p is the right tool for that.

In the second instance of bash, to reconstruct the serialized array,
you simply eval the serialization (or source it, if it's in a file).

In the first script:

declare -A hash=(...)
export serialized_hash="$(declare -p hash)"
otherscript

Then, inside otherscript:

eval "$serialized_hash"

Now you have an associative array named "hash", with a copy of the contents
from the first script.  If you want to "extract the elements", you simply
use "${hash[key1]}" and so on.

Re: repost of alike 'missing a way to extract data'

[Greg Wooledge]

On Thu, Mar 18, 2021 at 02:44:06PM +0100, Alex fxmbsw7 Ratchev wrote:
> isnt it obvious ..

No.

> ${var//@(abc|cde)@(bla|blubb)/\2\1} # just like sed
> and an extension for regex there

That's a feature request.  Chet can discuss that with you.

All I can tell you is that this feature is not in any way suitable for
retrieving an element from an associative array.  If it actually existed,
which of course it doesn't.

> as for example, i can give you my garbage data
> 
> begin='declare -A a' end='declare -p a' per2 'a[$per]=' *
> declare -A a=([""]="" ["] \$( eval echo shitt )"]="" [$'abc
> \034[22]="bar"']="" [$'\034']="" [$'\n']="" [2]="" ["]"]="" [" a   b"]=""
> ["' [b"]="" ["\" ["]="" [$'\' [1]=\034']="" ["];eval echo shitt"]=""
> [$'foo\034 [2]="blah" ']="" [abc]="" ["abc [22]=\"bar\""]="" [blah]="" )

I cannot make ANY sense of this.  I don't know what you're doing or why.

Whatever you're doing, it's ridiculously convoluted and obscure.  There
is probably some MUCH easier way to achieve the goal.  If you would just
tell us the goal, we might be able to help you do it.

Re: missing way to extract data out of data

[Greg Wooledge]

On Fri, Mar 19, 2021 at 09:10:06PM -0400, Dale R. Worley wrote:
> Alex fxmbsw7 Ratchev  writes:
> > yea well it does wonders, however was looking for a way without spawning
> > externals like gawk.. maybe in future there will be =)
> 
> Traditionally, shell scripts depended on external binaries to do a lot
> of the processing.  At the least, what newer shells do with "{NNN..MMM}"
> and "[[" used to be done by "seq" and "test" a/k/a "[".  And what can be
> done by the complex parameter expansions ${...%...} and ${...#...} was
> done by "sed".

Partly true.  seq(1) is a Linux thing, and was never part of any
tradition, until Linux people started doing it.  Of note, it does not
exist on any other systems.  Some BSDs have jot(1) which does a similar
thing, though.  Counting in POSIX sh involves a while loop, and i=$((i+1)).
(Before POSIX, it involved using expr(1) for every increment, which
is simply abominable.)

test or [ has been a builtin in every common shell for ages.  The days
of shells without a builtin test command are long gone.  The difference
between [[ and test is not really significant, if you're using the
operators that are shared between them (e.g. test -f "$f").

The % and # parameter expansions have been around since ksh88, and are
standardized in POSIX.  You may be thinking of ${var//old/new} which
is not part of POSIX, and would definitely have been done in sed(1).

While I'm nitpicking your details, you are correct about your overall
point, which is that shells are designed to use external tools to get
things done.  There's certainly a large grey area of tasks that can be
done by builtins or by external tools, of course, but beyond a certain
point, trying to force a shell to act like a Real Programming Language
is just not reasonable.

Re: about the local not-on-every-function-separately var issue

[Greg Wooledge]

On Mon, Mar 22, 2021 at 03:12:25AM +0100, Alex fxmbsw7 Ratchev wrote:
> i realize its somewhat of a big limitation, to have only global and
> one level further ( local ) args, no per function

One or more of your assumptions are wrong.

Bash uses "dynamic scope" when it expands variables.  This means that
it looks first in the current functions local variables; if the variable
isn't found there, it looks in the caller's local variables, and then
in the caller's caller's local variables, and so on, until it reaches
the global scope.


f() {
  local var=set_in_f
  g
}

g() {
  echo "var is $var"
}

var=global
f
# Prints "var is set_in_f"


Now, the big question is WHY you thought something which is not correct.

The most common reasons that people think something which is wrong are:

1) They heard or read it somewhere, and did not verify it themselves.

2) They encountered a problem with their own program, and while attempting
   to track down the problem, they took a wrong turn, and did not fully
   diagnose the situation.  They drew a wrong conclusion from partial data.

In your case, I suspect it's #2.

This project that you've been working on is so incredibly arcane,
convoluted and bizarre that *nobody* understands it, including you.  Who
knows how many other fallacious assumptions are baked into it, since you
are apparently incapable of simplifying anything down to the most basic
level for debugging, or even explaining what your *goal* is.

Re:

[Greg Wooledge]

On Mon, Mar 22, 2021 at 09:50:06AM +, Budi wrote:
> in a function k() in  ~/.bashrc
> 
> k(){ unset u;h=0; o=(${h:+ ! -path "./*.txt"}) ;u=($u "${o[@]}"); c=(.
> -regextype posix-extended "${b[@]}" -print); find "${c[@]}"
>  }
> 
> in output set -x:
> 
> + unset u
> + h-0

This is clearly a falsehood.  You didn't paste the actual output.

> + o=(${h:+ ! -path "./*.txt"})
> + u=($u "${o[@]}")

Why have the unquoted $u here when you know the variable is unset?

> + c=(. -regextype posix-extended "${b[@]}" -print)

What's in the array "b"?  You never showed any such array.

> + find . -regextype posix-extended ' ! -path ./*.txt' -print
> find: paths must precede expression: ` ! -path ./*.txt'
> 
> Why and how to solve ?

I think your real question is this one -- why did the exclamation point
and the word -path and the word ./*.txt all get squashed together into
a single word, instead of being inserted as three separate words?

If you can simplify your code down to *just* that case, and if you can
show an actual reproducer for it, then we might try to reproduce it
ourselves.

As it is now, I don't trust what you've written here, because you clearly
haven't been honest about it, and you haven't shown the actual commands
you ran ("set -x output" of what??), let alone the actual output.

Re: zsh style associative array assignment bug

[Greg Wooledge]

On Sun, Mar 28, 2021 at 10:01:14PM +0300, Oğuz wrote:
> On Sun, Mar 28, 2021 at 9:02 PM Eric Cook  wrote:
> 
> > in typeset -A ary=([key]=) an explicit empty string is the value
> 
> No. An "explicit" empty string would be '', "", or something like that.
> After `=' a value is expected but it's not there, so `[key]' is assigned
> the empty string.

Consider this assignment command:

x=

It's syntactically valid, and it assigns the empty string to the variable x.

It's perfectly reasonable, therefore, for

a=([key]=)

to assign the empty string to a[key].  You don't need '' or "" or whatever.

Now, as far as hash=(a b c) goes, I don't have a strong opinion about it.
Some of the other languages that have such a feature treat it as an error.
We've already heard from zsh.  Here's Tcl:

% array set hash {a b c}
list must have an even number of elements

Perl does what bash does, kinda:

unicorn:~$ perl -e 'use Data::Dumper; %hash=qw(a b c); print Dumper(\%hash);'
$VAR1 = {
  'a' => 'b',
  'c' => undef
};

Python is different:

>>> y = ["a", "b", "c", "d"]
>>> dict(zip(y[::2], y[1::2]))
{'a': 'b', 'c': 'd'}
>>> x = ["a", "b", "c"]
>>> dict(zip(x[::2], x[1::2]))
{'a': 'b'}

It seems to discard the last (unmatched) value.  Also, dear gods, what
is that horrible syntax they forced me to google for... and it took MANY
tries to find it, too.

So, that's 2 votes for error, 2 votes for using the empty string (or
null/undef) as the last value, and 1 vote for silently dropping the
last key.  Definitely not a consensus.

Re: Default PS1

[Greg Wooledge]

On Mon, Mar 29, 2021 at 01:42:10PM +0100, Chris Elvidge wrote:
> On 29/03/2021 12:04 pm, ილია ჩაჩანიძე wrote:
> > How can I set default PS1 variable from source code?
> > E.g I want it to display:
> > My-linux-distro $
> > And not:
> > Bash-5.1 $
> > 
> 
> Set it in $HOME/.bashrc

Given that the goal is clearly to make a new "default" for all users
on their custom operating system, a more viable answer is probably
to enable the SYS_BASHRC feature in config-top.h and then set the PS1
variable in /etc/bash.bashrc (or whatever file they point it to) to
advertise their operating system to people who already run it.

Re: Default PS1

[Greg Wooledge]

On Mon, Mar 29, 2021 at 01:49:41PM -0700, L A Walsh wrote:
> Or, what do you mean by 'default'?  Is it sufficient
> to set it in the system /etc/profile so it is the default
> for all users when logging in?

Sadly, that won't work.  There are plenty of *extremely* common paths
from power-on to shell which do not read /etc/profile.

This sort of request (a polite word) is precisely why SYS_BASHRC was
created.

Re: Default PS1

[Greg Wooledge]

On Mon, Mar 29, 2021 at 07:25:53PM -0700, L A Walsh wrote:
> On 2021/03/29 14:39, Greg Wooledge wrote:
> > On Mon, Mar 29, 2021 at 01:49:41PM -0700, L A Walsh wrote:
> > > Or, what do you mean by 'default'?  Is it sufficient
> > > to set it in the system /etc/profile so it is the default
> > > for all users when logging in?
> > 
> > Sadly, that won't work.  There are plenty of *extremely* common paths
> > from power-on to shell which do not read /etc/profile.
> 
>I have both /etc/profile and /etc/bashrc call my configuration
> scripts.  Are there common paths that don't call one of those?

A vanilla bash compiled from GNU sources with no modifications will
not source /etc/bash.bashrc or /etc/bashrc or any other such file.
The SYS_BASHRC feature is off by default, and must be enabled at
compile time.  Many Linux distributions enable it, but there are
surely systems with bash installed which have not enabled it.  So
that's one thing.

Any login that uses a graphical Display Manager does not read /etc/profile.
So that's the second thing.

Now imagine a system with a vanilla bash, and a Display Manager login.

Re: zsh style associative array assignment bug

[Greg Wooledge]

On Tue, Mar 30, 2021 at 12:42:46PM -0400, Eric Cook wrote:
> typeset -A tags=(); set --
> while IFS='|' read -ra ary; do
>   set -- "$@" "${ary[@]}"
> done < <(
>   exiftool -j *.flac |
>   jq -r '.[]| {Artist, Track, Genre, Title}|to_entries[]| .key + "|" + .value'
> )
> eval 'tags=('"${*@Q}"\)
> typeset -p tags
> declare -A tags=([lofi]="Title" [Track]="Genre" [Artist]="AK420" ["A2 - 
> Northern Lights"]="" )

This is just poorly written.

declare -A tags=()
while IFS=\| read -r tag value; do
  tags[$tag]=$value
done < <(exiftool ...)

In addition to the bug you demonstrated, the code as written also breaks
if there are any pipelines in the value field.  (Or the tag field, but I'm
guessing that's not possible.)

One of the most *beautiful* things in bash is how "read -r var1 var2"
works.  The entire remaining input line gets stored in var2, even if
it contains additional instances of the IFS delimliter.  Take advantage
of this.

> Yeah, it is a programming error that could've used better validation.
> I just find it weird that the assumption of an assignment with an odd number 
> of elements with
> this new syntax is that the odd number element is always a key missing a 
> value that is filled in.
> when any of the keys or values could've been missing during the assignment.

I'm not disagreeing with you here, but as I pointed out earlier, there
are languages that agree with how bash currently works, languages that
agree with the "throw an error" approach, and so on.

Re: Defaults -- any

[Greg Wooledge]

On Tue, Mar 30, 2021 at 09:54:35AM -0700, L A Walsh wrote:
> On 2021/03/29 20:04, Greg Wooledge wrote:
> > On Mon, Mar 29, 2021 at 07:25:53PM -0700, L A Walsh wrote:
> > > 
> > >I have both /etc/profile and /etc/bashrc call my configuration
> > > scripts.  Are there common paths that don't call one of those?
> > 
> > A vanilla bash compiled from GNU sources with no modifications will
> > not source /etc/bash.bashrc or /etc/bashrc or any other such file.
> 
> 
>So this manpage text is wrong (focusing on paths in /etc):
> 
>When  bash is invoked as an interactive login shell, or as a non-inter-
>active shell with the --login option, it first reads and executes  com-
>mands  from  the file /etc/profile.  Please note that the file
>/etc/profile includes an autodetection shell  code  w(h)ether  it  has
> to
>source /etc/bash.bashrc.
> 
>When  bash is invoked as an interactive login shell, or as a non-inter-
>active shell with the --login option, it first reads and executes  com-
>mands  from  the file /etc/profile, if that file exists.
>Please note that the file /etc/profile includes an autodetection
>shell  code w(h)ether  it  has  to source /etc/bash.bashrc.

This text is not part of upstream bash, nor part of Debian's patched
version of bash.

unicorn:/var/tmp/bash/bash-5.1/doc$ grep autodetect *
unicorn:/var/tmp/bash/bash-5.1/doc$ man bash | grep autodetect
unicorn:/var/tmp/bash/bash-5.1/doc$ 

I can't say whether it's correct on *your* system or not, since your
system is using a different set of patches than mine.  You'll have to
discuss it with your vendor.

>Not really pertinent how my distro compiles bash, as it's on my
> short list of programs I compile myself.

In that case, you *need* to discard that altered man page.  It's got
misinformation that will not be correct for your self-compiled bash.

>SGI, took pains to have the graphical display manager read the login
> scripts.  I think suse does as well.  I login with a Windows GUI (it
> doesn't)
> read *nix login scripts.

Oh yeah, SuSE.  SuSE is *notorious* for wreaking havoc with shell configs.

Good luck with that!

Re: incorrect character handling

[Greg Wooledge]

On Wed, Mar 31, 2021 at 02:31:46AM +0700, by.sm--- via Bug reports for the GNU 
Bourne Again SHell wrote:
> poc=whoami
> $poc
> python3 -c "print('!!')"
> 
> That return 'whoami' command. 

You're running into the csh-style history expansion.  A lot of us simply
disable it, because it's not worth the effort it takes to work around it.

set +o histexpand


If you insist on keeping it, and working around it, the key is to
understand that single quotes will protect you, but double quotes may
not.

echo 'hi!'

echo "hi!"


The actual behavior of ! (history expansions) inside double quotes has
changed across bash versions, so you may have more problems, or fewer
problems, depending on your bash version.  But there will never be
zero problems.

Re: zsh style associative array assignment bug

[Greg Wooledge]

On Wed, Mar 31, 2021 at 09:40:34AM +0200, felix wrote:
> As bash read loop could be something slow, I use (when I'm quiet about data
> origin) something like:
> 
>   declare -A tags="($(
>   sed -e 's/^\([^|]*\)|\?\(.*\)/[\1]="\2"/' < <(
>  exiftool ...)) )"

>   I'm not absolutely sure about border effect and security issues.

I'm absolutely sure that it's terribly unsafe.  If there are any double
quotes or (command) substitutions in the tag values, this breaks quite
hard.

I'm also confused about what the \? is supposed to be doing, and whether
you're using some kind of crazy GNUism there.  It looks like one to me.
Please remember that sed is supposed to use BRE, and the equivalent of
ERE's ? operator in BRE is \{0,1\} .

Re: select syntax violates the POLA

[Greg Wooledge]

On Thu, Apr 01, 2021 at 01:36:59AM -0700, greyw...@starwolf.com wrote:
>   The following is valid shell code:
> 
>   d=($(ls /usr/src/pkg/*/$1));

Syntactically valid, but semantically wrong.

If $1 is not a directory, then you want:

d=(/usr/src/pkg/*/"$1")

If $1 is supposed to be a directory, and therefore you want only filenames
and not full pathnames in the array, then we can arrange for that as well,
but it's slightly more tedious.

Using ls here is wrong, and will break.  It's Pitfall #1 on my list for
a reason.

https://mywiki.wooledge.org/BashPitfalls#pf1

>   cd ${dir} &&

Quotes.

>   make clean && {
>   make update ||
>   make install;
>   }

Is the inner || supposed to be && by chance?

>   if ((n > 1)); then {
>   select dir in ${d[@]}; do {
>   break;
>   } done;
>   }

"${d[@]}" with quotes.

What purpose do the extra curly braces serve?  Do you want them just
because you think they look pretty?

>   until [ -f ${tmpfile} ]; do {

Quotes.  And again, those extra curly braces don't serve any purpose.

>   for dir in ${d[@]}; do {

Quotes.

Re: select syntax violates the POLA

[Greg Wooledge]

On Thu, Apr 01, 2021 at 11:58:13PM +0700, Robert Elz wrote:
> 
>   | If $1 is not a directory, then you want:
> 
> It is a directory, or I'd guess, quite likely a pattern chosen

It's amazing how many people manage to post their code with NO comments
or explanations of what it's supposed to do, what assumptions are being
made about the inputs, etc.  This leaves us to guess.

I'm guessing that /usr/src/pkg/*/* is a list of files, not a list of
directories.  You're guessing that it's a list of directories.

In the original code, if it's a list of directories, then we have
"ls firstdir seconddir" which expands to a list of mangled filenames
without directory components.  And since there's more than one directory,
the array ends up looking like:

d=(README foo.c foo.h foo README bar.c bar.h foo.c foo.h foobar)

with repeated filenames, and no knowledge of which filename came out of
which subdirectory.

I was assuming that this was *not* how it was intended to work, and thus,
I made the guess that /usr/src/pkg/*/$1 expanded to a list of *files*,
and therefore generated a list of full pathnames, e.g.

d=(/usr/src/pkg/applesauce/Makefile.PL /usr/src/pkg/bananabread/Makefile.PL)

and then the user can make a meaningful selection from such a list.

Now, this is obviously just a guess, and I could be extremely wrong
about it.  For one thing, they used the variable name "d".  That seems
to indicate that it should be a list of directories, rather than a list
of files.

So, perhaps their code is simply *broken*, and where they *thought* they
would be getting a list of directories, they actually get a list of
mangled useless filenames, because they forgot the -d argument to ls.

>   | d=(/usr/src/pkg/*/"$1")
> 
> definitely not that, the quotes are wrong in any case

They are not!

> (but apart from
> that, if filename expansion happens in array assignments (it doesn't in
> normal ones, and I dislike arrays, so ...) then without the quotes that
> might work.
> 
> Alternatively
>   d=( $( ls -d /usr/src/pkg/*/$1 ) )

This needs quotes around "$1" but yes, it's quite possible that this was
their intent.  Of course, this is still *broken*, but it only breaks if
one of the directory names contains whitespace characters.  It's likely
that they didn't happen to have any such directories in their test
environment, and therefore, this bug would go unnoticed for a long time.

> or just
>   d=( $( printf %s\\n /usr/src/pkg/*/$1 ) )

This is not much better.  It still breaks on whitespace.

If a list of directories (not their contents) is the desired outcome,
then my original code is perfectly fine:

d=(/usr/src/pkg/*/"$1")

This will expand to something like

d=(/usr/src/pkg/vendor1/netcat /usr/src/pkg/vendor2/netcat)

> Just to be sure.Personally I'd do
> 
>   set -- /usr/src/pkg/*/$1
> 
> and then simply use the positional parameters.

This still needs quotes around "$1", and it's basically the same as
the array assignment -- just less flexible, and more suited to sh than
bash.  But certainly, this is viable in many cases.

> Yes.   But it turns out not to matter in this case, as none of
> the names will ever contain anything but "normal" characters
> (no spaces, newlines, asterisks, ...)

Famous Last Words™

Re: select syntax violates the POLA

[Greg Wooledge]

On Thu, Apr 01, 2021 at 02:54:55PM -0700, Greywolf wrote:
> the requirement
> to have ${var[ix]} instead of honouring $var[ix] with regard to arrays
> is another one).

Before the introduction of arrays, $var[ix] already had a meaning:
the value of the "var" parameter, followed by the 4-character string [ix].
Redefining $var[ix] to mean the same thing as ${var[ix]} would have
broken existing scripts.

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Tue, Apr 06, 2021 at 02:55:35PM +0900, Koichi Murase wrote:
> But, maybe we can introduce a special syntactic treatment of `unset'

When I made a comment about the possibility of unset becoming a keyword,
Chet said he had no plans to do that.

... here:
https://lists.gnu.org/archive/html/bug-bash/2021-03/msg00236.html

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Tue, Apr 06, 2021 at 11:28:13PM +0900, Koichi Murase wrote:
> 2) we
> can distinguish the erasure of the element associated with key=@
> `unset -v a[$key]' from the entire array erasure `unset -v a[@]'.

As a counter-proposal, Chet could entirely remove the special meaning
of unset 'a[@]' and introduce a new option to unset which would take
its place.  It appears -a is not yet used, so that would be a good pick.

Under this proposal, unset 'a[@]' would only remove the single element
whose key is '@', and unset -a a would remove all the elements of the
array.

I don't know whether any scripts actually use unset 'a[@]' to clear all
the keys out of an array, but this change would obviously break backward
compatibility if any such scripts exist.  Therefore I only bring it up
because this whole thread is about introducing new behaviors that break
compatibility.

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Tue, Apr 06, 2021 at 06:34:21PM +0300, Ilkka Virta wrote:
> $ declare -A a=([foo]=123 [bar]=456)
> $ unset 'a[@]'
> $ declare -p a
> bash: declare: a: not found
> 
>  i.e. both remove the whole array, not just the contents.

Huh, so it does.  I didn't know that.

In that case, I have no qualms about proposing that unset 'a[@]' and
unset 'a[*]' be changed to remove only the array element whose key is
'@' or '*', respectively, and screw backward compatibility.  The current
behavior is pointless and is nothing but a surprise landmine.  Anyone
relying on the current behavior will just have to change their script.

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Wed, Apr 07, 2021 at 01:46:27AM +0900, Koichi Murase wrote:
> I remember there was some discussion in the bug-bash list on the
> behavior of `test -v a[@]' which actually tests whether the array `a'
> has at least one element.

> It cannot be replaced by `test -v a' because
> `a' implies `a[0]' here.

Both of those behaviors look wrong to me, but I seem to be in the
minority.

> But if I correctly understand it, `test -v
> a[@]' can be replaced by `((${#a[@]}))'. Should any scripts relying on
> `test -v a[@]' to test the array elements also be changed?

I would never write a script like that, but it doesn't seem necessary
to break any scripts that are using it.  It's not a complete disaster
that destroys your entire array when the user types '@', so I could live
with it sticking around.

Anyone using test -v 'a[$key]' as an attempt to determine whether a key
exists may get the wrong answer when the key is * or @.

And for the record, 'a[@]' still needs to be quoted.


When you put all of these issues together, it's clear that associative
arrays in bash have *deep* flaws, and require some workarounds to use
safely.

1) The empty string cannot be used as an index, under any circumstances.

2) The indices * and @ may cause surprising behaviors in several different
   contexts.

Both of these can be worked around by adopting the old "put an x in front
of it" hack from the days of yore.  Ironic, isn't it, how some things
never die.

3) Expressions like a[$key] must be single-quoted in some contexts, or
   you get surprising behaviors.

4) Expressions like a[$key] must be backslash-protected in some other
   contexts, or you get surprising behaviors.

Just ugh.


> $ iref1=a[@]; printf '<%s>' "${!iref1}"; echo
> 
> $ key=@; iref2=a[$key]; printf '<%s>' "${!iref2}"; echo
> # <-- unexpected

This test is flawed; the $key is expanded during the assignment to iref2.

unicorn:~$ declare -A a=(['@']=x [1]=y)
unicorn:~$ key=@; iref2='a[$key]'; printf '<%s>' "${!iref2}"; echo


That said, the fact that you can put 'a[@]' in an indirect variable and
get an array expansion out of "${!x}" is completely repulsive to me.  In
the past, I have asked whether this is actually working as intended,
and Chet's answer simply confused me, but near as I can tell, he seems
to think it should continue doing this.  So I won't say anything more
about it.

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Wed, Apr 07, 2021 at 03:53:43AM +0800, konsolebox wrote:
> Also if Bash could just store associative array
> values as a list to preserve store order and stop expanding
> "${array[@]}" based on the sorted order of the keys, then the slice
> function can also be applied there.

There is no sorting in the output.  The keys come out in an order that
only makes sense to the little gerbils inside bash, not to us humans.
This is common with hash tables, and the same thing happens with python's
dictionaries, perl's hashes, and tcl's arrays (but not tcl's dictionaries,
which are closer to what you're envisioning).

unicorn:~$ perl -e 'use Data::Dumper; %hash=qw(a b c d e f g h); print 
Dumper(\%hash);'
$VAR1 = {
  'e' => 'f',
  'g' => 'h',
  'c' => 'd',
  'a' => 'b'
};

unicorn:~$ tclsh <<'EOF'
array set hash {a b c d e f g h}; puts [array names hash]
EOF
e a g c

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Tue, Apr 06, 2021 at 03:24:54PM -0500, Dennis Williamson wrote:
> Python 3.7 has insertion-order dictionaries. So these are dependable
> gerbils.

That explains my test results, which I didn't post.  I had been told
in the past that python's dictionaries did NOT remember insertion
order, but I'm not fluent in python and I don't keep up with its changes.

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Thu, Apr 08, 2021 at 03:37:33PM -0400, Chet Ramey wrote:
> On 4/6/21 1:42 PM, Greg Wooledge wrote:
> > That said, the fact that you can put 'a[@]' in an indirect variable and
> > get an array expansion out of "${!x}" is completely repulsive to me.
> 
> What do you think should happen?

What I would have expected originally: an error message and a non-zero
exit status.

But apparently someone stumbled upon this trick, and passed it around,
and now there's a whole subculture of people who use this as a hack for
trying to pass array variables to functions by reference.  (This hack
predates declare -n.)

So now I'm not advocating for any change to it.  It would break too
many scripts.  I can only advise people to start using declare -n
instead, with additional warnings about variable name scoping and
collisions (which are also problems with the indirection hack, or with
eval-based hacks).

Re: Changing the way bash expands associative array subscripts

[Greg Wooledge]

On Fri, Apr 09, 2021 at 08:17:52AM +0800, konsolebox wrote:
> On Fri, Apr 9, 2021 at 4:08 AM Greg Wooledge  wrote:
> > But apparently someone stumbled upon this trick, and passed it around,
> > and now there's a whole subculture of people who use this as a hack for
> > trying to pass array variables to functions by reference.  (This hack
> > predates declare -n.)
> 
> It's not a hack since indirection allows references to:
> 
> 1) Digit-based parameters ($1, $2, etc.)
> 2) Special variables (which includes $@)
> 3) Valid array references (array[@] and array[1 + 1] are one of them)
> 4) Legal identifiers
> 
> It's a definite shell feature despite lacking internal sanity checks.

Have you actually SEEN this?  This is how people use it:

myfunc() {
  # $1 is the name of an array we want to work with
  tmp=$1[@]   # And yes, they always name it "tmp".
  for i in "${!tmp}"; do
...
  done
}

How can you look at that code and call it anything other than a hack?
It's a piece of pure desperation.  You can only READ the array, not write
to it.  You can't do an index iteration, either -- only a value iteration.
And you still have all the same name collision issues that you'd get
with namerefs.  This makes it genuinely inferior to using a nameref to
acheive the same goal:

myfunc() {
  # $1 is the name of an array we want to work with
  declare -n aref=$1

  # Value iteration.
  for i in "${aref[@]}"; do
...
  done

  # Index iteration.
  for i in "${!aref[@]}"; do
...
  done

  # Writing.
  aref[42]=hey
}

(Insert here all of the problems with name collisions, which are real,
but you should already know them.)

Re: When "set -u", "which" outputs environment: line 1: _declare: unbound

[Greg Wooledge]

On Fri, Apr 09, 2021 at 03:08:25PM -0400, Craig Andrews wrote:
> Description:
> With "set -u", invoking "which" results in the output of:
> environment: line 1: _declare: unboundvariable
> That should not happen, and does not happen, with prior versions of
> bash.
> I'm using Fedora 34 (beta).
> 
> Repeat-By:
> Run this shell script:
> #!/bin/bash
> set -u
> echo "$(which bash)"
> 

I can think of two possible reasons for this:

1) which(1) on your system is a bash script, and the error message is
   coming from that script.

2) which on your system is a shell function, which is somehow made visible
   to the script that you ran (possibly having been exported through the
   environment, or declared in a file that's loaded via BASH_ENV).

Are you able to duplicate the error by running "which bash" in an
interactive shell, or does it only happen in scripts?

In whatever environment generates the error message, please run "type which"
to find out exactly how which is defined.

If it's a program (e.g. /usr/bin/which), please run "file" on that program
to see whether it's a (bash) script.

If it's a function, try to figure out where the function is being defined.

Re: enhancement merge request

[Greg Wooledge]

On Sun, Apr 18, 2021 at 04:12:10PM -0700, Ananth Chellappa wrote:
>  Sincerely appreciate your time. Far as I understand, there is no
> way to accomplish what I want - concisely : *get a true private mode* (no
> logging to HISTFILE *OR* recall with history command after exiting
> private-mode (toggle of history using set -/+ o) *without sacrificing
> productivity*. That is, when you are *in private mode, you DO WANT recall
> with !number and arrow keys.

bash --rcfile <(echo source ~/.bashrc; echo unset HISTFILE)

Re: enhancement merge request

[Greg Wooledge]

On Mon, Apr 19, 2021 at 01:16:48AM -0400, Grisha Levit wrote:
> I think you can do something similar to the patch directly in the shell by
> temporarily unsetting HISTFILE and then deleting history entries in a
> certain range and restoring HISTFILE when you are ready

I still think my solution fits the criteria exactly.  There was *no*
stated requirement that the "private mode" could not be a separate
shell process.  In fact, running a separate shell as your "private mode"
shell makes perfect sense to me.  It's more "secure" that way, right?

All the crazy backflips that are being attempted in order to make the
"private mode" coexist within the same shell process as "regular mode"
can simply go away once you decide to separate the two.

Re: Document variable names need to be all ASCII

[Greg Wooledge]

On Mon, Apr 19, 2021 at 08:28:54PM +0800, 積丹尼 Dan Jacobson wrote:
> $ e哈=1
> bash: e哈=1: command not found
> OK, but on man bash is it ever mentioned that a variable name must be all 
> ASCII?
> 
> ENVIRONMENT
>When a program is invoked it is given an array of  strings  called  the
>environment.   This  is  a  list  of  name-value  pairs,  of  the  form
>name=value
> 
> PARAMETERS
>A parameter is an entity that stores values.  It can be a name, a  num‐
>ber, or one of the special characters listed below under Special Param‐
>eters.  A variable is a parameter denoted by a name.  A variable has 
> ...

DEFINITIONS

   name   A word consisting only of  alphanumeric  characters  and  under‐
  scores,  and beginning with an alphabetic character or an under‐
  score.  Also referred to as an identifier.

"Alphanumeric" here clearly means [[:alnum:]] within the POSIX locale.

Re: Document that set -v inside case statements is special

[Greg Wooledge]

On Tue, Apr 20, 2021 at 11:07:51PM +0800, 積丹尼 Dan Jacobson wrote:
> Please document on the man page somewhere that set -v, +v inside case
> statements is special:
> $ cat A
> case x in x)
> set -v
> : B
> case y in y)
>  set -v
>  : Z
>  ;;
> esac
> ;;
> esac
> : C
> $ bash A
> : C
> 
> I.e., -v and +v in case statements are remembered, but only have effects
> after leaving all case statements. Unlike +x, -x. Same it turns out for 
> dash(1).
> Bash 5.1.4.

It's nothing to do with 'case' specifically.  You simply don't understand
what set -v actually does.

Unlike set -x, set -v reports when bash *reads* a line of input from the
script.  Not when it runs said line.

With case, or any other compound command, the entire compound command is
read from the script and parsed, before bash can start running it.  So, by
the time the set -v is executed, the entire case command has already been
read.

The next time bash reads a command is : C, so that's what set -v reports.

Re: Document that set -v inside case statements is special

[Greg Wooledge]

On Tue, Apr 20, 2021 at 09:25:04PM -0400, Dmitry Goncharov via Bug reports for 
the GNU Bourne Again SHell wrote:
> On Tue, Apr 20, 2021 at 2:37 PM Greg Wooledge  wrote:
> 
> > With case, or any other compound command, the entire compound command is
> > read from the script and parsed, before bash can start running it.  So, by
> > the time the set -v is executed, the entire case command has already been
> > read.
> 
> 
> Is not this what op wanted to be documented?

It's already documented.

  -v  Print shell input lines as they are read.

That's what it does.  That's all that needs to be said in the official
documentation.

Re: Problem with sequences with variables?

[Greg Wooledge]

On Wed, Apr 28, 2021 at 11:02:31PM +, Tom (AST) Watson via Bug reports for 
the GNU Bourne Again SHell wrote:
> Hi...
> 
> [user@box3 ~]$ echo $l
> 10
> [user@box3 ~]$ echo {1..${l}}
> {1..10}

https://mywiki.wooledge.org/BashPitfalls#pf33

Re: Brace expansion ordering vs. parameter expansion

[Greg Wooledge]

On Thu, Apr 29, 2021 at 03:12:09PM +0300, Ilkka Virta wrote:
> BTW, was there some background to why they're ordered like this? I'm not
> sure if I have heard the story, and didn't see anything about it in Greg's
> wiki or bash-hackers.org (of course they tell the "what", but not the
> "why"). I didn't dig through all the mailing lists, though.

I don't know the reason, other than what Chet said earlier in this
thread (that it was never designed to work that way).

> Of course there's other ways with subshells, temporary arrays and using
> e.g. seq (but I'm not sure that exists on all systems either).

seq(1) is not standard, and definitely does *not* exist on all systems.
Some have jot(1) instead, but many commercial Unixes have neither.

Re: RFE - support option for curses idea of term [tab]size.

[Greg Wooledge]

On Thu, Apr 29, 2021 at 07:39:37PM -0700, L A Walsh wrote:
>It doesn't always do it correctly because it doesn't always know
> where it is in a line.  As such it has a bug that would be fixed by
> having it really know where it was at (from talking with libcurses)
> as well as what the tabstops were really set to.

First of all, "just link with libcurses" does not magically grant the
power to know the cursor position in all terminals.  Many of them have
no such capability at all, and of the ones that do, the means of
learning that information differs greatly.

Even curses applications don't try to ask the terminal where the cursor
is.  They *put* the cursor where they need it to be, and keep track of
where they left it.

Second of all, converting bash/readline to use curses instead of the
primitive terminal calls that it currently uses would involve a whole
new layer of complexity that I do not feel is justified, or even wanted.

Curses applications take control of the entire terminal.  They are
initialized, and they must be cleaned up.  At any time in the middle,
the stats of the terminal is not really well-defined.  The curses
application could be in the middle of some operation.  Any outside entity
that touches the terminal in any way could mess everything up.

But with shell scripts (or interactive shells), the normal operating
procedure is for the shell to spend a lot of its time waiting for an
external command to finish.  That external command may want to write
to the terminal, or even read from it.

This works out fine for bash as it currently exists, because bash turns
over control of the terminal in a relatively well-defined state, and
resumes control of the terminal when the external program is finished.
The only assumption bash makes upon resumption is that the cursor will be
in the leftmost column.

Re: "unset x[0], a BASH bug?"

[Greg Wooledge]

On Thu, May 06, 2021 at 10:42:36AM +0300, Oğuz wrote:
> 6 Mayıs 2021 Perşembe tarihinde Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> yazdı:
> >
> > But why is $# decremented after the first unset?
> 
> Because `x[0]' existed then, and the number of x's members changed with its
> removal. `${#x[@]}' doesn't give you the top index of `x', it expands to
> the number of elements in `x'.

A helpful tip might be to use declare -p to show the array, instead of
simply echoing the values.

unicorn:~$ x=(1 2 3)
unicorn:~$ declare -p x
declare -a x=([0]="1" [1]="2" [2]="3")
unicorn:~$ unset 'x[0]'
unicorn:~$ declare -p x
declare -a x=([1]="2" [2]="3")

The other thing you need to watch out for is pathname expansion.  x[0]
is a valid glob, and if you have a file named x0 in the current directory,
an unquoted x[0] will be expanded to x0, and then you'll be running
"unset x0" instead of what you intended.

Use quotes the way I showed in my example to avoid that issue.

Re: Question about case statement in Bash docs

[Greg Wooledge]

On Mon, May 10, 2021 at 09:12:33PM +1000, AlvinSeville7cf wrote:
>x=test
>case $x in
>"test") echo Y
>esac
> 
>Pattern is quoted but no quote removal is performed according to docs.

Quote removal is essential, because of the way the empty string
is matched:

case $foo in
  "") echo "foo is empty";;
  ...

So, it may be a documentation omission, or maybe it's mentioned in some
other paragraph, but either way quote removal definitely happens here.

Re: 'command' builtin doesn't seem to work for external 'printf'

[Greg Wooledge]

On Mon, May 17, 2021 at 04:00:10PM -0700, Carl Thompson wrote:
> Nevermind! I was misunderstanding what the 'command' builtin does!

For the archives: 'command' only suppresses functions.  It doesn't
suppress shell builtins.

The typical use is when writing a wrapper function with the same name
as the command it's wrapping.

git() { LESS='-R -X' command git "$@"; }

Re: Sort command doesn't sort '@' character correctly

[Greg Wooledge]

On Thu, May 20, 2021 at 04:43:49PM +, Michael Jensen wrote:
> Repeat-By:
> 
> echo "xxaxxon" > test.txt
> echo "@zorg" >> test.txt
> echo "@jupiterlander" >> test.txt
> cat test.txt | sort
> 
> Note it prints out:
> 
> @jupiterlander
> xxaxxon
> @zorg
> 
> Expected:
> 
> @jupiterlander
> @zorg
> xxaxxon

The sort command is not part of bash; it's part of your operating system.
If you believe it's misbehaving for your locale, you should file a bug
report with your operating system vendor.

However, it's possible that this is "working as intended" for your locale,
which is a thing that -- once again -- your operating system vendor has
decided.  The behaviors of locales tend to be pretty arbitrary and
surprising.

If you want strict ASCII-based sorting, with no characters ignored, then
you should use the "POSIX" or "C" locale setting.  For example:

unicorn:~$ printf '%s\n' xyz @zorg @jupiter | sort
@jupiter
xyz
@zorg
unicorn:~$ printf '%s\n' xyz @zorg @jupiter | LC_COLLATE=POSIX sort
@jupiter
@zorg
xyz

Hope this helps.

Re: [patch #10070] toggle invert flag when reading `!'

[Greg Wooledge]

On Thu, May 27, 2021 at 05:46:41PM +0200, Alex fxmbsw7 Ratchev wrote:
> why doesnt it accept ! after |

Because ! negates a pipeline, not a simple command inside a pipeline.

Re: declare -g -x in function does not get exported in this script - not on ubunto 20.04 and centos 8

[Greg Wooledge]

On Wed, Jun 02, 2021 at 03:54:34PM -0600, v...@it4us.top wrote:
> --first script where declare -x -g works
> bar(){
> 
>   [[ $cnt == 1 ]] && declare -g -x nestedbug="not a nested variable-bug 
> !!"
> }
> 
> foo(){
>   [[ $cnt == 1 ]] && bar && echo ${nestedbug:?"variable should be  shown"}
> }
> foo

In this script, you're calling foo in the main shell process, and foo
calls bar, which sets the variable.  All of this takes place within
a single process.

> --begin of not working script 
> 
> doOptions(){
>   declare -g -x nestedbug="NOT BUG"
>   return 0 
> }
> doRun(){
> run=$( doOptions $* )

... and here, you are calling doOptions inside a subshell.

The variables that you create within the subshell vanish when the
subshell does.  Even if you export them.

In the future, please try to create minimal examples that reproduce
the problem.  I had to delete a *lot* of extraneous code.

Re: Prefer non-gender specific pronouns

[Greg Wooledge]

On Sun, Jun 06, 2021 at 05:12:21PM +0300, Oğuz wrote:
> If that really is a problem that has to be addressed and not
> bike-shedding, let's compromise and say "his/her" instead of "his" or
> "their".

*sigh*

I probably shouldn't do this, but let's dive into this just a bit, because
apparently it's too late to turn back now.

As background information, I was born in 1970 and was educated in the
public school system in the United States.  I supply this background
information because how one responds to this discussion depends greatly
on WHEN and WHERE one learned English grammar.  That's why there is such
incredible divisiveness on this topic.

In the grammar classes that I took in school, we were taught that "they"
and "their" are plural pronouns, and should always be used as such.  I
believe at least one other person on this list was taught a similar thing,
and has not updated his or her standards.  We all know the person to whom
I'm referring.  It's the really loud, angry one.

Now, I'm not an expert on the entire history of English grammar education
around the world, but it's my understanding that this ironclad insistence
on the plural-ness of "they" was prevalent mostly in the United States,
in the mid to late 20th century.  Thus, it was how many of us were raised,
but not all of us.

The issue, just for the record, is the lack of an acceptable gender-neutral
singular pronoun set in the current English language.  There are a few
choices, all of them bad:

1) "It".  This is considered offensive, because it implies a lesser status
   (an inanimate object rather than a person).

2) "Him or her".  This is awkward and long, and nobody likes it.

3) "Sie" and other non-English words.  These were suggested in the 1990s
but never caught on.

4) "They".  Some people strongly dislike this because it conflicts with the
   grammar rules they were taught.

5) "Him".  One of the older style guidelines suggested that when a person's
   gender is unknown, one should default to the masculine pronoun set, and
   the reader will understand that this is being used as a placeholder.

So, English gives us 5 choices, and they all suck.

Bash's documentation is using choice number 5, which was (as I understand
it) commonly taught prior to the mid 20th century.  It's basically 19th
century usage, and reflects and reinforces the patriarchal societies of
that time period.  It's quite understandable that someone would dislike
this choice strongly enough to offer a patch to undo it.

Out of these 5 choices, the one that seems to suck the *least*, according
to observed usage patterns in current written and spoken English, is
"they".

Personally, I've been trying to update my usage to embrace "they".
It's not easy, and sometimes I forget and fall back into older patterns,
but at least I'm trying.  This is the way the language is evolving.
If you refuses to embrace it, you will be left behind.  (Unless of course
a massive popular usage shift occurs and some other choice becomes the
new front-runner.  I've seen no signs of this happening.)

Re: When searching in command history, highlighting is missing the first character

[Greg Wooledge]

On Sun, Jun 20, 2021 at 07:51:03PM +0300, Teemu Leisti wrote:
> Repeat-By:
> 1. Open Bash, and command: abcdef
> 2. Click Ctrl-R, and type: bcde
> Expected behavior: The command "abcdef" is shown on the command line,
> and "bcde" is highlighted.
> Observed behavior: The command "abcdef" is shown on the command line,
> and "cde" is highlighted.

I believe what you're seeing is the result of the cursor still being
on the "a".  The cursor in my terminal (urxvt) is simply a reverse-video
filter, exactly the same as the "highlighting" that you're getting from
the reverse-i-search.  The two reverses cancel each other out.

If I move the mouse pointer out of the terminal in question and into
a different window, the textual cursor goes away, and I can see the
reverse-video highlighting on all the selected characters, including the
"a".

Your results may differ, depending on the type of terminal emulator you're
using, and the window manager and its focus paradigm (mine is fvwm, set
to use focus follows mouse).  You might need to click a different window
to transfer focus away from the terminal in which you're doing the
reverse-i-search.

Re: 'wait' in command group in pipeline fails to recognize child process

[Greg Wooledge]

On Tue, Jun 22, 2021 at 02:42:40AM -0700, Martin Jambon wrote:
> I ran into something that looks like a bug to me, although I'm not super
> familiar curly-brace command groups.
> 
> Bash version: latest from GitHub mirror (commit ce23728: Bash-5.1 patch 7)
> 
> Minimal repro:
> 
>   $ sleep 1 & { wait $!; } | cat
>   [1] 665454
>   bash: wait: pid 665454 is not a child of this shell
> 
> I was expecting a success, just like we get without the pipeline:

A pipeline creates two or more subshells, one for each command in the
pipeline.  Therefore, your wait command is running in a different
process than the one which created the sleep background job.

The curly braces are irrelevant here.

unicorn:~$ sleep 1 & wait "$!"|cat
[1] 1290127
bash: wait: pid 1290127 is not a child of this shell

Re: 'wait' in command group in pipeline fails to recognize child process

[Greg Wooledge]

On Tue, Jun 22, 2021 at 01:12:10PM -0700, Martin Jambon wrote:
> On 6/22/21 4:31 AM, Greg Wooledge wrote:
> > A pipeline creates two or more subshells, one for each command in the
> > pipeline.  Therefore, your wait command is running in a different
> > process than the one which created the sleep background job.
> > 
> > The curly braces are irrelevant here.
> > 
> > unicorn:~$ sleep 1 & wait "$!"|cat
> > [1] 1290127
> > bash: wait: pid 1290127 is not a child of this shell
> 
> Thank you! Now I know that a subshell is not a shell, $$ being the process
> ID of the shell, not of the subshell.

It's a forked child process of bash, and as such, it's still a shell.
Specifically, it's bash.

There's some formal definition that you can ignore.  The real definition
of a subshell is "the direct result of a fork()".  It inherits all of
the shell variables, functions, aliases, and so on that the parent
shell process had at the time of the fork.

This is different from a direct call to "bash", which would not inherit
shell variables and so on -- only environment variables.

In the pipeline, there are two subshells created.  One of them runs
the command wait "$!", and the other runs the command cat.

The special parameter $! is inherited from the parent shell, and contains
the PID of the sleep process that the parent shell created.  But the
child subshell can't wait for that process, because it belongs to someone
else.  Hence the error message.

Re: 'wait' in command group in pipeline fails to recognize child process

[Greg Wooledge]

On Tue, Jun 22, 2021 at 06:54:18PM -0700, Martin Jambon wrote:
> This is confirmed by this definition from posix:
> 
> A subshell environment shall be created as a duplicate of the shell
> environment, except [...]

> In the posix definition, a subshell
> - is not necessarily implemented as a separate process
> - belongs to a unique shell
> - is not a shell

I have no idea how you came to the conclusion that it's "not a shell".

It's a DUPLICATE OF A SHELL, and therefore a shell.

The point is that it's a DIFFERENT shell.  Not the original one.

I mentioned earlier that there's a formal definition you can largely
ignore, and just go with "a subshell is the direct result of a fork()".[1]
That's all you really need to know.  The formal language exists to allow
for implementations in some theoretical non-Unix-like environment where
fork() is not a thing.  It's simply describing what a fork() does, without
using that term.

[1] Except for that bit about resetting signal handlers.

Re: 'wait' in command group in pipeline fails to recognize child process

[Greg Wooledge]

On Wed, Jun 23, 2021 at 03:17:48PM -0700, Martin Jambon wrote:
> > What is the magic quality that imparts shellness?
> 
> '$$' matching the process ID.

That just signifies that you're in the original shell, or the main shell,
or the root shell, or the fundamental shell, or make up whatever term
you like for it.

Every bash (or sh) process is a shell, regardless of whether it's a
subshell or not.  That includes subshells, and non-sub-shells.

Here's a brief demonstration: I have two terminals opened.  The first
one is pts/3:

unicorn:~$ ps
PID TTY  TIME CMD
966 pts/300:00:00 bash
1363991 pts/300:00:00 ps
unicorn:~$ (sleep 10; true)

In the second terminal, while the sleep is running:

unicorn:~$ ps f -t pts/3
PID TTY  STAT   TIME COMMAND
966 pts/3Ss 0:00 bash
1364016 pts/3S+ 0:00  \_ bash
1364017 pts/3S+ 0:00  \_ sleep 10

Process 966 is the interactive bash shell, which is not a subshell.  It's
the other kind of shell.

Process 1364016 is the subshell created by the (parentheses) to run the
command which was specified inside them.

They're both instances of bash.  They're both shells.

> I'm not sure why I have to fight over this. It's clearly my
> misunderstanding. That's why I suggest clarifications in the documentation,
> if you're interested in creating a better experience for users like me.

Nobody knows what the hell to CALL it.  That's the problem.  It doesn't
have a clearly defined name, because nobody ever NEEDED one.

Re: Word splitting for $@ in variable assignment

[Greg Wooledge]

On Thu, Jun 24, 2021 at 05:52:47PM +1000, Alvin Seville wrote:
>  Hello! I want to understand why the following code doesn't produce any
> error:
> set -- "a b" "c"
> a="$@"

It doesn't produce an error because the designers of the shell tried
to make it "user-friendly" by having it guess what you really want,
whenever you do silly things.  Sometimes it guesses correctly.  Sometimes
not.

In this case, you've written code that makes no sense to an experienced
programmer, because on the surface you're assigning a vector (list, array)
value to a scalar (string) variable.

But the shell decided it would try to "help" you by assuming you really
meant a="$*".  Except, it's not actually that simple, and it just gets
more and more ugly the deeper you dive into it.

> ? I expected smth like: main.sh: line 2: b: command not found due to word
> splitting according to documentation
> .
> What am I missing?

Word splitting does not occur on the right hand side of a string variable
assignment.  That's why you can do this:

a='two words'
b=$a

This is well-defined behavior, completely portable across every Bourne-type
shell ever written.

The difference between this and your case (a="$@") is that "$@" doesn't
actually involve word splitting.  It's like... the opposite of word
splitting.  Word retention, maybe?  It doesn't have a name.

But it doesn't make sense for a retained list of words to be assigned to
a string variable.  There has to be a single string that gets assigned.
So the shell tries to come up with a result that it thinks will make
sense.

What it ends up doing is taking the list of words generated by "$@" and
concatenating them all into a single string with spaces between them.
It does NOT use the contents of IFS -- it simply chooses "space" as
the separator.

unicorn:~$ bash
unicorn:~$ IFS=x
unicorn:~$ set -- "a b" c
unicorn:~$ a="$*"; echo "$a"
a bxc
unicorn:~$ a="$@"; echo "$a"
a b c

This surprising leniency is one of the reasons there are so many terrible
shell scripts (and shell script writers) in the world.  The shell lets you
do many ridiculous and even dangerous things, with no warnings of any kind.

Re: function def crash

[Greg Wooledge]

On Thu, Jun 24, 2021 at 09:48:59AM +0200, Phi Debian wrote:
> $ function echo { \echo the function version; \echo "$@"; }

For the record, this isn't how you write a function that wraps a builtin.
Use the "builtin" command instead.

echo() {
  printf 'echo() wrapper invoked.\n'
  builtin echo "$@"
}

Or, if you aren't sure whether the command you're wrapping is a builtin
or a program, use "command":

echo() {
  printf 'echo() wrapper invoked.\n'
  command echo "$@"
}

The backslash hack only stops aliases from being used.  It doesn't stop
your function from calling itself recursively.

Re: Word splitting for $@ in variable assignment

[Greg Wooledge]

On Fri, Jun 25, 2021 at 01:47:01AM +0200, Nora Platiel wrote:
> To me, "$@" expanding to multiple words would mean that:
> 
> $ var="$@" foo
> 
> for $# > 0, behaves the same as:
> 
> $ var="$1" "${@:2}" foo
> 
> which is obviously not the case.

"$@" expands like "$1 "$2" ... when used in most contexts.  For example,

foo --bar "$@"

passes the script's arguments along to foo, exactly as received, with
the extra --bar argument in front of them.

Likewise,

args=("$@")

stores the script's arguments as the elements of an indexed array.  Each
argument is retained as a separate word, becoming one element of the
new array.

var="$@" is simply a special case.  It's an assignment *first*, and it's
parsed as such.

The assignment-ness of the command overrides everything else.  It
completely changes how the "$@" expansion occurs.  Now, instead of
expanding like "$1" "$2" ... it expands like "$1 $2 ...".  It's
completely unique.  "$@" does not act like that in any other context.

Bash is ALL about these special cases.  If you don't like it, don't write
code that uses it.  In any sensible programming language, var="$@"
would have been an error.  In bash, it's not.  But that doesn't mean
you have to *write* it.

Re: Word splitting for $@ in variable assignment

[Greg Wooledge]

On Fri, Jun 25, 2021 at 03:15:25PM +0200, Nora Platiel wrote:
> But at least it is documented, and that part of the doc is clear to me.
> The part of the doc that I'm complaining about is this one:
> 
> | A variable may be assigned to by a statement of the form
> | name=[value]
> | [...]
> | Word splitting is not performed, with the exception of "$@" as
> | explained below.

This isn't the full sentence in the current man page.  In the bash 5.1
man page, it says:

   Word splitting is not
   performed, with the exception of "$@" as explained below under  Special
   Parameters.

So, we need to look at Special Parameters:

   @  Expands to the positional parameters,  starting  from  one.   In
  contexts  where  word  splitting is performed, this expands each
  positional parameter to a separate word; if  not  within  double
  quotes,  these words are subject to word splitting.  In contexts
  where word splitting is not performed, this expands to a  single
  word  with each positional parameter separated by a space.

That matches the behavior that I saw (and pasted on this mailing list the
other day).  (Which by the way is *not* the same as "$*" unless IFS happens
to be unset or to begin with a space.)

If Chet feels that a change is needed here, I would remove the "with
the exception of" clause entirely.  Just say "Word splitting is not
performed."

Re: The cursor moves abnormally

[Greg Wooledge]

On Sun, Jun 27, 2021 at 12:19:00PM +, cheng wensui wrote:
> bash  5.1.4(1)-release (x86_64-pc-linux-gnu) 
> 
> when ps1 is too long  ,The cursor moves abnormally  ,Entering letters too 
> many  too long is also abnormal 
> 
> example :
> PS1='\033[5;37m[\033[0m\u\033[1;35m@\033[0m\033[4m\H\033[0m\033[1;31m(\t)\033[0m\W
>  \033[0;33m\#/\!\033[0m\033[5;37m]\033[0m\033[0;37m\$ '

You're missing the \[ \] around your escape sequences.

See  for more details.

Re: simple prob?

[Greg Wooledge]

On Tue, Jun 29, 2021 at 01:21:44PM -0700, L A Walsh wrote:
> I hope a basic question isn't too offtopic.

More of a help-bash question, in my opinion, but close enough.

> Say I have some number of jobs running:
> 
> >  jobs|wc -l
> 3

OK.

> Would like to pass a varname to njobs to store the answer in, like:
> >  njobs n
> echo "$n"
> 3

"How do I pass a value back from a function to its caller, you know,
like a function in any other programming language can do" is one of the
holy grails of shell programming.  There is no sane answer.  You appear
to be going down the "pass a variable name by reference" path, so:

unicorn:~$ jobs | wc -l
3
unicorn:~$ njobs() { local _n=$(jobs | wc -l); eval "$1=\$_n"; }
unicorn:~$ njobs walsh
unicorn:~$ echo "$walsh"
3

Now you just need to add sanity-checking on the argument of njobs, to
avoid whatever code injection the malicious caller wants to perform.

And maybe consider adding "make sure $1 isn't _n" to the list of your
sanity checks, since the caller can't be permitted to use that particular
variable name.

unicorn:~$ njobs _n
unicorn:~$ echo "$_n"


(At least it didn't go into an infinite loop!)

Re: simple prob?

[Greg Wooledge]

On Tue, Jun 29, 2021 at 09:47:30PM +0100, Kerin Millar wrote:
> On Tue, 29 Jun 2021 16:35:28 -0400
> Greg Wooledge  wrote:
> 
> > unicorn:~$ njobs() { local _n=$(jobs | wc -l); eval "$1=\$_n"; }
> > unicorn:~$ njobs walsh
> > unicorn:~$ echo "$walsh"
> > 3
> > 
> > Now you just need to add sanity-checking on the argument of njobs, to
> > avoid whatever code injection the malicious caller wants to perform.
> 
> I can't fathom the switch to eval there. Why not printf -v "$1" %s "$_n", for 
> example? It even rejects invalid identifiers.

declare, printf -v, local -n, eval -- they're mostly equivalent.  Some
of them may prevent *some* possible code injections, but none of them
prevent *all* possible code injections.

unicorn:~$ njobs2() { printf -v "$1" %s 42; }
unicorn:~$ njobs2 'x[0$(date >&2)]'
Tue Jun 29 17:00:29 EDT 2021

No matter which one of these you choose, you still have to sanity-check
the input.  Or else declare that you do not care if the user shoots their
own foot off (which is a valid stance as long as your code is never used
in a context where the user can elevate their privileges/capabilites).

Re: simple prob?

[Greg Wooledge]

On Tue, Jun 29, 2021 at 02:05:46PM -0700, L A Walsh wrote:
>That would be 'me', so I'm going to rule out malicious
> code injection! :-), but that's also why I used printf to
> write the output, the worst that could happen is some varname
> is overwritten with the answer, no?

Sadly, this is not correct.  You can still get code injections with
printf -v.

unicorn:~$ njobs2() { printf -v "$1" %s 42; }
unicorn:~$ njobs2 'x[0$(date >&2)]'
Tue Jun 29 17:00:29 EDT 2021

> Simpler -- don't use a variable:
> 
>   njobs() { printf ${1:+-v $1} "%s\n" "$(jobs |wc -l)"; }

You're adding a newline to the variable.  That may not be what you want.

> So...hmmm...how is it that jobs picks up the right answer in what
> would seem to be a subshell?  Special cased?

The subshell inherits the job list from its parent -- it just can't
*wait* for any of those jobs, since it isn't their parent.  But it can
still count them.

Re: simple prob?

[Greg Wooledge]

On Tue, Jun 29, 2021 at 02:58:28PM -0700, L A Walsh wrote:
> njobs() { printf ${1:+-v $1} "%s\n" "$(jobs |wc -l)"; }
> 
> Using that with your input:
> 
> njobs 'x[0$(date >&2)]'
> 
> bash: printf: `x[0$(date': not a valid identifier

This is because you didn't quote "$1".  Since you only ever tested
the cases where $1 was a valid variable name, you never ran into that
particular result... until now.

As you can see, the unquoted $1 underwent word splitting, so you're
effectively running printf -v 'x[0$(date' '>&2)]' '%s\n' "...".

This won't protect against all code injections, of course; only the
ones that contain a whitespace character.

Re: simple prob?

[Greg Wooledge]

On Tue, Jun 29, 2021 at 04:29:05PM -0700, L A Walsh wrote:
> > > njobs() { printf ${1:+-v $1} "%s\n" "$(jobs |wc -l)"; }

>Which is detected as "illegal input" and disallowed.  If you don't enable
> some security errors, they can't be as easily introduced.

Are you *still* insisting that your failure to quote is a SECURITY
FEATURE?

Come *on*!

unicorn:~$ njobs() { printf ${1:+-v $1} "%s\n" "$(jobs |wc -l)"; }
unicorn:~$ njobs 'x[0$(date>&2)]'
Tue Jun 29 19:49:16 EDT 2021

All I had to do was remove the space.  You're not even trying.

Your failure to quote is simply a failure.  If you want to prevent
code injection attacks, you need to sanity-check the input.

There is no other way.

Re: simple prob?

[Greg Wooledge]

On Tue, Jun 29, 2021 at 10:11:31PM -0400, Eli Schwartz wrote:
> On 6/29/21 7:29 PM, L A Walsh wrote:
> >>> njobs() { printf ${1:+-v $1} "%s\n" "$(jobs |wc -l)"; }

> It also rejects this perfectly reasonable code, which I arbitrarily
> decided not to support because if you want this you're weird:
> 
> $ declare -A jobs
> $ njobs 'jobs[first run]'
> error: invalid characters [ ] found in var 'jobs[first run]'
> hacker detected...

It's not the weirdest thing I've seen.  Without your validation:

unicorn:~$ declare -A jobs
unicorn:~$ njobs 'jobs[first]'

All cool, right?  Uh

unicorn:~$ touch jobss
unicorn:~$ njobs 'jobs[second]'
unicorn:~$ declare -p jobs
declare -A jobs=([first]=$'0\n' )
unicorn:~$ declare -p jobss
declare -- jobss="0
"

Failure to quote strikes again.  What if we fix it?

unicorn:~$ nqjobs() { printf ${1:+-v "$1"} "%s\n" "$(jobs |wc -l)"; }
unicorn:~$ nqjobs 'jobs[third]'
unicorn:~$ nqjobs 'jobs[sixth]'
unicorn:~$ declare -p jobs
declare -A jobs=([sixth]=$'0\n' [first]=$'0\n' [third]=$'0\n' )

Now it works even if there's a file which matches the glob pattern.

> Well, the diagnostic jumped to a conclusion that is probably false.

Your validation is too strict in this case.  Right idea, wrong
implementation.  Of course, one might argue that allowing an array with
an index is a dangerous precedent, and stick with your strict validation
on principle.  That's a design decision, and I could see either one
being correct depending on the goals.

Oh, and by the way, there's at least one more category of errors that
failure-to-quote is susceptible to:

unicorn:~$ unset foo bar
unicorn:~$ njobs 'foo -v bar'
unicorn:~$ declare -p foo bar
bash: declare: foo: not found
declare -- bar="0
"

Without quotes, we get printf -v foo -v bar '%s\n' "$(jobs|wc -l)"
with the result shown.

With quotes, we get:

unicorn:~$ nqjobs 'foo -v bar'
bash: printf: `foo -v bar': not a valid identifier

It's not robust against code injection attacks (we need actual validation
for that), but at least it gives a sensible error message in some common
cases.

tl;dr: When in doubt, quote it.  There is a REASON we beat this into
people's heads.

Re: simple prob made into a security tragedy...oh well.... ;^/

[Greg Wooledge]

On Thu, Jul 01, 2021 at 02:12:10AM -0700, L A Walsh wrote:
> What are you talking about?
> 
>  njobs() { printf ${1:+-v $1} "%s\n" "$(jobs |wc -l)"; }
> 
> I don't see any arrays, let alone indexed.

The arrays were in the calls.  We demonstrated a few cases already.
Since you clearly weren't paying attention, here they are again.

First, the classic code injection:

unicorn:~$ njobs 'x[0$(date>&2)]'
Thu Jul  1 07:12:35 EDT 2021

Second, a user who wants to (legitimately) use an existing array, with
a space in the key:

unicorn:~$ declare -A jobs
unicorn:~$ njobs 'jobs[first time]'
bash: printf: `jobs[first': not a valid identifier

Third, a user who wants to (legitimately) use an existing array, where
the indexed reference is accidentally treated as a glob:

unicorn:~$ touch jobsx
unicorn:~$ njobs 'jobs[sixth]'
unicorn:~$ declare -p jobs jobsx
declare -A jobs
declare -- jobsx="0
"

Fourth, a user who accidentally or maliciously passes additional printf
options:

unicorn:~$ njobs 'foo -v bar'
unicorn:~$ declare -p foo bar
bash: declare: foo: not found
declare -- bar="0
"

Quoting "$1" in your code would stop three of these four examples from
doing the wrong thing.  It makes no difference in the first case.


Looking at the Subject: header reminds me of:

  43. A "quick" or "simple" question will be neither.

What you perceive as a "simple prob" (and for the love of glob, can you at
least spell out "problem" correctly?!) is absolutely NOT simple.  It's the
tip of an exploding carcinogenic iceberg full of frozen soul-eating
demons.  Everyone knows about this iceberg.  There are buoys all around
it warning sailors to stay away.  All the navigational maps have marked
this territory as too dangerous.

Despite this, you've decided it would be a great idea to have a little
picnic on top of it.

And people don't understand why we're so cranky.

Re: simple prob?

[Greg Wooledge]

On Fri, Jul 02, 2021 at 09:09:34AM +0200, Phi Debian wrote:
> PW$ function njobs
> > { [ "$1" != "n" ] && typeset -n n="$1"
> >typeset -a t ; readarray t <<<"$(jobs)" ;n=${#t[@]}
> > }

<<<$() is a poor imitation of < <() which is what you really want.

readarray -t t < <(jobs); n=${#t[@]}

Combining <<< and $() only gives an approximation of the same result
(the command substitution strips off all trailing newline characters,
and then the here-string syntax appends one newline), and it's less
efficient, because it involves slurping the entire input into memory,
then writing it out to a temporary file, then opening the temporary
file for input and unlinking it, then reading it back in a second time.

Using < <() avoids the newline alterations and the temporary file.

(Note: in some sufficiently new versions of bash, there may not be a
temporary file in some cases.  But it's still not the best solution,
as it still involves storing and reading the whole output multiple times.)


Stepping back a moment, you're using the name reference version of the
"pass an output variable by reference" strategy.  This requires bash 4.3,
which is reasonable, and it requires some additional sanity checks which
you did not show.

What's really interesting to me is that you did a *partial* sanity check,
refusing to create a circular name reference if the user passed "n"
as their output variable name.  But you forgot to check for "t", which is
another local variable you're using.  Also, in Linda's original example,
the output variable was literally named "n", so choosing that as your
name reference and explicitly disallowing it is a really spiteful choice.

Finally, you didn't do any sanity checking of the output variable name
(beyond comparing it to one of your two local variable names), so your
function is susceptible to the same code injection attacks we discussed
earlier in the thread.

unicorn:~$ njobs_ref() { typeset -n n="$1"; n=42; }
unicorn:~$ njobs_ref 'x[0$(date>&2)]'
Fri Jul  2 07:54:49 EDT 2021

As I mentioned a few days ago, all variants of the "pass a variable name
by reference" method are basically equivalent to each other, and all
of them need input sanity checking in order to avoid code injections.
(Some of the variants avoid *some* flavors of code injection, but none of
them avoid this one.)

Re: simple prob?

[Greg Wooledge]

On Fri, Jul 02, 2021 at 05:45:23PM +0200, Phi Debian wrote:
> Regarding the code injection I am not sure I got it.
> 
> If you are sitting at a prompt, why would you trick
> 
> unicorn:~$ njobs_ref 'x[0$(date>&2)]'
> 
> when you could simply type
> unicorn:~$ date
> 
> I assume protected script/source (the ones you can't write into), are wise
> enough not to run command based on user input, in short I guess no
> protected script are doing thing like read in; eval $in :) that is the
> simplest code injection :) and then would never let you have a chance to
> enter 'x[0$(date>&2)]' at any time.

For functions that you've written exclusively for personal use, it's
not an immediate concern.  It's more of a thing that you want to be
aware of for the future.

Where it becomes important is when you're writing scripts for other
people to use, or which run as different user accounts, or with
different privileges.

The classic example of this is a script that's run by a web server in a
CGI environment, which accepts query parameters from the end user.  If
one of those query parameters is used in an unsafe way, it can execute
undesired commands on the web server.

Of course, there are *many* other places that shell scripts are used,
such as booting an operating system, starting various services, and
so on.  In some of these cases, there is no external input being read,
or the external inputs are "trusted" files owned and edited only by
the system admin (root).  But in other cases, untrusted input may be
read.

So, there's merit in adopting a proactive strategy to shell script
security.  Maintaining a slightly paranoid mindset can help you spot
potential security holes and possibly avoid disasters.

Re: parameter expansion with `:` does not work

[Greg Wooledge]

On Wed, Jul 07, 2021 at 11:50:01PM +0200, lisa-as...@perso.be wrote:
> Have noticed that parameter expansion with `:` does not work
> 
> : ${fltype:-"texi,org"}  # alternative to `fltype=`

What did it do?  What did you *expect* it to do?

This looks like it's related to your ongoing project to make your
simple script as complex as possible.  As such, it appears you're
digging into the bowels of the Bourne shell and trying to dig up
every obscure piece of syntax you can find, so that you can use
them all, to be completely sure you can't possibly read the script
a week from now.

I advise you to stop doing this.

In this particular instance, you're assigning a LIST of two values
to a STRING variable, using a comma as a delimiter between them.
Presumably you're going to split this list apart later on.

Wouldn't it be a whole lot simpler to store the list in an ARRAY
variable in the first place, and avoid the need to split it?

Re: parameter expansion with `:` does not work

[Greg Wooledge]

On Thu, Jul 08, 2021 at 12:10:27AM +0200, lisa-as...@perso.be wrote:
> The line 
> : ${fltype:-"texi,org"}  
> should be an alternative to 
> fltype=${fltype:-"texi,org"}  

As Chet guessed earlier, you probably meant := instead of :- .

I started typing something to that effect in my first response, but then
I noticed the payload you were "assigning" (though you messed up the
syntax) and decided to try to teach you to fish, rather than tossing
you the decrepit, smelly old fish you had dropped in the dirt.

You are using horrible syntax to do the *wrong thing*.

Lists should be stored in indexed array variables, not in strings with
silly delimiters.

Initialize an array variable holding the default list of extensions at
the beginning of your program.  Then, in your option-processing code,
if the user supplies their own delimited list of extensions, split it
apart and use the resulting list to overwrite the array variable.

That's it.  No need for 40-year-old crappy syntax.  No need to store
your default list as a delimited string and then spend system resources
splitting it apart.  You only need to expend those resources if the user
supplies a delimited list, and the user only needs to do that because of
the crappy design choices you've made.

But since the user is you, if that's how you *really* want the program
to work, then so be it.  Just don't make it even worse than it needs to
be.

Re: parameter expansion with `:` does not work

[Greg Wooledge]

On Thu, Jul 08, 2021 at 12:37:07AM +0200, lisa-as...@perso.be wrote:

> >From: Dennis Williamson 
> >$ : ${foo:-bar}
> >$ : ${foo:=bar}
> >The first form is a substitution and the second form is an assignment.

> So you used `:` at the beginning and it worked?

Are you incapable of seeing the difference between the two lines?

I weep for the human race.  There's no telling what you're even seeing
on your monitor (assuming you can see, and aren't using a screen reader).
You're clearly using some mail user agent that works like Microsoft Outlook,
and there's no guessing what it's doing to the messages you receive.

If you literally cannot see the difference between :- and := in the
above example, then all is lost, and there is nothing more we can say
to you.  Because you can't see it.

Re: parameter expansion with `:` does not work

[Greg Wooledge]

On Thu, Jul 08, 2021 at 12:33:04AM +0200, lisa-as...@perso.be wrote:
> Talking about neater design choices, you seem to imply you have some other
> way to take user options that are composed of lists, so that one does not use
> a delimited string. What is it?

I wrote a detailed message containing many different choices:

https://lists.gnu.org/archive/html/help-bash/2021-07/msg00019.html

Re: parameter expansion with `:` does not work

[Greg Wooledge]

> > fdir=${fdir:=$PWD}
> 
> Ack!

I know.  I simply gave up.  At least this will "work", even if it's
completely silly.

Re: parameter expansion with `:` does not work

[Greg Wooledge]

On Thu, Jul 08, 2021 at 04:38:25AM +0200, lisa-as...@perso.be wrote:
> I'd rather understand what's going on, rather than simply never use it.

OK.  Let's start from the beginning.

In sh and bash, anything that begins with a $ is potentially a
substitution, a.k.a. an expansion.  The parser will try to unravel
the punctuation soup to figure out where the beginning and ending of
the substitution are.  Then, that piece of the command will be replaced
with zero or more words.  All of it is dependent on context, and rules
that have grown organically and chaotically over a span of decades.

Let's say you have a command like this:

foo $bar

The thing on the right hand side, which begins with $ and ends with r,
is a substitution.  Specifically, it's a parameter expansion, which means
a "parameter" (which is either a variable or a special parameter, in this
case a variable) will have its value pulled from memory and used for the
substitution.

If we have a variable named bar, its value gets substituted into the
command.  Since $bar is not quoted, that value undergoes two more rounds
of substitutions (word splitting, and filename expansion).  At the end
of all that action, we will have a list of zero or more words, and these
words will become the arguments of the foo command.

OK so far?  Good.

Now let's say we have this command:

: $bar

Once again, the value of the variable named bar is substituted, and then
undergoes word splitting and filename expansion, and the results of that
become the arguments to the : command.

The : command does nothing, so the end result of all this work is ...
nothing.  Except that we may hit the file system a few times in order to
perform filename expansions, if there are any glob characters in the
variable's value.  But that's it.

So, why would anyone write a command like this?  It's because some
substitutions have side effects.

Let's look at this command next:

: $((x++))

Now, this is a silly command, and you wouldn't write this in real life,
because it's just more complex than it needs to be.  But I'm demonstrating
something, so stick with it for a moment, please.

This time, we don't have a parameter expansion.  We have an arithmetic
substitution instead.

The stuff inside the $(( )) gets passed to a special arithmetic parser,
which has its own special rules.  These rules look a lot like the rules
of the C language, by some strange coincidence.

This particular arithmetic expression x++ uses the post-increment operator
++ to add 1 to the value of an existing (or even nonexistent) variable.
This is a *side effect*, meaning that it does something more than just
producing a value for our substitution.  It has some lasting effect.

So, what happens here?  In order:

1) The value of x is pulled from memory and stored in a temporary spot.
   If x doesn't exist, we use the value 0.  If x contains a string that
   can be treated as an integer, we use that value.  Otherwise, we attempt
   to perform recursive arithmetic evaluation.  We won't cover all of that
   right now.

2) We take the value from step 1, add 1 to it, and store this back into x.

3) The value from step 1 (before we added 1) is used as the value of the
   substitution.

4) The value of the substitution would undergo word splitting and filename
   expansion because of the lack of quotes, except that the result of
   an arithmetic expansion is always an integer, and therefore can't do
   those things.

5) The value of the substitution is used as the argument of the : command,
   which does nothing.

So, the whole point of this demonstration was what happens in step 2.  The
value of x is changed, even though we used a command that normally does
nothing.  The change happens *during* the expansion.  It's independent of
the command that we used.

The only reason we have the : command here at all, is so that we don't
try to execute the value of x as a command.  If we left out the : we
would get something like

bash: 0: command not found

We don't want that.  So that's why the : is there.

Now that you understand everything that's going on, let's look at this
crazy shit from 1977 that you've fallen in love with:

: ${foo:=bar}

What happens here?  We have a parameter expansion with a special modifier.
According to the documentation which has been quoted at you multiple
times already, this is a two-step expansion.  The value of the variable
named foo is pulled from memory.  If this value is the empty string, or
if the variable foo does not currently exist, then an *assignment* takes
place, and the string bar is stored in the variable foo, and then that
string (bar) also becomes the value of the substitution.

So, we perform the following steps:

1) The value of the variable foo is pulled from memory.

2) If the value from step 1 is the empty string (or if there's no variable
   named foo yet), the string bar is *assigned* to the variable foo, and
   the string bar also becomes the value that we pulled from memory.

3) The str

Re: Unexpected behaviour when using process substitution with stdout and stderr

[Greg Wooledge]

On Sun, Jul 11, 2021 at 11:09:10AM +0100, earnestly wrote:
> What appears to be happening is that the output from standard error is
> being mixed into the function handling standard out, even more
> surprisingly that xtrace output is also being consumed and filtered as
> well.

First, xtrace (set -x) output is written to stderr by default.  You can
change this by setting the BASH_XTRACEFD variable.  But in the absence
of that change, it goes to stderr, along with any other content that's
written to stderr, either by bash itself, or by commands that bash calls.

> stdout() {
> local line
> 
> while read -r line; do
> printf 'from stdout: %s\n' "$line"
> done
> }
> 
> stderr() {
> local line
> 
> while read -r line; do
> printf 'from stderr: %s\n' "$line"
> done
> }

> generate > >(stdout) 2> >(stderr)
> wait

You're sending the stderr of generate to a subshell where the stderr()
function is executed.  But the stderr() function writes to stdout.
Therefore, all of the output from generate is ultimately ending up going
to the script's stdout, with potentially weird interleaving depending
on the timing of the two background subshells.

If you want to keep them separate, you either need to change the stderr()
function to write to stderr, or change the >(stderr) call to >(stderr >&2).

Re: Crash on large brace expansion

[Greg Wooledge]

On Thu, Jul 15, 2021 at 05:28:04PM +0200, Léa Gris wrote:
> Le 15/07/2021 à 16:36, Gabríel Arthúr Pétursson écrivait :
> > Hi all,
> > 
> > Executing the following results in a fierce crash:
> > 
> > $ bash -c '{0..255}.{0..255}.{0..255}.{0..255}'
> 
> Brace expression expands all the value in memory.
> 
> Here you are actually telling Bash to expand 256⁴ or 4294967296 42 Billion
> entries.

4.2 billion, but who's counting? :-)

Re: gettext feature request

[Greg Wooledge]

On Sat, Jul 24, 2021 at 04:35:30PM +0200, Jean-Jacques Brucker wrote:
> Planning to use the /$"string/" feature in my bash code made me think too
> much : https://github.com/foopgp/bash-libs/tree/main/i18n

The security problem with $"..." is known.  Unfortunately, Chet seems
set on the idea that $"..." should perform parameter expansions and
command substitutions, simply because it looks like "...".

(Also, apparently people have been using it that way in real life as
well, which makes it even more complicated to change.)

That doesn't leave us with a clear path forward.  There's no way to get
a translated string without performing unwanted expansions afterward
on the translated value, simply because of the syntax characters that
were selected.

So, most people have simply written the whole thing off as a lost cause.
The best advice I can give is, "if you need localization in your program,
don't write it in bash".

> ...what I really *love* to see in bash, is a /$'string'/ feature, which
> doesn't parse any «`» or «$» characters.

$'...' already exists, so you can't use that for localization.

> Did you already discuss about that ? Is there a hope to use such powerful
> syntax without introducing security issue ?

https://mywiki.wooledge.org/BashFAQ/098
https://lists.gnu.org/archive/html/bug-bash/2009-02/msg00258.html
https://lists.gnu.org/archive/html/bug-bash/2009-03/msg0.html

I'm sure there are other past discussions, but good luck trying to
find them, when the $"..." feature has no official *name* and Google
won't let you search for $" as a key string.

Re: Infinite loop triggered by extglob +(!(x))y

[Greg Wooledge]

On Sun, Jul 25, 2021 at 07:09:50PM +0200, andrej--- via Bug reports for the GNU 
Bourne Again SHell wrote:
> Description:
>   An extglob like +(!(x))y causes Bash to freeze with a 100% CPU 
> utilization.
>   At the first glance the freeze doesn't seem to depend on the 
> (non-)existence of matching paths.

The files do seem to matter.

unicorn:~/tmp$ echo +(!(x))y
servers-to-try
unicorn:~/tmp$ cd /tmp
unicorn:/tmp$ echo +(!(x))y
^C^C^ZKilled

It's a really bad runaway, requiring SIGKILL from another terminal to
stop it.

Re: Vars not set when sourcing w/ abs path, set -a

[Greg Wooledge]

On Tue, Jul 27, 2021 at 11:42:02PM +, Van Klaveren, Brian N. wrote:
> [bash-test@227f958499c1 ~]$ cat sourceme
> #!/bin/bash
> FOO=bar
> FOO=bar2
> set -a
> env | grep FOO

You probably want to put set -a at the top, before the variables are
set.  The -a flag isn't retroactive.  It only applies to variables
that are set *while* it's turned on.

The typical ways that set -a is used are:

1) Around a block of variable assignments within a script, in order to
   avoid having to type "export" several times.

2) Right before sourcing (dotting in) a second file which contains only
   variable assignments, so that all of those variable assignments are
   copied to the environment, without having to modify the sourced file.

Re: Extended regex match (shmatch.c:115) malloc error

[Greg Wooledge]

On Mon, Aug 09, 2021 at 01:08:32PM -0400, Gyorgy Matyasfalvi wrote:
> Machine: ia64
> OS: hpux11.31
> Compiler: cc
> Compilation CFLAGS: -O -I/usr/local/include -D_XOPEN_SOURCE_EXTENDED +DD64
> uname output: HP-UX sovmh352 B.11.31 U ia64 1429105484 unlimited-user
> license
> Machine Type: ia64-hp-hpux11.31
> 
> Bash Version: 5.1
> Patch Level: 8
> Release Status: release

> malloc: shmatch.c:115: assertion botched
> free: start and end chunk sizes differ
> Aborting...ABORT instruction (core dumped)

I can't reproduce your results on Linux.  I suspect there's eomething
wrong with your build of bash, or possibly with your system's regex
functions (in libc).

Even on HP-UX, I still can't reproduce it.  Granted, my HP-UX systems are
PA-RISC, running older versions of HP-UX, and running bash 4.3, but still.

You also appear to be using bash's builtin malloc, rather than the
system's malloc from libc.  I'm not sure whether that's relevant here,
but it's noteworthy.

Re: GROUPS

[Greg Wooledge]

On Mon, Aug 09, 2021 at 05:35:56PM -0400, Franklin, Jason wrote:
> Should bash, invoked with "--posix" or as "sh", omit the special
> treatment of variables such as GROUPS?

I would say no.  Using all-caps variable names is a bad idea for
precisely this reason -- you never know when it'll be something
special, in your shell or in your operating system.

Always use at least one lower-case letter in each variable name, unless
you're actually exporting something as an environment variable, and
you know that the name is not used by anything else in your working
environment.

Re: GROUPS

[Greg Wooledge]

On Mon, Aug 09, 2021 at 10:00:25PM -0400, Franklin, Jason wrote:
> I did not write the scripts in question.  These were actually Debian
> package maintainer scripts that started failing.
> 
> Perhaps I'll get a test added to "checkbashisms" that looks for this.

A bug report against that Debian package would be in order.

This isn't a bashism, really.  It's simply a badly written shell script,
and it could fail if run under some other POSIX-compatible shell that
happens to define a special behavior for any of the other badly chosen
variables in the script.

If on the other hand "checkbashisms" is really a "check for anything
that was done wrong" linter, a la shellcheck, then sure, it would be
worth adding "don't use all-caps variables" to it.

> If I run a script against a POSIX-compliant (PC) shell using only PC
> syntax/features and then I run the same script with Bash in PC mode, I
> would expect an identical result.  To me, anything else indicates a bug
> in one of the shells.  Perhaps this assessment is naive.

Sadly, it is.  POSIX defines a minimal set of behaviors that all compliant
shells (and certain utilities) must implement.  Shells and other programs
are allowed to implement additional features (extensions), and *all*
shells do this.  There is no such thing as a "strictly POSIX sh".  Even
dash and busybox sh have extensions.

It's up to the script writer to ensure that they only use portable
features and good practices.  No linter is ever going to catch every
problem.

Re: Bug concatenating member of large array with string

[Greg Wooledge]

On Tue, Aug 10, 2021 at 02:26:56PM +0100, Joe Pater wrote:
> The attached bash script (named 'test'), when run on my laptop,
> produces the following output:
> 
> xyzerg
> 
> Instead of the expected:
> 
> Aabergxyz

OK, let's see if we can reproduce that.  My initial thought is carriage
return poisoning, but your attached "file" doesn't have CRs.  Then again,
that could be an artifact of an email system.

> #!/bin/bash
> 
> words=(`cat file`)
> echo ${words[0]}xyz

You really should be using mapfile for this, and quoting properly.

#!/bin/bash
mapfile -t words < file
echo "${words[0]}xyz"

> Aaberg
> Aachen
> aahing
[...]

With the script as I wrote it above, and those three lines in the input
file, I get:

unicorn:~$ ./foo
Aabergxyz

If I edit the file and put carriage returns at the end of each line,
I get:

unicorn:~$ ./foo
xyzerg

This matches your result.  So, it would appear you have carriage returns
in your input file.

Re: RFE: new option affecting * expansion

[Greg Wooledge]

On Wed, Aug 11, 2021 at 10:43:12AM -0400, Chet Ramey wrote:
> On 8/10/21 5:08 PM, Chris F.A. Johnson wrote:
> > 
> > It would be nice if there were an option to allow * to expand sorted
> > by timestamp rather than aphabetically.
> 
> When you say `timestamp' I assume you mean by last modification time.

That was my interpretation too.

Re: GROUPS

[Greg Wooledge]

On Wed, Aug 11, 2021 at 08:00:12PM -0400, Franklin, Jason wrote:
> This doesn't work unless it was recently fixed.  A variation does...
> 
> bash-5.0$ echo $BASH_VERSION
> 5.0.17(1)-release
> bash-5.0$ GROUPS=FOO bash -c 'echo $GROUPS'
> 1000
> bash-5.0$ GROUPS=FOO bash --posix -c 'echo $GROUPS'
> 1000
> bash-5.0$ env GROUPS=FOO bash -c 'echo $GROUPS'
> FOO
> bash-5.0$ env GROUPS=FOO bash --posix -c 'echo $GROUPS'
> FOO

You can't overwrite GROUPS while you're inside of bash, apparently not
even in the temporary assignment context before a simple command.  So,
in order for the first form to work as expected, you'd have to be in a
shell other than bash.

unicorn:~$ ksh
$ GROUPS=FOO bash -c 'echo "$GROUPS"'
FOO

And so on.  Either that, or unset GROUPS first.

Re: GROUPS

[Greg Wooledge]

On Fri, Aug 13, 2021 at 10:10:42AM -0400, Chet Ramey wrote:
> As long as you stick to things POSIX standardizes. Relevant here, the
> standard even includes a list of variables you should avoid using because
> various shells and applications use them.

Just out of curiosity, where is that?  It's not under section
2.5.3 Shell Variables, which is where I looked first.

The closest thing I could find is a section in the Rationale that
describes some variables that are no longer covered by the standard,
but either used to be, or are commonly used as extensions:

https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_05_03

Is there another section that I've missed?

> Obviously not, since they never tested it on bash. It's more likely they
> expected it to run on Debian's /bin/sh -- hence the hashbang -- which is
> dash.

For the record, Debian uses dash as /bin/sh by default in modern versions,
but historically, it used bash.  And it still offers the user a supported
means of switching "back" to bash as /bin/sh, for systems that require it.

A Debian shell script which uses #!/bin/sh should be written to work
under bash *or* dash (at the very minimum).  If it fails under either
of those shells, it's a bug.

Re: RFE: new option affecting * expansion

[Greg Wooledge]

On Mon, Aug 16, 2021 at 10:35:12PM -0400, Dale R. Worley wrote:
> Back in the old, old days, there was a program named "glob" that did
> pathname expansions.  So you wouldn't say
> 
>cat *
> 
> you'd say
> 
>cat $( glob * )

Tcl still does it that way.  Not with that syntax, but the command
name is "glob".  (It doesn't sort them for you, though.)

> where glob would get one argument, "*", and output a list of file
> names.  A glob-by-modification-date program would be a better solution
> for this need, IMHO.

Would this be a shell builtin, or a loadable bash builtin, or an external
program?  Honestly it just sounds like more buck-passing.

There's a history of other projects refusing to provide this kind of
functionality, even when it would have been super easy, and even when
a patch was provided.[1]

Trying to reintroduce an external glob(1) utility will go over like a
lead balloon, I'm sure.  Even if by some miracle you do manage to get
one included in a single Linux distribution, it won't be universally
available, which means it's pretty useless for shell scripting.

[1] https://lists.gnu.org/archive/html/coreutils/2014-02/msg5.html


Right now, there are zero great ways to get a list of the files in a
directory sorted by mtime.  If you happen to be on a recent GNU coreutils
system, you can use ls -t --quoting-style=shell-escape together with
bash's eval.  This isn't too far off from your glob(1) suggestion,
but it only works on systems that have this particular version of ls(1),
and only in bash.

Otherwise, if you happen to have GNU find and GNU sort available, you
can cobble something together with find -printf '%T@ %p\0' and sort -zn.
It's clumsy and ugly, and it's still not portable, but it might be
an available option on some systems that don't have a new enough GNU
coreutils.  Maybe.

What else... well, you could write your own Bubble Sort algorithm in bash,
using [[ $a -nt $b ]] to compare array entries and swap them.  The good
news is that this would be portable to any system with bash.  The bad
news: speed?  What is speed?  I guess you could live with it for a typical
log directory where you only have like 30 files (one month's worth).  It
wouldn't be fun or elegant, though.

The most common solution?  "Redesign your whole application so that the
filenames contain ASCII-sortable timestamps."

An option to change the glob sorting order would actually be useful,
not for a large number of scripts, but for a very specific problem domain
where the shell is often used.  Is it critically important?  No.  But it
would be helpful.

Re: RFE: new option affecting * expansion

[Greg Wooledge]

On Tue, Aug 17, 2021 at 10:26:21AM +0300, Oğuz wrote:
> On Tue, Aug 17, 2021 at 6:31 AM Greg Wooledge  wrote:
> > An option to change the glob sorting order would actually be useful,
> > not for a large number of scripts, but for a very specific problem domain
> > where the shell is often used.  Is it critically important?  No.  But it
> > would be helpful.
> 
> I agree. It would be even better if we could have the results of
> filename expansion in reverse-lexicographic and natural order too.

"Natural order"... eh.  Maybe.  I'm not convinced it's super useful, but
if it's easy to include, then it might be worth throwing in the pot.

But reverse order, you can do yourself, just by storing the results in
an array and then iterating backwards.

Re: bash-5.1.8 does not compile anymore on HP-UX due to invalid shell syntax

[Greg Wooledge]

On Tue, Aug 17, 2021 at 01:02:06PM +0200, Osipov, Michael (LDA IT PLM) wrote:
> this is basically the same issue as I reported in readline:
> https://lists.gnu.org/archive/html/bug-readline/2021-08/msg0.html

As Chet said in that message, you have misinterpreted the POSIX
documentation.

> The bad hunk seems not to be POSIX shell compliant. Both vars are set for me
> on HP-UX because the expansion is incomplete. According to [1] it has to be
> $(GCC:+..} and not $(GCC+..}. I have changed ./configure locally and it
> works with /bin/sh.

What version of HP-UX?  What is the actual error you're getting?

I only have access to certain older versions of HP-UX, but I don't
see what you're claiming.

# exec /bin/sh
# uname -a
HP-UX minea B.11.11 U 9000/785 4239047153 unlimited-user license
# GCC=gcc
# echo "[${GCC+-foo}]"
[-foo]
# unset GCC
# echo "[${GCC+-foo}]"
[]
#

>From your email, my first guess was that you thought there was a parsing
ambiguity when the +- characters appeared together, but I cannot reproduce
any problem.  Please give more detail.

(P.S. you also have typos in your email.  You've got an open left paren
which is "matched" by a closing right curly brace.  If you actually
edited some file and introduced that typo, that might account for whatever
error you're seeing.  If on the other hand there really is a mismatched
paren-brace error in one of bash's source code files, that would be a
real bug.)

Re: bash-5.1.8 does not compile anymore on HP-UX due to invalid shell syntax

[Greg Wooledge]

On Tue, Aug 17, 2021 at 04:03:17PM +0200, Osipov, Michael (LDA IT PLM) wrote:
> Am 2021-08-17 um 15:53 schrieb Greg Wooledge:
> > What version of HP-UX?  What is the actual error you're getting?
> 
> Latest and greatest: 11.31 v3
> 
> > I only have access to certain older versions of HP-UX, but I don't
> > see what you're claiming.
> >
> > # exec /bin/sh
> > # uname -a
> > HP-UX minea B.11.11 U 9000/785 4239047153 unlimited-user license
> > # GCC=gcc
> > # echo "[${GCC+-foo}]"
> > [-foo]
> > # unset GCC
> > # echo "[${GCC+-foo}]"
> > []
> > #
> This is *not* the same as in the configure script. It does not unset the
> variable, but sets to empty.

# exec /bin/sh
# uname -a
HP-UX minea B.11.11 U 9000/785 4239047153 unlimited-user license
# GCC=gcc
# echo "[${GCC+-foo}]"
[-foo]
# unset GCC
# echo "[${GCC+-foo}]"
[]
# GCC=
# echo "[${GCC+-foo}]"
[-foo]
#

What error are you seeing?

Re: bash-5.1.8 does not compile anymore on HP-UX due to invalid shell syntax

[Greg Wooledge]

On Tue, Aug 17, 2021 at 07:30:45AM -0700, L A Walsh wrote:
> The pairs are about 4 related operations.  If you let P = the oPerator
> then the odd lines are about ':P' and the even lines are about 'P' (no
> colon).
> The Pairs from 1-4 are about the operators: '-', '=', '?', '+'
> 
> Pair 4 shows effects of ':+' and '+'.
> 
> Isn't that what you are talking about?
> 
> Yeah -- w/o the ':' looks a bit 'off', but it has a separate meaning
> and has been around for a long time.
> 
> (I first encountered it when porting 1980's era scripts)

The form without the colon (':') is the original form, and comes from
the Bourne shell.  If you read sufficiently old literature, you'll see
*only* those forms mentioned.

The form with the colon was added by the first Korn shell, and became
widely adopted by other shells.  Eventually it became standardized
in POSIX.

So yes, all 8 forms are standardized by POSIX.

I'm still wondering what issue the OP is actually seeing.  If they claim
that changing ${GCC+-stuff} to ${GCC:+-stuff} in some file fixes things,
and if they also claim that *something* is setting the GCC variable to
the empty string rather than unsetting it, then perhaps it's not a
syntactic error at all, but rather a logical error.

Either the thing that's setting GCC to the empty string ought to be
changed to unset GCC, or else the ${GCC+-stuff} check ought to be changed
to ${GCC:+-stuff} (at the risk of breaking Bourne shell compatibility,
if we care about that).

But this is speculative until they tell us what the actual problem is.