Re: Indices of array variables are sometimes considered unset (or just display an error).
On Mon, Nov 05, 2018 at 08:04:49PM -0500, Great Big Dot wrote:
> Hold up... when I view this email on the public archives, all of my
> "${array[@]}"'s (that is, "${array[]}"'a) got turned to
> "address@hidden"'s.
Sadly, there's nothing we can do about that. The maintainers of the
list archive would have to make their anti-spam measure a bit smarter
than it currently is.
Re: Indices of array variables are sometimes considered unset (or just display an error).
On 11/5/18 4:42 PM, Great Big Dot wrote:
> Description:
> The parameter expansion "${!var[@]}" expands to the indices of an array
> (whether linear or associative). The expansion "${var-string}"
> returns "${var}" iff var is set and 'string' otherwise. These two
> features do not play well together:
You seem to have neglected a significant section of the documentation:
"If the first character of parameter is an exclamation point (!), and
parameter is not a nameref, it introduces a level of indirection. Bash
uses the value formed by expanding the rest of parameter as the new
parameter; this is then expanded and that value is used in the rest of
the expansion, rather than the expansion of the original parameter.
This is known as indirect expansion. The value is subject to tilde
expansion, parameter expansion, command substitution, and arithmetic
expansion. If parameter is a nameref, this expands to the name of the
parameter referenced by parameter instead of performing the complete
indirect expansion. The exceptions to this are the expansions of
${!prefix*} and ${!name[@]} described below. The exclamation point
must immediately follow the left brace in order to introduce indirec-
tion."
> $ declare -a -- array=([0]='helloworld')
> $ printf -- '%s\n\n' "${!array[@]-Warning: unset}"
> Warning: unset
This happens to "work" because there is a single array element set, and it
expands to a single word.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Indices of array variables are sometimes considered unset (or just display an error).
On 11/5/18 8:44 PM, Great Big Dot wrote:
> What's actually happening here is that the *indirection* expansion
> "${!foo}", and not the *indices* expansion "${!foo[@]}", is what is being
> preformed on something like "${!array[@]-}". Both expansions, while
> unrelated, happen to use the same syntax, with the exception that
> indirections apply to normal variables and index expansions apply to array
> variables. For some reason, adding on the "${foo-default}" expansion causes
> the former to be used instead of the latter.
The `some reason' is that the current behavior is documented and has been
the way bash has worked since I added indirect expansion (bash-2.0) and the
array keys expansion (bash-3.0).
> This can be seen here:
I'm going to use bash-4.4 for my explanation.
>
> $ array=(foo)
> $ printf -- '%s\n' "${!foo[@]-unset}"
> unset
Of course. There is no variable "foo"; the indirect expansion results in a
null string.
> $ foo='hello world'
> $ printf -- '%s\n' "${!foo[@]-unset}"
> hello world
(I don't get that result with any version of bash. I went back to bash-3.0
before I quit trying.)
This is a case where bash is trying to be helpful, maybe more so than is
desired. `foo' isn't an array variable, but the !foo[@] is first identified
as a candidate for indirect expansion, then checked to see whether or not
it is the entire expansion between ${ and }. Since it's not, it's not a
candidate for array keys expansion (this is as documented).
Since the array keys expansion isn't valid, the attempt to perform variable
indirection holds. This is where the helpful part comes in. Bash variables
can be referenced as arrays, even if they are not, using `0', `@', or `*'
as subscripts. That means that foo[@] gets expanded into "hello world",
which the shell tries to use as a variable name, resulting in:
$ ../bash-4.4-patched/bash ./x16
./x16: line 2: hello world: bad substitution
The error message in bash-5.0 is a little better:
$ ../bash-5.0-beta/bash ./x16
./x16: line 2: hello world: invalid variable name
>> This pattern of behavior is apparently unaffected by changes to IFS[...]
>
> Upon further examination, and in light of the above realization, this
> actually isn't true. In particular, iff the first character of IFS is
> alphanumeric or an underscore (or if IFS is the empty string), and if you
> use the "${array[*]}" form instead, then the expansion doesn't throw an
> error when the array contains more than one element.
Sure, since a double-quoted expansion using `*' separates words using the
first character of IFS.
> $ foo_bar='Beto2018'
> $ printf -- '%s\n' "${!array[*]-Warning: unset}"
> Beto2018
Nice. Let's hope he pulls it off today.
> Is there a good reason for treating "${!array[@]-}" and "${!array[*]-}"
> like indirections instead of index expansions (or just throwing an error)?
When I added array variables and the array keys expansion, I used the ksh93
syntax (${!var[sub]}) and tried to avoid conflict with the existing
indirect expansion as much as possible (that was back when we still thought
there was a chance that POSIX would standardize arrays and it was useful to
have consistent implementations). The ksh93 expansion syntax made it
invalid to use the array keys expansion as part of the ${param:-word}
expansion, so that's how I made it work. Since it wasn't a candidate for
that family of expansions, the existing variable indirection syntax
controls.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Indices of array variables are sometimes considered unset (or just display an error).
On 11/5/18 8:44 PM, Great Big Dot wrote:
> > What's actually happening here is that the *indirection* expansion
> > "${!foo}", and not the *indices* expansion "${!foo[@]}", is what is being
> > preformed on something like "${!array[@]-}". Both expansions, while
> > unrelated, happen to use the same syntax, with the exception that
> > indirections apply to normal variables and index expansions apply to array
> > variables. For some reason, adding on the "${foo-default}" expansion causes
> > the former to be used instead of the latter.
> The `some reason' is that the current behavior is documented and has been
> the way bash has worked since I added indirect expansion (bash-2.0) and the
> array keys expansion (bash-3.0).
Fair enough. In retrospect, the dual usage of "!" should have been the
first thing I should have blamed when it didn't behave as I expected.
> > $ foo='hello world'
> > $ printf -- '%s\n' "${!foo[@]-unset}"
> > hello world
> (I don't get that result with any version of bash. I went back to bash-3.0
> before I quit trying.)
Oops, must have made a copy-paste error. Sorry for wasting your time
with that! What I meant was something like:
$ foo='hello world'
$ array=(foo)
$ echo "${!array[@]-unset}"
hello world
>> Upon further examination, and in light of the above realization, this
>> actually isn't true. In particular, iff the first character of IFS is
>> alphanumeric or an underscore (or if IFS is the empty string), and if you
>> use the "${array[*]}" form instead, then the expansion doesn't throw an
>> error when the array contains more than one element.
> Sure, since a double-quoted expansion using `*' separates words using the
> first character of IFS.
Yeah, that was what led me to make that observation. Before it hit me
that bash was interpreting it as an indirection, on a hunch I messed
around a bit with IFS and nothing seemed to change, and said so. After
I realized what was happening, I just wanted to correct my false claim
that IFS was irrelevant.
>> Is there a good reason for treating "${!array[@]-}" and "${!array[*]-}"
>> like indirections instead of index expansions (or just throwing an error)?
> When I added array variables and the array keys expansion, I used the ksh93
> syntax (${!var[sub]}) and tried to avoid conflict with the existing
> indirect expansion as much as possible (that was back when we still thought
> there was a chance that POSIX would standardize arrays and it was useful to
> have consistent implementations). The ksh93 expansion syntax made it
> invalid to use the array keys expansion as part of the ${param:-word}
> expansion, so that's how I made it work. Since it wasn't a candidate for
> that family of expansions, the existing variable indirection syntax
> controls.
Makes sense. IMO, it still might be worth mentioning explicitly in the
manual what happens when you try to mix and match various expansions,
at least in cases where there's more than one reasonable
interpretation. (Granted, a closer reading of the relevant sections
would have made it clear that *only* array[@] and array[*] were privy
to the secondary meaning of "!", as opposed to variations.)
Fwd: Indices of array variables are sometimes considered unset (or just display an error).
Crap, looks like I accidentally just replied to a single person instead of
the whole list. Here it is (and sorry if I'm uselessly cluttering up
everyone's inboxes!):
-- Forwarded message -
From: Great Big Dot
Date: Tue, Nov 6, 2018 at 5:45 PM
Subject: Re: Indices of array variables are sometimes considered unset (or
just display an error).
To:
> On Mon, Nov 5, 2018 at 10:38 PM Eduardo Bustamante
wrote:
> Sorry, I'm having a hard time following this email thread.
My bad, sorry if I wasn't clear enough. I expected this:
$ array=(foo)
$ printf '%s\n' "${!array[@]-unset}"
0
That is, I expected the keys of 'array' to be printed, unless the array was
unset. Instead, I either got errors or the "unset" string, and never the
actual keys. I didn't even realize that the indirection had something to do
with it, though once it hit me it made the behavior much clearer. I
implicitly assumed that the indirection stuff only applied to non-array
variables, and index-expansion only to array variables.
> What is your ultimate goal or the actual problem you're trying to solve?
Honestly, I was just messing around at the command line to get the hang of
some of the expansions. IIRC, I was looking at the "parameter
transformation" stuff ("${var@Q}", "${var@A}", etc.). I was looking for any
edge cases that might trip me up in the future, and eventually I wondered
what would happen if you used an array. Somehow that led to me trying other
expansions with arrays, and I ended up getting errors I couldn't explain.
> (BTW, I would recommend against trying to do three expansions in one.
> It might be more terse, but it's hard to read and as you found out,
> leads to weird behavior)
Yeah, I think I'm starting to get that now...
