UUID as Array Keys strangely not possible
Howdy,
The following cannot work because, for some reason, the array subscript
parser insists on doing math on array indices even when the array is
associative instead of numeric
typeset -A UUID_TABLE
...
UUID_TABLE+=( [${SOME_UUID}]=${SOME_VALUE} )
...
some_command ${UUID_TABLE[${SOME_UUID}]}
The parser and evaluator insist on doing math on ${SOME_UUID} no matter
how its quoted or whatever. This seems extremely wrong.
At a minimum putting the index in double quotes should suppress the
arithmetic evaluation. In the ideal the decision to math-or-not decision
should happen after the array type is known, thought that's probably too
hard.
Alternately some sort of builtin hash function to map complex strings
into usable associative array indexes would be nice.
UUIDs are simply too useful and common in current systems to sustain
their current second-class status.
(DISCLAIMER: I'm not a subscriber to this list.)
Re: UUID as Array Keys strangely not possible
On 1/22/19 10:23 PM, Chet Ramey wrote:
On 1/22/19 3:32 PM, Robert White wrote:
Howdy,
The following cannot work because, for some reason, the array subscript
parser insists on doing math on array indices even when the array is
associative instead of numeric
typeset -A UUID_TABLE
...
UUID_TABLE+=( [${SOME_UUID}]=${SOME_VALUE} )
...
some_command ${UUID_TABLE[${SOME_UUID}]}
The parser and evaluator insist on doing math on ${SOME_UUID} no matter how
its quoted or whatever. This seems extremely wrong.
Do you have some sample UUID data to test this with?
I'm going to have to provisionally withdraw this report. The problem
only seems to happen in the custom /init script in my initramfs. Trying
to recreate it with a simpler script (and the same bash binary) on a
fully running system using the the UUIDs I collected with blkid doesn't
have a problem at all. So something "mysterious" is going on.
The initscript is part of https://sourceforge.net/projects/underdog/ (if
you care) and is part of my attempt to build a concordance of UUID to
device to manager (e.g. lvm vs mdadm vs whatever) app. It works well
except when it doesn't.
Thanks for the prompt response. If I isolate the test case or the issue
in general I'll be back. Even just to say never mind if I find a super
stupid mistake. 8-)
--Rob White.
Re: UUID as Array Keys strangely not possible
On 1/26/19 4:37 AM, Dennis Williamson wrote:
On Fri, Jan 25, 2019, 9:51 PM Robert White
On 1/22/19 10:23 PM, Chet Ramey wrote:
On 1/22/19 3:32 PM, Robert White wrote:
Howdy,
The following cannot work because, for some reason, the array subscript
parser insists on doing math on array indices even when the array is
associative instead of numeric
typeset -A UUID_TABLE
...
UUID_TABLE+=( [${SOME_UUID}]=${SOME_VALUE} )
...
some_command ${UUID_TABLE[${SOME_UUID}]}
The parser and evaluator insist on doing math on ${SOME_UUID} no matter
how
its quoted or whatever. This seems extremely wrong.
Do you have some sample UUID data to test this with?
I'm going to have to provisionally withdraw this report. The problem
only seems to happen in the custom /init script in my initramfs. Trying
to recreate it with a simpler script (and the same bash binary) on a
fully running system using the the UUIDs I collected with blkid doesn't
have a problem at all. So something "mysterious" is going on.
The initscript is part of https://sourceforge.net/projects/underdog/ (if
you care) and is part of my attempt to build a concordance of UUID to
device to manager (e.g. lvm vs mdadm vs whatever) app. It works well
except when it doesn't.
Thanks for the prompt response. If I isolate the test case or the issue
in general I'll be back. Even just to say never mind if I find a super
stupid mistake. 8-)
--Rob White.
I'm not going to try to do a full code review. It took me more time than I
was willing to spend already to find a file that has an associative array
referring to UUIDs in the first place and to put this message together.
Here are a few comments about prototype/init:
In the global scope UUID_TABLE is declared as an indexed array at one point
then an associative array later.
In the get_ (something) function an array element is set violating
separation of get and set.
In that same function, a max_index variable is set to the highest index of
a numeric array then the array is iterated using a C-style for loop. Arrays
in Bash are sparse and the correct way to iterate is to step over the
elements or the indices.
for element in "${array[@]}"
for index in "${!array[@]}"
But it seems you just want to add an element.
array+=(element)
There is a variable called AA. I didn't look to see if that means something
but I shouldn't have to. A better name is needed.
Forgive me if I misunderstood anything. It was just a cursory once over.
Also, I didn't immediately notice what the UUID indexing problem initially
reported is caused by. Except that I played with an indexed array with a
UUID-like index. In that case the index *is* evaluated mathematically so
this may be your problem. That makes sense based on the redefinition I
mentioned above since the second declare will produce an error without
affecting the array.
Yeah, that was me trying to hack around the UUID problem.
I tried to reduce the UUID index errors into a meta index where I look
for the UUID in an indexed array and append it to the array if not
found, and then use that index as the index elsewhere. It's a dumb trick
that I used once in another environment (NexExpert, circa 1995 if you
care) that didn't give me valid associative arrays.
The script as it currently exists is currently a confused abortion of
hackery surrounding the original subscript errors.
But it started because just using the UUIDs as the associative array was
crashing... but only on some UUIDs.
It started clean but it got ugly damn fast when the UUID errors initally
showed up.
(so the get_ is a find-or-append-then-return-index() nonsense thing.)
It turned into a game of wac-a-mole.
Like I said, it got ugly. ha ha ha.
I've now got to back out the worst of the hacks and figure out why the
original ${DEVICE_TABLE[$UUID]} style indexing failed in the init script
when it clearly works with my collected UUIDs in regular user contexts.
I've already gone through and replaced or refactored all the
double-square-bracket evaluations with single-square-bracket
expressions, and otherwise looked for any explicit arithmatic contexts
that might be confusing the issue.
The current state is hideous, which is why I need to back out the hacks;
but the ugly hacks went in because of the indexing errors, so something
is fishy.
--Rob.
General Associative Array problem -- Was: UUID as Array Keys strangely not possible
Consider the script at the end of this email. It's effectively
impossible to get or to accept the various values.
Either that or I am doing something incredibly stupid.
(It took me a while to factor this down and try virtually every
workaround that would not involve writing external files.)
#!/bin/bash -f
# These three arrays never get values no matter what
typeset -A
typeset -A
typeset -A
typeset -A INLINE
typeset -A INFUNC
function testfunc() {
eval INFUNC=( $( blkid /dev/${RECORD[3]} -o export | sed -e 's;\\
;_;' -e 's;\(.*\)=;[\1]=;' ) )
echo "Trying: +=( [${INFUNC[UUID]}]=boo )"
eval +=( [${INFUNC[UUID]}]=boo )
}
tail --lines=+3 /proc/partitions | grep -v ram |
while read -a RECORD
do
echo "Trying /dev/${RECORD[3]}"
testfunc ${RECORD[3]}
VV=$( blkid /dev/${RECORD[3]} -o export | sed -e 's;\\ ;_;' -e
's;\(.*\)=;[\1]=;' )
eval INLINE=( ${VV} )
if [ ${INLINE[UUID]} ]
then
echo "Trying: [${INLINE[UUID]}]=boo"
eval [${INLINE[UUID]}]=boo
fi
echo "INLINE Keys: " "${!INLINE[@]}" = "${INLINE[@]}"
echo "INFUNC Keys: " "${!INFUNC[@]}" = "${INFUNC[@]}"
for AA
in "${!INLINE[@]}"
do
if [ ${AA} == UUID ]
then
echo "Trying: [${INLINE[$AA]}]=${INLINE[DEVNAME]}"
[${INLINE[$AA]}]=${INLINE[DEVNAME]}
fi
done
echo
done
echo " Keys: " "${![@]}" = "${[@]}"
echo " Keys: " "${![@]}" = "${[@]}"
echo " Keys: " "${![@]}" = "${[@]}"
Re: General Associative Array problem -- Was: UUID as Array Keys strangely not possible
On 2/22/19 4:17 PM, Eduardo Bustamante wrote:
On Fri, Feb 22, 2019 at 3:24 AM Robert White wrote:
(...)
tail --lines=+3 /proc/partitions | grep -v ram |
while read -a RECORD
do
(...)
eval [${INLINE[UUID]}]=boo
(...)
done
echo " Keys: " "${![@]}" = "${[@]}"
echo " Keys: " "${![@]}" = "${[@]}"
echo " Keys: " "${![@]}" = "${[@]}"
The `while' loop executes in a subshell and you're not using
`lastpipe'. See: https://mywiki.wooledge.org/BashFAQ/024
As a general recommendation: when submitting bug reports, try to
simplify things as much as possible, e.g.:
* Replace the `blkid' command invokation with a `printf' (not everyone
has the same block devices, nor runs Linux)
* Replace `/proc/partitions' with a copy of its contents (again, not
everyone has the same files in their systems, makes it impossible to
reproduce what you're experiencing)
* Remove the and arrays, we only need a sample case to
understand what's going on, not three
* Remove the sed commands, you're going to be using static values
(`printf') anyways
* Does the `set -f' at the top of the file have any effect in the
outcome in this particular case? no? then remove it
* I think you get the idea.
Just to help you understand my perspective, this is what I see when I
try to troubleshoot your script (a bunch of errors), that doesn't
really help me help you:
dualbus@system76-pc:~$ shellcheck foo.sh
In foo.sh line 12:
function testfunc() {
^-- SC1009: The mentioned syntax error was in this function.
^-- SC1073: Couldn't parse this brace group. Fix
to allow more checks.
In foo.sh line 13:
eval INFUNC=( $( blkid /dev/${RECORD[3]} -o export | sed -e 's;\\
^-- SC1036: '(' is invalid here. Did you forget to escape it?
^-- SC1056: Expected a '}'. If you have one, try a ; or
\n in front of it.
^-- SC1072: Missing '}'. Fix any mentioned problems and
try again.
^-- SC1098: Quote/escape special characters when using
eval, e.g. eval "a=(b)".
Your syntax checker is straight tripping on that SC1036 error dude.
Array assignment using ARRAYNAME=( expression ) is completely legal and
correct with or without the eval. The structure even allows for line
continuation just like pipelines
ARRAYNAME=(
)
Same for the error emitted for using the "function" keyord
function somename() {
}
It does look like it got line-wrapped to death in the sed, but according
to my sent items directory that happened after it left here. So my bad
for lack of attachment.
I get your point for blkid and /proc/partitions. I was getting
frustrated since I'd tried all three forms of substitution I didn't know
if the function calls, the evaluation order, or the expansions were to
blame so I included all three contexts.
Turns out I've been doing the pipe-while-read thing blithely for a good
twenty years now (going back to the borne shell on SVR4) and never
thought about the pipeline itself.
So I was right, for stupid definitions of right, when I wondered if I
was missing something incredibly obvious. /sigh. 8-)
Thanks for all the help.
--Rob.
