Re: Short doc for readarray synonym for mapfile needs to be updated
On 9/19/16 7:27 PM, jhank...@homewood.k12.al.us wrote: > Bash Version: 4.4 > Patch Level: 0 > Release Status: release > > Description: > bash-4.4 mapfile supports -d DELIM, and the short doc in the help has > been updated to reflect this. However, the short doc for the readarray > synonym hasn't been updated. Thanks for the report; good catch. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Command substitution with null bytes generates warning
On 9/19/16 12:28 PM, L. A. Walsh wrote: > > > Chet Ramey wrote: >> On 9/16/16 1:51 AM, Eric Pruitt wrote: >> >> >>> Bash Version: 4.4 >>> Patch Level: 0 >>> Release Status: release >>> >>> Description: >>> I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux* >>> ]];`. >>> All /proc/*/cmdline include null bytes, and as of Bash 4.4, this >>> results in >>> a warning being spewed on stderr which did not happen in Bash 4.3. >>> >> >> Other users have expectations that differ from yours. I received messages >> reporting the the bash-4.3 behavior (the longtime bash behavior) as a bug. >> Warning the user that bash discards some characters from the command >> substitution output seemed like the course that would let everyone know >> what's happening regardless of their expectations. >> > --- >If users were relying on this behavior (I know I have scripts that read > things from proc -- a text interface that uses \0 to display values similar > to MS's multi-string Values in the Windows registry. Don't assume that every use of something like this has to do with /proc. Here's a representative report: "I was wondering what would happen if I'd do something like that: $ foo="$(cat file_containing_ascii_null_byte)" or faster $ foo="$(http://www.gnu.org/software/bash/manual/bash.html#Command-Substitution didn't help me. Is this behaviour intended? Is there any possibility to fix that (if it's no feature)?" -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Command substitution with null bytes generates warning
Chet Ramey wrote: Don't assume that every use of something like this has to do with /proc. Here's a representative report: "I was wondering what would happen if I'd do something like that: $ foo="$(cat file_containing_ascii_null_byte)" or faster $ foo="$(http://www.gnu.org/software/bash/manual/bash.html#Command-Substitution didn't help me. Is this behaviour intended? Is there any possibility to fix that (if it's no feature)?" I asked the same question and wanted the same resolution -- some way to accomplish what I intended. I never asked for a WARNING... that's literally putting salt in the wound. I can see it wouldn't be easy to implement given other constraints, but that doesn't mean I wouldn't like the feature if it was able to be implemented.But giving a warning that you can't globally turn off? Just because people *want* a feature doesn't mean you should ~flaunt~ (hyperbole) Bash's lack of the feature with warnings that can't be disabled. _If_ I didn't know better, I'd say it was malicious checks to ensure no one would expect it. Whereas I would counter with: is it documented under the read function? If it wasn't documented, what would you expect people to "expect" or "ask about"? So what happens if I read in invalid UTF8, or if I read in a file from disk that has no "\n" at the end of the file -- do you add one? Do you throw away the line? Various pieces of software feel a need to change previous behavior (nothing special was done and usually you had a partial line at the end of the file), by either ignoring the last line (rare, usually accident) or appending one for you (which corrupts text files that are size-checked as a means for determining validity). Now bash emits warnings for what has been *normal* behavior. So anyone with working code/scripts that expect the normal behavior will now be penalized with a new warning that can't be globally turned off -- only by editing all the instances. That's not a good way to deal with this -- especially if all these borderline cases aren't documented in the manpage. -l
Re: Command substitution with null bytes generates warning
I probably shouldn't even bother at this point, but morbid curiosity
compels me to foolish ends.
What are you DOING with these files that contain NUL bytes that makes
it permissible to simply drop the NUL bytes on the floor, with no
explicit step like tr -d \\0 to remove them?
How is your script anything but buggy, if it is relying on a program
(shell) to drop certain input bytes, when there is no documentation
stating that it does this?
If you are reading a file that uses \0 as a separator between fields,
then how it is *OK* for your script to mash the fields all together
into one giant mess? Why aren't you reading the fields separately?
Take for example the file /proc/1/cmdline on Debian 8 (with mostly
default settings):
wooledg@wooledg:~$ cat -vtue /proc/1/cmdline
/lib/systemd/systemd^@--system^@--deserialize^@16^@wooledg@wooledg:~$
I count 4 NUL bytes there, each one presumably terminating a field
of some kind.
If you simply use $(< /proc/1/cmdline) in some naive script running
under bash 2.0 ~ 4.3 then you get something like
/lib/systemd/systemd--system--deserialize16
Again, how is a script that produces this as a value not simply buggy?
Perhaps what you really wanted was something like:
{ read -r -d '' a; read -r -d '' b; read -r -d '' c; read -r -d '' d; } \
< /proc/1/cmdline
which gives:
wooledg@wooledg:~$ declare -p a b c d
declare -- a="/lib/systemd/systemd"
declare -- b="--system"
declare -- c="--deserialize"
declare -- d="16"
I don't know what these fields *mean*, but that's OK, because I'm not
the one using them in a script. But this would be a much better way
of getting them, than a blind command substitution.
Re: Command substitution with null bytes generates warning
Greg Wooledge wrote:
I probably shouldn't even bother at this point, but morbid curiosity
compels me to foolish ends.
What are you DOING with these files that contain NUL bytes that makes
it permissible to simply drop the NUL bytes on the floor, with no
explicit step like tr -d \\0 to remove them?
Usually to write a quick & dirty 1-off script to read something
and give me some info.
If I want to improve upon it or have it handle the null data better,
I'd probably use perl. But as an example see attached. I've had a few
versions of this script (this one is dated 2011).
If you run it in a dir where there is binary
data, you are almost certain to see have some nulls in the data.
I used something this script to look at values in /proc or /sys.
Run as a normal user -- I get many unreadables, but enough interesting
things to help me understand what to do and how to use some of the values:
/sys/class/net/br0> /tmp/showvals
addr_assign_type: 1
addr_len: 6
address: 00:15:17:bf:be:b2
/tmp/showvals: line 63: printf: `invalid format character
brforward:'`f#สง
7ridge/ageing_time:3
bridge/bridge_id: 8000.001517bfbeb2
bridge/default_pvid: 1
bridge/flush:
bridge/forward_delay: 1500
bridge/gc_timer: 23319
bridge/group_addr: 1:80:c2:0:0:0
bridge/group_fwd_mask: 0x0
bridge/hash_elasticity:4
bridge/hash_max: 512
bridge/hello_time: 200
bridge/hello_timer:0
bridge/max_age:2000
etc... brforward sure looks like binary, for example.
One can do the same on windows in /proc/registry (in cygwin) --
lots of binary values that display just fine, in spite of dropping
nulls.
This is just 1 example...But since these are knockoff scripts, I often
don't remember how/where to find them ... this one was by luck, last
modified ~ 5 years ago.
How is your script anything but buggy, if it is relying on a program
(shell) to drop certain input bytes, when there is no documentation
stating that it does this?
How is it buggy? ... works fine for the listed requirements! ;-)
Remember What I said about you being "too advanced" to relate to
some of these things: case in point. ;-)
#!/bin/bash
# designed for displaying config vals stored in files;
# recursively display values (or 1st line) of files in cur dir
# or given on cmd line;
# dirs given or below targets will be expanded;
#
# (c) 2011 L. Walsh (lawa...@tlinx.org);
# License: copy and distribute freely, but for this source:
# * leave attribution on unmodified or slightly modified copies. (OR)
# * leave 'honorable mention' if this is > 30% of any final work
(OR)
# * Else, Do as thow wilt... and enjoy!
# (Hopefully, that's not too onerous a license...)
# this is writtin 'split', so if one wants auto-priv-raising,
# it can be done once and not on every recursive call
# If you want to force root to be used when running this
# instead of raising privs as needed, written / split this file can be
'sourced' and the function
# will define to mem and so if 'sudo' is needed
USE_SUDO=1
# to prevent automatically trying sudo, uncomment next line
#USE_SUDO=''
sudop=$([[ -n $USE_SUDO && ! -n $_SUDO ]] && echo '/usr/bin/sudo')
shopt -s expand_aliases extglob
getfacl=$(type -P getfacl)
if [[ -n $getfacl ]]; then
function _showvals_tread {
local fn=$1
local -a acc
readarray acc < <($getfacl -c $fn)
ans="$(echo ${acc[*]##*:*([^r])})"
if [[ -n $ans ]]; then
return 0
else
return 1
fi
}
else
function _showvals_tread { #use broken test...
test -r $1
}
fi
export -f _showvals_tread
#type _showvals_tread
#export sudoed_showvals='
prefx=${prefx:-""}
function showvals {
local -i width=30
for n in "${@}" ; do
[[ -z $n || $n == \* ]] && return 0
if [[ -d $n && ! -L $n ]] ; then
[[ $n == . ]] && return 0
(cd "$n"; shift;
export prefx="${prefx:-""}${prefx:+/}$n";
shift; showvals * )
elif [[ -f $n ]] ; then
if _showvals_tread "$n" ; then
read -t .1 -r val <"$n"
printf -v val -- "$val"
else
val=''
fi
key="${prefx:+$prefx/}$n:";keyl="${#key}"
[[ -n "key" ]] && keylen=${#key}
width=$((width /dev/null; then
export _SUDO="$sudop"
fi
fi
hash -r &>/dev/null
if [[ $0 =~ showvals ]] ;then
s
Re: Command substitution with null bytes generates warning
Ooops -- that script does try to run w/sudo.. forgot about that... Like I said, fast & dirty
Re: Command substitution with null bytes generates warning
On Tue, Sep 20, 2016 at 12:17:37PM -0700, L. A. Walsh wrote: > /sys/class/net/br0> /tmp/showvals > addr_assign_type: 1 > addr_len: 6 > address: 00:15:17:bf:be:b2 > /tmp/showvals: line 63: printf: `invalid format character > brforward:'`f#?? > 7ridge/ageing_time:3 > bridge/bridge_id: 8000.001517bfbeb2 So you are just "catting" each file in the current working directory? Or maybe you just want the first "line" of each file? Why not simply do: wooledg@wooledg:/sys/class/net/eth0$ for f in *; do [[ -f $f ]] || continue; read -r line < "$f" 2>/dev/null; [[ $line ]] && printf '%-25.25s %s\n' "$f:" "$line"; done addr_assign_type: 0 address: 6c:3b:e5:2b:f7:19 addr_len: 6 broadcast:ff:ff:ff:ff:ff:ff carrier: 1 carrier_changes: 2 ... Your example script didn't even *have* a command substitution with a NUL-byte-containing file in it, as far as I could tell. Of course, the script was 20 times as large & complex as it needed to be, so I might have missed some crazy dynamically-constructed thing. (If you also want it to show the contents of subdirectories, then it becomes slightly larger. I won't bother writing that one out.)
Re: Command substitution with null bytes generates warning
The script is 5 years old. I don't remember any of the design decisions that went in to it -- so you wanna pick it apart, go ahead. I am already quite able to do so myself on stuff I wrote even 3 years ago, often, so 5 years, hey, go for it. I wasn't even using aliases to help my coding back then or have "include" files. Shame bash doesn't have those natively. (include files only get included once by default and can be specified by a path relative to PATH...) BTW, did I mention that was only 1 version of that and that it was intended as a one-of hack?
