"Neagtive overflow" & other flaws in binary operations.
Hello,
Recently I've tried to use BASH to do some binary operations, that task turned
out to be far more difficult, that I could ever imagined.
First of all, I've found out that 2 shifted to the left edge of boundary
(2**63) produces as a result a negative number "-9223372036854775808" ("funny"
thing is, when you substract "1" from this value, it gives positive value as a
result), this problem is of course related to "u2 code", but still it's an
error, which can cause various difficult to perceive problems during script
execution - i.e. PERL deals with it by substituting integer value, with
floating point value, which at least has a proper sign.Another thing, which
doesn't behave exactly the way it should, is bit shifting. While shifting
positive value to the right gives 0 at the end, then shifting negative value
gives "-1" at the end, again it's related to "u2 code". I.e. try to shift "1",
63 times to the left (it will turn into negative value), then shift it back to
the right, you'll never guess it won't evaluate to "0" but to "-1"! This
is error in "logic", which lays beneath this whole idea. Bit operations
shouldn't be sensitive to "u2 coding", as coding is on higher abstract level
then any of bit operations, which deals with "raw bits" on lowest
level!Genrally speaking, there is significant flaw in BASH, if it comes to any
binary operations - with whole do respect, your whole idea is one big
misconception. You've introduced all kinds of bit operators but they "works"
only on numbers, if anyone try to use it on character, they won't work properly
(those are some other "magical bits"?!). BASH for some reason is concentrated
almost solely on strings (almost like PERL), reading & writing characters
is relatively easy thing to be done but doing same thing with "raw data"
(binary data) requires making crude workarounds (like using coreutils "od" to
read data as numbers, not as characters & ab-using "printf" to later write
them to the file), while it should be possible to do it easily by using only
read & write commands - data stored in memory or on disc (or any other
medium), are just bunch of bits, meaning of those bits is determined later,
during processing (like in "C" language, where type "char" is in fact numeric
& it's interpretation by some commands gives it special meaning) &
programmer should be being able to decide the way he want to perceive those
data!I can see two solutions (one dosen't exclude the other) to this problem.
First one is to add commands which allows to read & write data the way,
they could be processed always as numbers.
The other is to introduce user defined data types or at least add ability to
control "data perception" (if they should be perceived as numbers or characters
or some other way...i.e. references). Giving a user/programmer an ability to
control "general" data type, will makes various operations easier, its results
more predictable & the best thing is, it can be introduced the way, that
doesn't require to change whole logic of BASH or even force scripts to be
re-written - it might be an extension, which user uses only if necessary, if
not the BASH behave as it is now.
To be honest, I would never dream (or I rather should write "to have a
nightmare") to use BASH to do binary ops but unfortunatelly I've faced task
which requires me to relay on it (& thankfully some of "coreutils"), I'm
able to overcome all flaws mentioned above, using very coarse workarounds
(essentially "hacking" some commands), to do the things, that should have been
done easily, with full support of the enviroment.
Re: "Neagtive overflow" & other flaws in binary operations.
On Sun, Mar 12, 2017 at 4:15 AM, wrote:
> Hello,
>
> Recently I've tried to use BASH to do some binary operations, that task
> turned out to be far more difficult, that I could ever imagined.
Bash has limited support for arithmetic operations on fixed width
signed integer numbers. Furthermore, it does not support floating
point numbers. I'd advise against using Bash for serious mathematical
tasks.
Here's the official Bash documentation on the support for shell
arithmetics:
https://www.gnu.org/software/bash/manual/html_node/Shell-Arithmetic.html#Shell-Arithmetic
Evaluation is done in *fixed-width integers with no check for
overflow*, though
division by 0 is trapped and flagged as an error. The operators and their
precedence, associativity, and values are the same as in the C language. The
following list of operators is grouped into levels of equal-precedence
operators. The levels are listed in order of decreasing precedence.
If you're interested in the actual implementation of the left and
right shift operations, you can read:
http://git.savannah.gnu.org/cgit/bash.git/tree/expr.c?h=devel#n794
Which as you can tell, just performs the C operations on intmax_t values.
>
> First of all, I've found out that 2 shifted to the left edge of boundary
> (2**63) produces as a result a negative number "-9223372036854775808"
> ("funny" thing is, when you substract "1" from this value, it gives positive
> value as a result), this problem is of course related to "u2 code" [...]
dualbus@debian:~$ echo $((2<<64))
2
dualbus@debian:~$ echo $((2<<63))
0
dualbus@debian:~$ echo $((2<<62))
-9223372036854775808
Yes, Bash does that. C does too:
dualbus@debian:~$ cat main.c
#include
#include
int main() {
printf("%ld\n", (int64_t)2 << (int64_t)62);
}
dualbus@debian:~$ gcc -o main main.c; ./main
-9223372036854775808
This is what happens when you perform bitwise shifts on signed fixed
width integers.
> Another thing, which doesn't behave exactly the way it should, is bit
> shifting. While shifting positive value to the right gives 0 at the end,
> then shifting negative value gives "-1" at the end, again it's related to
> "u2 code". I.e. try to shift "1", 63 times to the left (it will turn into
> negative value), then shift it back to the right, you'll never guess it
> won't evaluate to "0" but to "-1"! This is error in "logic", which lays
> beneath this whole idea. Bit operations shouldn't be sensitive to "u2
> coding", as coding is on higher abstract level then any of bit operations,
> which deals with "raw bits" on lowest level!
What is this "u2 code" you keep referring to?
Bash is not the right tool for the job. Please use a language that has
full support for integer and floating point arithmetic. Bash will let
values silently overflow, and do all the kinds of crazy things you
found out. But that's okay: it behaves as documented (i.e it's not
meant to be used in the way you're trying to).
> I can see two solutions (one dosen't exclude the other) to this problem.
Feel free to submit patches. Start by looking into
http://git.savannah.gnu.org/cgit/bash.git/tree/expr.c?h=devel (the
current implementation of arithmetical evaluation) and
http://git.savannah.gnu.org/cgit/bash.git/tree/examples/loadables/hello.c?h=devel
which should give you a rough idea of what you need to do to implement
a new builtin shell command that allows for arbitrary precision
integer arithmetic which fulfills your needs.
PS-embedded comsub `\w' evaluation doesn't replace $HOME with ~
For example:
$ PS1='\w $(echo \w) $ '
~ /home/levit $
Obviously, this is easy enough to work around in most cases by
replacing \w with ${PWD/#$HOME/~}. I had a silly application for
calculating prompt length and this was throwing it off, but this
behavior has been there since at least 2.05b so not sure if it's worth
changing..
\! and \# in PS1 vs PS2 vs PS4, PS0 and ${var@P}
These expansions both seem to get values that are inconsistent when
expanded in the various prompt strings.
For example, starting with the following:
PS0='PS0 \! \# _ \n'
PS1='PS1 \! \# $ '
PS2='PS2 \! \# > '
PS4='PS4 \! \# + '
PSV='PSV \! \# V '
set -x
The prompts evaluate like so:
PS1 529 45 $ echo \
PS2 530 45 > ${PSV@P}
PS0 530 46 _
PS4 530 46 + echo PSV 530 46 V
PSV 530 46 V
PS1 530 46 $
i.e. \! is increased as soon as the first line is read, even though
the subsequent line(s) are to become part of the history entry
associated with the number displayed in PS1.
Bash Manual inconsistency regarding PS* in which expansions apply
The manual says [1]: In addition, the following table describes the special characters which can appear in the prompt variables PS1 to PS4: ..which misses PS0 (and seems to imply that PS3 gets expanded in this fashion as well). [1] https://www.gnu.org/software/bash/manual/bash.html#Controlling-the-Prompt
Re: PS-embedded comsub `\w' evaluation doesn't replace $HOME with ~
On Sun, 12 Mar 2017, Grisha Levit wrote:
For example:
$ PS1='\w $(echo \w) $ '
~ /home/levit $
This seems to work as documented, for the same reason this happens:
╶➤ echo ~
/home/rob
The expansion order is perhaps non-obvious: the prompt string backslash
escapes are replaced before the string is subject to any other expansions.
Obviously, this is easy enough to work around in most cases by
replacing \w with ${PWD/#$HOME/~}. I had a silly application for
calculating prompt length and this was throwing it off, but this
behavior has been there since at least 2.05b so not sure if it's worth
changing..
Doing anything that depends on calculating (any part of) the prompt length
at expansion time in PS1 is... complicated.
╶➤ echo ${#PS1}
1485
I have on occasion wondered how difficult it'd be to make ${#...}
applicable to any arbitrary expansion, but I also can't think of another
use case...
-Rob
exiting i-search using end-of-line on multi-line prompt in multi-byte locale
As a minimal example: LC_CTYPE=en_US.UTF-8 PS1='\n\[\e[0m\]' # put something into history: echo foo # ^R search for something that should put point in the middle of the string (reverse-i-search)`fo': echo foo # attempt to exit with end-of-line (^E) echo foo ^<---point is here Trying to move beyond point does not work. Also it seems that the prompt is re-drawn after the text? For example, setting the prompt as: PS1='\n\[\e[0m\e[K\]' and repeating the above puts point in the middle of a blank line.
