REQUEST - bash floating point math support
On Tue, Jun 4, 2024 at 4:01 PM Saint Michael wrote: > > > > > It's time to add floating point variables and math to bash. > > It just makes so much easier to solve business problems without external > calls to bc or Python. > Please let's overcome the "shell complex". Let's treat bash a real language. You want to expound on use cases? I've seen one, for myself: writing a script to bind keyboard shortcuts to, so I could change the level of screen magnification in GNOME by increments of less than 100%. The magnification factor is handled as a fractional number - 1.5, 1.75, etc. So, to change the magnification factor by increments of 0.25 or 0.5, I had to print an expression into bc in a command substitution. The math that people want to do in bash is going to be integer the vast majority of the time, though, and scripts are of course written to expect integer math. Bash could potentially detect floating point literals within arithmetic expansions and adjust the operations to use floating point math in that case. I believe C treats a literal 10 as an integer and a literal 10.0 as a floating point number, for instance, so this wouldn't really be going against the grain. For completeness, a floating point variable attribute could potentially be added. 'declare -i var' will cause var to be treated as an integer, though var can be referenced within an arithmetic expansion without this attribute. declare -f and -r (real, anybody?) are already taken for other things, so I'm not sure what the natural choice of option would be. Zack
Re: REQUEST - bash floating point math support
Date:Wed, 5 Jun 2024 08:40:51 -0400
From:Zachary Santer
Message-ID:
| The magnification factor is handled as a
| fractional number - 1.5, 1.75, etc. So, to change the magnification
| factor by increments of 0.25 or 0.5, I had to print an expression into
| bc in a command substitution.
You don't need bc for that (and if you did, for scripts, I'd generally
recommend awk - you can easily write whole programs in that instead of just
expressions) all you need is scaled integers.
Treat everything as in units of .001 (scaled by 1000) and just do normal
arithmetic (given that you are never going to want to multiply or divide
magnification factors by each other - if you do, in some other example,
you need to work just a little harder to keep the scale known and sane).
Then when you're ready to pass the result to some program that is expecting
floating point input just:
printf %d.%.3d $(( val / 1000 )) $(( val % 1000 ))
and you're done. You can use whatever scale factor allows the integer
part to avoid overflows (which isn't a limit for most applications) and
gives sufficient precision to the fractional part.
Applications that really need floating point arithmetic are very rarely
going to be appropriate to write as a shell script, if you have one of
those, you really ought to be using a language (compiled, or interpreted)
which can handle it properly, and so you get access to all the maths
functions that floating point applications tend to want to use (and bc
has only a very small fraction of those in its library - awk even less).
Also note that to actually put floating support in the shell, more is
needed than just arithmetic, you also need floating comparisons in test
(or in bash, in [[ ) and a whole bunch more odds and ends that aren't
obvious until you need them, and they're just not there (like a mechanism
to convert floats back into integers again, controlling how rounding happens).
kre
ps: as a real example, where real floating arith is needed, the following
is a fragment from a script, which (when this part is encountered) has in
variable L an ordered list of floating point numbers (which happen to be
durations in seconds, with unspecified precision and range (ie: not integers),
but that explains some constants in the code). The objective is to
find the mean of the values from that list which are > 0.75 * and < 1.25 *
of the (approx) median value (that is, ignoring extreme values, high and low)
provided that most of the values are within that range. The output is that
average (mean) as an integer number of minutes, rounded to the nearest 5.
That is, to give a very rough approximation of what the typical duration is,
in units humans more easily deal with than thousands of seconds (to N decimal
places for some N I don't know or care about). If the result is big enough
that presenting it in hrs & mins is better, other code can make that happen
(that's simple integer arithmetic).
This isn't written for bash (hence uses no bash extensions) but there's
no reason I can think of why bash couldn't run it. The script is run
quite frequently (several times every few minutes) with entirely
acceptable performance (this particular fragment usually gets executed
30 to 40 times each execution of the script - that is, to deal with that
many different lists, the actual number varies).
unset IFS
set -- $( printf '%s\n' $L | sort -n )
N=$#
if test "$N" -eq 0
then
exit
fi
shift $(( (N - 1) / 2 ))
M=$1
printf '%s\n' $L | awk -v M="$M" '
BEGIN {
L = M * 0.75
U = M * 1.25
N=0
O=0
T=0
}
NR == 1 {
if ($1 < L || $1 > U) {
A = ($1 + 30) / 60
A = int( (A + 2.5) / 5 ) * 5
printf "x %.0f", A
exit
}
}
{
if ($1 < L || $1 > U ) {
O = O + 1
next
}
T = T + $1
N = N + 1
}
EN
Re: REQUEST - bash floating point math support
On Wed, Jun 05, 2024 at 09:57:26PM +0700, Robert Elz wrote: > Also note that to actually put floating support in the shell, more is > needed than just arithmetic, you also need floating comparisons in test > (or in bash, in [[ ) and a whole bunch more odds and ends that aren't > obvious until you need them, and they're just not there (like a mechanism > to convert floats back into integers again, controlling how rounding happens). Ironically, that last one is the one we already *do* have. hobbit:~$ printf '%.0f\n' 11.5 22.5 33.5 12 22 34 As long as you're OK with "banker's rounding", printf does it.
Re: REQUEST - bash floating point math support
2024年6月5日(水) 21:41 Zachary Santer : > Bash could potentially detect floating point literals within > arithmetic expansions and adjust the operations to use floating point > math in that case. [...] ksh and zsh are already behaving in that way, and if Bash would support the floating-point arithmetic, Bash should follow their behavior. Actually, there's already a stub for the double-precision floating-point number for shell-variable values in variables.h [1], though it seems to be there for more than 20 years since 2.05b. [1] https://git.savannah.gnu.org/cgit/bash.git/tree/variables.h?h=devel&id=dbb48b978671394bcb33c9f34656a9aadf40a318#n75 > For completeness, a floating point variable attribute could > potentially be added. 'declare -i var' will cause var to be treated as > an integer, though var can be referenced within an arithmetic > expansion without this attribute. declare -f and -r (real, anybody?) > are already taken for other things, so I'm not sure what the natural > choice of option would be. ksh uses -E for the double-precision floating-point numbers, and `-E' is not used by Bash. By the way, people are writing shell libraries to perform the floating-point arithmetic in pure Bash, though I've never tried them and am unsure about their quality, performance, and API experience. https://github.com/clarity20/shellmath https://github.com/m1ndflay3r/bash-float We can also do the fixed-point arithmetic more easily and efficiently (except for special functions such as sin, cos, tan, exp, log, etc.). These are examples for specific purposes without needing precisions (and thus not all the operations are implemented): https://github.com/aneeshdurg/bash-raytracer/blob/main/math.sh https://github.com/akinomyoga/blesh-contrib/blob/master/colorglass.bash#L247-L381 -- Koichi
Re: REQUEST - bash floating point math support
Date:Wed, 5 Jun 2024 11:09:45 -0400
From:Greg Wooledge
Message-ID:
| > to convert floats back into integers again, controlling how
| > rounding happens).
|
| Ironically, that last one is the one we already *do* have.
Yes, I know about printf (and while POSIX doesn't require that floats
be supported in printf(1), all the implementations I am aware of, do) but:
| As long as you're OK with "banker's rounding"
that is expressly not "controlling how rounding happens" - applications
dealing with floats sometimes want round to nearest (which is what is
happening in printf - the tiebreaker algorithm when up or down are equally
near might be relevant, but usually isn't), others want round down (towards 0,
that is, simply discard the fractional part - that's easy in sh with ${v%.*},
though you might need to deal with "-0" as the result - others round up
(away from 0) (harder in sh, but achievable), others want round towards
minint (ie: positives round down, negatives round up), and I suppose round
towards maxint (the opposite) might occur sometimes too, though I don't
think I've ever seen a use for that one.
Most of this can be done, with some difficulty sometimes, but they
really ought to be done with arithmetic functions - in fact, it is
hard to imagine any real floating point work that can be done without
the ability to define functions that can be used in an arithmetic
context.
My whole point is that as simple as it seems to "just add float support
to shell arithmetic" might seem, it wouldn't end there, it is almost
certainly better to just not go there. There are plenty of other
languages that can work with floats - not everything needs to be a shell
script using only shell primitives. Use the appropriate tool, don't
just pick one, and make it try to be everything.
kre
I have considered all this as I once thought of adding float arith to
the shell I maintain, and it took very little of this kind of thought
process to abandon that without ever writing a line of code for it.
Setting HISTTIMEFORMAT is causing problems in >=bash-5.2
Hello!
I would like to report an issue with bash version >=5.2.
For years, I have had the following line in my ~/.bashrc:
export HISTTIMEFORMAT="[$(tput setaf 6)%F %T$(tput sgr0)]: " # colorful date
This worked perfectly up to and including bash version 5.1.0(16).
However, since bash version 5.2.0(2), I have observed the following problem:
When I connect via SSH to a system with bash >=5.2.0 and execute
"shorewall compile" directly (which generates a shell script in
/var/lib/shorewall/firewall), the script becomes corrupted:
# /bin/sh /var/lib/shorewall/firewall help
/var/lib/shorewall/firewall: line 2239: syntax error near unexpected token `('
/var/lib/shorewall/firewall: line 2239: `GCC_SPECS='export
HISTTIMEFORMAT=$'[\E[36m%F %T\E(B\E[m]: '''
This does not occur when I run "shorewall compile" via tty or, for
example, in a tmux session. It only happens when I compile the firewall
script directly via SSH.
When I compare the script generated with bash <5.2 to that generated
with bash >=5.2, this is the difference:
# diff -u /var/lib/shorewall/.safe /var/lib/shorewall/.restart
--- /var/lib/shorewall/.safe2024-06-03 02:53:02.687389414 +0200
+++ /var/lib/shorewall/.restart 2024-06-03 02:53:02.564050925 +0200
@@ -1,6 +1,6 @@
#!/bin/sh
#
-# Compiled firewall script generated by Shorewall 5.2.8 - Mon Jun 3 02:50:56
AM CEST 2024
+# Compiled firewall script generated by Shorewall 5.2.8 - Mon Jun 3 02:53:02
AM CEST 2024
#
# (c) 1999-2019 - Tom Eastep (teas...@shorewall.net)
#
@@ -2234,6 +2234,10 @@
IP=ip
TC=tc
IPSET=ipset
+#
+# From the params file
+#
+GCC_SPECS='export HISTTIMEFORMAT=$'[\E[36m%F %T\E(B\E[m]: ''
g_stopping=
The responsible function is a Perl function, which can be found here:
https://gitlab.com/shorewall/code/-/blob/2673e6e60cce99b4d9723ecd13d5d6f17e27d390/Shorewall/Perl/Shorewall/Config.pm#L6051
I have stepped through the function with the Perl debugger but still do
not understand what is going wrong (the below for full debug output):
%ENV contains GCC_SPECS -- but as an empty variable. When I create
another empty environment variable like "GZZ_TEST", the error shifts,
i.e., the error message then reads:
/var/lib/shorewall/firewall: line 2239: `GZZ_TEST='export
HISTTIMEFORMAT=$'[\E[36m%F %T\E(B\E[m]: '''
"export HISTTIMEFORMAT..." is not visible within %ENV.
The variable is only written into the script because no such variable
with this value exists in the current environment (lines 6081-6085). Of
course this seems to reveal a quoting issue in the Shorewall code.
However, I'll address this later in the Shorewall project. The
underlying problem is hopefully not related to Shorewall.
If I remove the mentioned line from my .bashrc, the problem does not
occur anymore.
As mentioned, this problem does not occur with bash versions <5.2. In
these versions, the last variable starting with G is empty, exactly as
defined -- empty.
So why does "export HISTTIMEFORMAT..." appear within Perl at all? What
has changed in bash >=5.2? Is this a bug? Am I doing something wrong and
was just lucky that it worked that way for so long?
Thank you!
PS: Here's the hopefully related Perl debug output (I set a break point
in export_params function after line 6062):
# shorewall compile -d
Compiling using Shorewall 5.2.8...
Loading DB routines from perl5db.pl version 1.77
Editor support available.
Enter h or 'h h' for help, or 'man perldebug' for more help.
main::(/usr/share/shorewall/compiler.pl:85):
85: my $export= 0;
DB<1> c
Processing /etc/shorewall/params ...
Processing /etc/shorewall/shorewall.conf...
Loading Modules...
Compiling /etc/shorewall/zones...
Compiling /etc/shorewall/interfaces...
Determining Hosts in Zones...
Locating Action Files...
Compiling /etc/shorewall/policy...
Running /etc/shorewall/initdone...
Adding Anti-smurf Rules
Adding rules for DHCP
Compiling TCP Flags filtering...
Compiling Kernel Route Filtering...
Compiling Martian Logging...
Compiling MAC Filtration -- Phase 1...
Compiling /etc/shorewall/rules...
Compiling /etc/shorewall/conntrack...
Compiling MAC Filtration -- Phase 2...
Applying Policies...
Generating Rule Matrix...
Optimizing Ruleset...
Shorewall::Config::export_params(/usr/share/shorewall/Shorewall/Config.pm:6065):
6065: if ( $shell == BASH ) {
DB<1> p $param
CCACHE_DIR
DB<2> c
Shorewall::Config::export_params(/usr/share/shorewall/Shorewall/Config.pm:6065):
6065: if ( $shell == BASH ) {
DB<2> p $param
CONFIG_PROTECT
DB<3> p $value
/usr/share/gnupg/qualified.txt
DB<4> c
Shorewall::Config::export_params(/usr/share/shorewall/Shorewall/Config.pm:6065):
6065: if ( $shell == BASH ) {
DB<4> p $param
CONFIG_PROTECT_MASK
DB<5> c
Shorewall::Config::export_params(/usr/share/shorewall/Shorewall/Config.pm:6065):
6065: if ( $shell == BASH ) {
DB<5> p $param
EDITOR
DB<6> p $value
/bin/nano
DB<7> c
Shorew
Re: REQUEST - bash floating point math support
the most obvious use of floating variables would be to compare
balances and to branch based on if a balance is lower than a certain
value
I use:
t=$(python3 -c "import math;print($balance > 0)")
and the
if [ "$t" == "False" ];then
echo "Result <= 0 [$t] Client $clname $clid Balance $balance"
fi
There must be a solution without Awk or Python or BC. Internal to bash
On Wed, Jun 5, 2024 at 11:49 AM Robert Elz wrote:
>
> Date:Wed, 5 Jun 2024 11:09:45 -0400
> From:Greg Wooledge
> Message-ID:
>
> | > to convert floats back into integers again, controlling how
> | > rounding happens).
> |
> | Ironically, that last one is the one we already *do* have.
>
> Yes, I know about printf (and while POSIX doesn't require that floats
> be supported in printf(1), all the implementations I am aware of, do) but:
>
> | As long as you're OK with "banker's rounding"
>
> that is expressly not "controlling how rounding happens" - applications
> dealing with floats sometimes want round to nearest (which is what is
> happening in printf - the tiebreaker algorithm when up or down are equally
> near might be relevant, but usually isn't), others want round down (towards 0,
> that is, simply discard the fractional part - that's easy in sh with ${v%.*},
> though you might need to deal with "-0" as the result - others round up
> (away from 0) (harder in sh, but achievable), others want round towards
> minint (ie: positives round down, negatives round up), and I suppose round
> towards maxint (the opposite) might occur sometimes too, though I don't
> think I've ever seen a use for that one.
>
> Most of this can be done, with some difficulty sometimes, but they
> really ought to be done with arithmetic functions - in fact, it is
> hard to imagine any real floating point work that can be done without
> the ability to define functions that can be used in an arithmetic
> context.
>
> My whole point is that as simple as it seems to "just add float support
> to shell arithmetic" might seem, it wouldn't end there, it is almost
> certainly better to just not go there. There are plenty of other
> languages that can work with floats - not everything needs to be a shell
> script using only shell primitives. Use the appropriate tool, don't
> just pick one, and make it try to be everything.
>
> kre
>
> I have considered all this as I once thought of adding float arith to
> the shell I maintain, and it took very little of this kind of thought
> process to abandon that without ever writing a line of code for it.
>
>
>
Re: REQUEST - bash floating point math support
On Wed, Jun 05, 2024 at 01:31:20PM -0400, Saint Michael wrote: > the most obvious use of floating variables would be to compare > balances and to branch based on if a balance is lower than a certain > value > I use: > t=$(python3 -c "import math;print($balance > 0)") > and the > if [ "$t" == "False" ];then > echo "Result <= 0 [$t] Client $clname $clid Balance $balance" > fi > There must be a solution without Awk or Python or BC. Internal to bash The example you show is just comparing to 0, which is trivial. If the $balance variable begins with "-" then it's negative. If it's "0" then it's zero. Otherwise it's positive. For comparing two arbitrary variables which contain strings representing floating point numbers, you're correct -- awk or bc would be the minimal solution.
Re: REQUEST - bash floating point math support
I think that we should do this in the shell. I mean. It will get done at some point, in the next decades or centuries. Why not do it now? Let's compile some C library or allow inline C On Wed, Jun 5, 2024, 2:12 PM Greg Wooledge wrote: > On Wed, Jun 05, 2024 at 01:31:20PM -0400, Saint Michael wrote: > > the most obvious use of floating variables would be to compare > > balances and to branch based on if a balance is lower than a certain > > value > > I use: > > t=$(python3 -c "import math;print($balance > 0)") > > and the > > if [ "$t" == "False" ];then > > echo "Result <= 0 [$t] Client $clname $clid Balance $balance" > > fi > > There must be a solution without Awk or Python or BC. Internal to bash > > The example you show is just comparing to 0, which is trivial. If > the $balance variable begins with "-" then it's negative. If it's "0" > then it's zero. Otherwise it's positive. > > For comparing two arbitrary variables which contain strings representing > floating point numbers, you're correct -- awk or bc would be the minimal > solution. > >
Re: REQUEST - bash floating point math support
Date:Wed, 5 Jun 2024 13:31:20 -0400
From:Saint Michael
Message-ID:
| the most obvious use of floating variables would be to compare
| balances and to branch based on if a balance is lower than a certain
| value
In addition to what Greg suggested, for the very simple case you
outlined ---
That's a perfect case for scaled integers - no-one ever deals with
fractions of cents in this kind of thing (a bank won't ever tell you
that your balance is $5678.17426 for example, even if the interest
calculations computed accurately might arrive at that number.)
So, just do the calculations/tests using cents instead of dollars and
fractional dollars (aka cents) (or pounds & pence, or Euros and whatever,
or ...)
So just
dbal=${balance%.*}
cbal=${balance#*.}
case ${cbal} in ?) cbal=${cbal}0;; esac # just in case balance is 100.5
cents=${dbal}${cbal}
If you need to deal with European notations, use [.,] in each place
instead of just . (From the way you used inserted $balance into the
python script, it is clear there are no grouping characters in the
value - ie: not 1,234.56 or 1 234,56 or similar.)
If sometimes the balance has no cents, either ends in '.' or no '.' at all,
rather than explicitly having .00 at the end, a few extra lines will handle
that as well. You could also easily add more error/sanity checking on
the input values, if needed.
Then you can just use $cents and do integer arithmetic everywhere (and to
print it, if unchanged just use $balance, otherwise use the technique I
described in the previous message, except the scale factor is just 100,
so you want %.2d to print ${cents}%100 - etc).
Money is one of the most obvious cases where floating point isn't needed.
Not only because it is easy to work using the least valuable currency unit,
as above, but also because the calculations tend to be trivial, add, subtract,
and some simple multiplication (add tax at N% or whatever), and that's
about it. No-one takes square roots, or ln() or whatever of their bank
balance, or required credit card payment. But start doing floating point
and you'll soon get people saying "what do you mean I need to use bc to
calculate the natural log of ...? Why can't I just do that in bash?".
And of course, soon to be followed by "now we have floating point numbers,
we need complex numbers and arithmetic as well".
Further, if you're going to use something like python in your
script to work on some aspect, why not just write (almost) all
of the code in python? Sometimes you can make a program in
another language (like python) but where some operations there
are harder to achieve than in shell, where it is reasonable to
combine the two together - have the shell script do whatever is
needed to make it easy for python to do most of the work, then
when that's done, the script can do any required cleanup, etc.
Sometimes even several small python programs linked together by
a shell script might work.
[I wouldn't do that, I detest python, but I would use other similar
systems for the same effect.]
And last "There must be" is not an argument for absolutely anything
unless you can demonstrate why that is so. Nothing you cannot
prove "must be".
kre
ps: and why would floating point be the thing to consider adding,
why not URLs as file names? Surely bash should be able to do
cat < https://some.host/path/to/file > mailto:bug-bash@gnu.org
Why do we need external programs for network access - something far
more likely for a shell script to want to do that real number arithmetic.
And of course, no, this is not a serious suggestion.
Re: REQUEST - bash floating point math support
Le 05/06/2024 à 17:09, Koichi Murase écrivait : 2024年6月5日(水) 21:41 Zachary Santer : Bash could potentially detect floating point literals within arithmetic expansions and adjust the operations to use floating point math in that case. [...] ksh and zsh are already behaving in that way, and if Bash would support the floating-point arithmetic, Bash should follow their behavior. Bash isn't even consistent with the floating point data/input format when using printf '%f\n' "$float_string" as it depends on LC_NUMERIC locale. LC_MESSAGES=C LC_NUMERIC=fr_FR.UTF8 printf '%f\n' 3.1415 bash: printf: 3.1415: invalid number 3,00 Chet explained it is because Bash is following the POSIX norm and C printf rules. Now imagine if Bash introduce floating-point support, you write a Bash script with decimal point floating-point numbers and your script is incompatible with systems whose locale use a decimal comma instead. -- Léa Gris
