Bug#973620: bash: overflow on integer variables greater than 9223372036854775807
Control: found -1 5.2.21-2
Control: tags -1 upstream
X-Debbugs-CC: bug-bash@gnu.org
On Mon, 2 Nov 2020 16:46:14 +0100 Antonio wrote:
Dear Maintainer,
recently while I was running some tests, I ran into this strange overflow:
$ declare -i n=9223372036854775800; for((i=0; i<15; ++i)); do echo "$i -> $n";
n+=1; done
0 -> 9223372036854775800
1 -> 9223372036854775801
2 -> 9223372036854775802
3 -> 9223372036854775803
4 -> 9223372036854775804
5 -> 9223372036854775805
6 -> 9223372036854775806
7 -> 9223372036854775807
8 -> -9223372036854775808
9 -> -9223372036854775807
10 -> -9223372036854775806
11 -> -9223372036854775805
12 -> -9223372036854775804
13 -> -9223372036854775803
14 -> -9223372036854775802
The integer handled by bash is obviously very large, but I believe
that in the event of an overflow it would be better to reset the
variable and issue an error flow warning, rather than remain silent.
Bash 5.2.21 is affected by this issue:
$ declare -i n=$((2**63 - 2))
$ for i in {1..4}; do echo "$i -> $n"; n+=1; done
1 -> 9223372036854775806
2 -> 9223372036854775807
3 -> -9223372036854775808
4 -> -9223372036854775807
$ declare -i n=36893488147419103234; echo $?
0
$ echo $n
2
Would it be possible to detect this overflow (or non-representable
numbers, like in the second case) and warn about it?
Regards,
--
Gioele Barabucci
Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators
On 4/14/24 5:16 AM, Emanuel Attila Czirai wrote: Bash Version: 5.2 Patch Level: 26 Release Status: release Description: the [ test with -n or -z on a string that's only the angle bracket char followed by -a or -o operators, fails like: bash: [: syntax error: `-n' unexpected Repeat-By: $ [ -n ">" -a -n "something" ] || echo hmm bash: [: syntax error: `-n' unexpected hmm I think the part your analysis is missing is that `<' and `>' are binary operators, so this really is an ambiguous expression. POSIX test specifies what happens when there are four or fewer arguments (and the upcoming issue 8 will remove -a/-o/(/) altogether); when you have more than four you're dealing with historical algorithms. Historical parsing gave the string comparison binary operators higher precedence than the unary operators. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Examples of concurrent coproc usage?
On Sat, 13 Apr 2024, Chet Ramey wrote:
The original intent was to allow the shell to drive a long-running
process that ran more-or-less in parallel with it. Look at
examples/scripts/bcalc for an example of that kind of use.
Thanks for mentioning this example. As you understand, this model use
case does not require closing the coproc fds when finished, because they
will be closed implicitly when the shell exits. (As bcalc itself admits.)
And if the coproc is left open for the lifetime of the shell, the
alternate behavior of deferring the coproc deallocation (until both coproc
fds are closed) would not require anything extra from the user.
The bcalc example does close both coproc fds though - both at the end, and
whenever it resets. And so in this example (which as you say, was the
original intent), the user is already explicitly closing both coproc fds
explicitly; so the alternate deferring behavior would not require anything
extra from the user here either.
...
Yet another point brought to light by the bcalc example relates to the
coproc pid variable. The reset() function first closes the coproc pipe
fds, then sleeps for a second to give the BC coproc some time to finish.
An alternative might be to 'wait' for the coproc to finish (likely faster
than sleeping for a second). But you have to make and use your
$coproc_pid copy rather than $BC_PID directly, because 'wait $BC_PID' may
happen before or after the coproc is reaped and BC_PID is unset. (As the
bcalc author seems to understand.) So in general the coproc *_PID
variable only seems usable for making a copy when starting the coproc.
The info page has the following:
The process ID of the shell spawned to execute the coprocess is
available as the value of the variable 'NAME_PID'. The 'wait' builtin
command may be used to wait for the coprocess to terminate.
But it seems to me that the copy is necessary, and it is never reliable to
run 'wait $NAME_PID'. Because any time the shell is in a position to wait
for the coproc to finish, by that time it's going to be a race whether or
not NAME_PID is still set.
So this is another example for me of why it would be handy if coproc
deallocation were deferred until explicit user action (closing both coproc
fds, or unsetting the coproc variable). That way ${NAME[@]} and $NAME_PID
could reliably be used directly without having to make copies.
Anyway, just food for thought if down the line you make a shell option for
coproc deallocation behavior.
Now, if you built bash with multiple coproc support, I would have
expected you could still rig this up, by doing the redirection work
explicitly yourself. Something like this:
coproc UP { stdbuf -oL tr a-z A-Z; }
coproc DOWN { stdbuf -oL tr A-Z a-z; }
# make user-managed backup copies of coproc fds
exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}
coproc THREEWAY { tee /dev/fd/$up_w /dev/fd/$down_w; }
But the above doesn't actually work, as it seems that the coproc shell
(THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even
the user-managed ones explicitly copied with exec.
File descriptors the user saves with exec redirections beyond [0-2] are
set to close-on-exec. POSIX makes that behavior unspecified, but bash
has always done it.
Ah, ok, thanks. I believe I found where this gets set in
do_redirection_internal() in redir.c. (Whew, a big function.)
As far as I can tell the close-on-exec state is "duplicated" rather than
set unconditionally. That is, the new fd in a redirection is only set
close-on-exec if the source is. (Good, because in general I rely on
redirections to be made available to external commands.) But apparently
coproc marks its pipe fds close-on-exec, so there's no way to expose
manual copies of these fds to external commands.
So, that explains the behavior I was seeing ...
It's just a bit too bad for anyone that actually wants to do more
elaborate coproc interconnections with manual redirections, as they're
limited to shell builtins.
...
I might pose a question to ponder about this though:
With the multi-coproc support code, is it still necessary to set the
coproc pipe fds close-on-exec? (If, perhaps, they're already getting
explicitly closed in the right places.)
Because if the new coproc fds are _not_ set close-on-exec, in general that
would allow the user to do manual redirections for external commands (eg
tee(1) or paste(1)) to communicate with multiple coproc fds together.
Shells don't offer any standard way to modify the state of that flag,
but there is the `fdflags' loadable builtin you can experiment with to
change close-on-exec.
Thanks for the tip. It's nice to know there is a workaround to leave
copies of the coproc fds open across exec; though for now I will probably
continue setting up pipes in the shell by methods other than the coproc
keyword.
Che
Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators
On Mon, Apr 15, 2024, 15:08 Chet Ramey wrote: > On 4/14/24 5:16 AM, Emanuel Attila Czirai wrote: > > > Bash Version: 5.2 > > Patch Level: 26 > > Release Status: release > > > > Description: > > the [ test with -n or -z on a string that's only the angle bracket > char > > followed by -a or -o operators, fails like: > > bash: [: syntax error: `-n' unexpected > > > > Repeat-By: > > > > $ [ -n ">" -a -n "something" ] || echo hmm > > bash: [: syntax error: `-n' unexpected > > hmm > > I think the part your analysis is missing is that `<' and `>' are binary > operators, so this really is an ambiguous expression. > In my superficial report, I definitely didn't think of that. I even forgot to mention that it works when escaped like "\>" I've encountered it in the "adduser" FreeBSD sh script that runs as root, while trying to set a one char password like ">", so I thought I'd mention it here as well in case it might be helpful since I saw it happens in bash as well. Thanks, everyone. Have a great day&all! > > > POSIX test specifies what happens when there are four or fewer arguments > (and the upcoming issue 8 will remove -a/-o/(/) altogether); when you > have more than four you're dealing with historical algorithms. Historical > parsing gave the string comparison binary operators higher precedence than > the unary operators. > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ > >
Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators
On Mon, Apr 15, 2024 at 07:04:23PM +0200, Emanuel Attila Czirai wrote: > In my superficial report, I definitely didn't think of that. I even forgot > to mention that it works when escaped like "\>" > > I've encountered it in the "adduser" FreeBSD sh script that runs as root, > while trying to set a one char password like ">", so I thought I'd mention > it here as well in case it might be helpful since I saw it happens in bash > as well. Sounds like you've found a nontrivial bug in FreeBSD (in the adduser script, not in sh). I hope this gets reported and fixed, and in any case, good work and thank you.
Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators
On Mon, Apr 15, 2024 at 7:56 PM Greg Wooledge wrote: > On Mon, Apr 15, 2024 at 07:04:23PM +0200, Emanuel Attila Czirai wrote: > > In my superficial report, I definitely didn't think of that. I even > forgot > > to mention that it works when escaped like "\>" > > > > I've encountered it in the "adduser" FreeBSD sh script that runs as root, > > while trying to set a one char password like ">", so I thought I'd > mention > > it here as well in case it might be helpful since I saw it happens in > bash > > as well. > > Sounds like you've found a nontrivial bug in FreeBSD (in the adduser > script, not in sh). I hope this gets reported and fixed, and in any case, > good work and thank you. > It's nothing really, there's code in adduser that does this: [ -z ">" -a -z ">" ] && continue which errors like: [: -a: unexpected operator but the > are $passwordvars so if I want to set the password to ">" for example, I get to see that error, but still works as expected in the end.
Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators
On Mon, Apr 15, 2024 at 08:13:23PM +0200, Emanuel Attila Czirai wrote: > On Mon, Apr 15, 2024 at 7:56 PM Greg Wooledge wrote: > > Sounds like you've found a nontrivial bug in FreeBSD (in the adduser > > script, not in sh). I hope this gets reported and fixed, and in any case, > > good work and thank you. > > > It's nothing really, > > there's code in adduser that does this: > [ -z ">" -a -z ">" ] && continue > which errors like: > [: -a: unexpected operator > but the > are $passwordvars And that's a bug. That code is wrong, and it should be written this way instead: [ -z "$var1" ] && [ -z "$var2" ] && continue
Re: Examples of concurrent coproc usage?
On 4/14/24 2:43 PM, Zachary Santer wrote: On Sat, Apr 13, 2024 at 4:10 PM Chet Ramey wrote: The original intent was to allow the shell to drive a long-running process that ran more-or-less in parallel with it. Look at examples/scripts/bcalc for an example of that kind of use. $ ./bcalc equation: -12 ./bcalc: line 94: history: -1: invalid option Good catch, thanks. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ OpenPGP_signature.asc Description: OpenPGP digital signature
Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators
On Apr 15 2024, Greg Wooledge wrote: > And that's a bug. That code is wrong, and it should be written this way > instead: > > [ -z "$var1" ] && [ -z "$var2" ] && continue Or just [ -z "$var1$var2" ] && continue -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Re: Examples of concurrent coproc usage?
On Mon, Apr 15, 2024 at 1:57 PM Carl Edquist wrote:
>
> the thing discussed in my last email to the list (about
> coproc fds being set close-on-exec) makes them unusable for anything
> beyond stdin/stdout/stderr.
>
> [It might sound like an obscure use case, but once you realize what you
> can do with it, it becomes the main use case.]
>From what Chet was saying, I thought something like this would still work:
$ exec {cat}> >( cat; )
$ coproc tee { { tee /dev/fd/${cat2}; } {cat2}>&"${cat}"; }
[2] 1952
tee: /dev/fd/11: No such file or directory
Just dup another fd without using exec, and then use that. Evidently
not. Evidence for what you thought was actually going on I guess.
As much as you can just printf the same thing once for each fd, that
doesn't work super well for binary data.
I've thought about splitting and recombining pipelines like this, but
I've never had a reason to.
Re: Examples of concurrent coproc usage?
On Mon, 15 Apr 2024, Zachary Santer wrote: As much as you can just printf the same thing once for each fd, that doesn't work super well for binary data. Well, you _can_ shovel binary data too: (*) while IFS= read -rd '' X; do printf '%s\0' "$X"; done and use that pattern to make a shell-only version of tee(1) (and I suppose paste(1)). Binary data doesn't work if you're reading newline-terminated records, because you cannot store the NUL character in a shell variable. But you can delimit your records on NULs, and use printf to reproduce them. But the shell is pretty slow when you ask it to shovel data around like this. The 'read' builtin, for instance, cautiously does read(2) calls of a single byte at a time. And printf will write(2) each null-terminated recored unbuffered. Meanwhile cat(1) and tee(1) (which read and write larger blocks) are much faster. But as I said before, the main work of the shell is making it easy to set up pipelines for other (very fast) programs to pass their data around. [(*) It's slow compared to cat(1) or cp(1), but it does work. I had do resort to this once in a botched glibc system upgrade that had wrecked my libc symlinks. A broken libc symlink meant none of the external commands worked at all, so I had to fix everything with only the builtin commands of my existing live root shell. I was able to copy all the (binary) .so files to another directory (/dev/shm) using the above read/printf loop, with the names fixed, then export LD_LIBRARY_PATH to point to the new copies. That then made it possible to run external commands again (ln(1), in particular) to actually fix the symlinks.] I've thought about splitting and recombining pipelines like this, but I've never had a reason to. "If you build, it they will come." ;) Carl
