Re: current bash.git.devel segfaults on my code
i am nowhere ready, at slow thinking though i am.. does ' git bisect run ' support shell syntax, and where to enter the cleanup ' make clean ' there On Sun, Oct 24, 2021, 00:25 Chet Ramey wrote: > On 10/22/21 9:03 PM, Alex fxmbsw7 Ratchev wrote: > > ./bash --noprofile --norc > > > > bash-5.1# ./bash ../xbl5/xbl > > > > malloc: unknown:0: assertion botched > > malloc: 0x3000211ab0: allocated: last allocated from glob.c:1150 > > realloc: underflow detected; mh_nbytes out of range > > Aborting...Aborted > > bash-5.1# > > You have everything you need to automatically run `git bisect': a non- > interactive reproducer, a failing commit, and a successful commit. > > You can get the commit IDs from > > http://git.savannah.gnu.org/cgit/bash.git/log/?h=devel > > (click on an individual commit message to view the commit ID). > > Look at my previous message for a link to a quick description of how to > use `git bisect'. > > Good luck! Let us know what you find. > > Chet > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ >
Re: current bash.git.devel segfaults on my code
and whats the cmd to fetch or change a commit state instead of newest On Sun, Oct 24, 2021, 09:53 Alex fxmbsw7 Ratchev wrote: > i am nowhere ready, at slow thinking though i am.. does ' git bisect run ' > support shell syntax, and where to enter the cleanup ' make clean ' there > > On Sun, Oct 24, 2021, 00:25 Chet Ramey wrote: > >> On 10/22/21 9:03 PM, Alex fxmbsw7 Ratchev wrote: >> > ./bash --noprofile --norc >> > >> > bash-5.1# ./bash ../xbl5/xbl >> > >> > malloc: unknown:0: assertion botched >> > malloc: 0x3000211ab0: allocated: last allocated from glob.c:1150 >> > realloc: underflow detected; mh_nbytes out of range >> > Aborting...Aborted >> > bash-5.1# >> >> You have everything you need to automatically run `git bisect': a non- >> interactive reproducer, a failing commit, and a successful commit. >> >> You can get the commit IDs from >> >> http://git.savannah.gnu.org/cgit/bash.git/log/?h=devel >> >> (click on an individual commit message to view the commit ID). >> >> Look at my previous message for a link to a quick description of how to >> use `git bisect'. >> >> Good luck! Let us know what you find. >> >> Chet >> >> -- >> ``The lyf so short, the craft so long to lerne.'' - Chaucer >> ``Ars longa, vita brevis'' - Hippocrates >> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ >> >
Option for read to handle incomplete last line
Hello,
my apologies if there's a much easier solution for the following problem - in
this case please let me know!
>From time to time a run into troubles when reading a file with a while-read
>loop where the last "line" is not terminated with a newline.
I found an ugly looking solution (probably relying on undocumented features)
when reading the whole line into one variable (see below).
The attached patch for bash-5.1.8 will add an -E option to the builtin read
that will avoid the problem.
To test it run the patched bash on the following script:
>>>
input=$'Line 1\nLine 2\nIncomplete line 3'
echo "while read line"
printf '%s' "$input" | while read line; do printf ' %s\n' "$line"; done
echo "while read line || [[ \$line != '' ]]"
printf '%s' "$input" | while read line || [[ $line != '' ]]; do printf ' %s\n'
"$line"; done
echo "while read -E line"
printf '%s' "$input" | while read -E line; do printf ' %s\n' "$line"; done
echo "while read -E line with no characters between last \\n and EOF"
printf '%s\n' "$input" | sed 's/Incomplete l/L/' | while read -E line; do
printf ' %s\n' "$line"; done
<<<
The patch has not been tested intensely - first I would like to hear if I'm on
a sensible way.
Best regards
Martin
--- ../bash-5.1.8-ori/builtins/read.def 2020-06-05 19:18:28.0 +0200
+++ builtins/read.def 2021-10-23 21:23:37.067915781 +0200
@@ -22,7 +22,7 @@
$BUILTIN read
$FUNCTION read_builtin
-$SHORT_DOC read [-ers] [-a array] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
+$SHORT_DOC read [-eErs] [-a array] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
Read a line from the standard input and split it into fields.
Reads a single line from the standard input, or from file descriptor FD
@@ -40,6 +40,7 @@
-d delim continue until the first character of DELIM is read, rather
than newline
-e use Readline to obtain the line
+ -E return text between last newline and EOF
-i text use TEXT as the initial text for Readline
-n nchars return after reading NCHARS characters rather than waiting
for a newline, but honor a delimiter if fewer than
@@ -179,7 +180,7 @@
int size, nr, pass_next, saw_escape, eof, opt, retval, code, print_ps2, nflag;
volatile int i;
int input_is_tty, input_is_pipe, unbuffered_read, skip_ctlesc, skip_ctlnul;
- int raw, edit, nchars, silent, have_timeout, ignore_delim, fd;
+ int raw, edit, eof_terminates_line, nchars, silent, have_timeout, ignore_delim, fd;
int lastsig, t_errno;
int mb_cur_max;
unsigned int tmsec, tmusec;
@@ -209,6 +210,7 @@
USE_VAR(input_is_pipe);
/* USE_VAR(raw); */
USE_VAR(edit);
+ USE_VAR(eof_terminates_line);
USE_VAR(tmsec);
USE_VAR(tmusec);
USE_VAR(nchars);
@@ -229,6 +231,7 @@
i = 0; /* Index into the string that we are reading. */
raw = edit = 0; /* Not reading raw input by default. */
+ eof_terminates_line = 0;
silent = 0;
arrayname = prompt = (char *)NULL;
fd = 0; /* file descriptor to read from */
@@ -245,7 +248,7 @@
ignore_delim = nflag = 0;
reset_internal_getopt ();
- while ((opt = internal_getopt (list, "ersa:d:i:n:p:t:u:N:")) != -1)
+ while ((opt = internal_getopt (list, "eErsa:d:i:n:p:t:u:N:")) != -1)
{
switch (opt)
{
@@ -263,6 +266,9 @@
edit = 1;
#endif
break;
+ case 'E':
+ eof_terminates_line = 1;
+ break;
case 'i':
#if defined (READLINE)
itext = list_optarg;
@@ -788,7 +794,14 @@
discard_unwind_frame ("read_builtin");
- retval = eof ? EXECUTION_FAILURE : EXECUTION_SUCCESS;
+ if (!eof_terminates_line)
+{
+ retval = eof ? EXECUTION_FAILURE : EXECUTION_SUCCESS;
+}
+ else
+{
+ retval = eof && strlen (input_string) == 0 ? EXECUTION_FAILURE : EXECUTION_SUCCESS;
+}
assign_vars:
Re: Option for read to handle incomplete last line
> my apologies if there's a much easier solution for the following
> problem - in this case please let me know!
We can always define a shell function (which would work in all the
POSIX shells):
read_line() { read line || test -n "$line"; }
printf '%s' "$input" | while read_line; do printf ' %s\n'; done
> [...] (probably relying on undocumented features)
All the POSIX shells (bash, zsh, ksh, dash, etc.) behave in this way
and I think this is the behavior implied by the POSIX standard:
> XBD 3.206
> (https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206)
> 3.206 Line
> A sequence of zero or more non- characters plus a terminating
> character.
>
> XCU 4 read
> (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html)
> NAME
> read - read from standard input into shell variables
> [...]
>
> DESCRIPTION
> The read utility shall read a single logical line from standard input into
> one or more shell variables.
> [...]
>
> EXIT STATUS
> The following exit values shall be returned:
> 0
> Successful completion.
> >0
> End-of-file was detected or an error occurred.
Re: Option for read to handle incomplete last line
On Sun, Oct 24, 2021 at 12:42 PM Koichi Murase wrote: > and I think this is the behavior implied by the POSIX standard: The spec also says: > STDIN > The standard input shall be a text file. A text file is either empty, or ends with a newline. The behavior of `read' is undefined when the input is not a text file.
Re: Option for read to handle incomplete last line
On Sun, Oct 24, 2021 at 06:41:59PM +0900, Koichi Murase wrote:
> > my apologies if there's a much easier solution for the following
> > problem - in this case please let me know!
>
> We can always define a shell function (which would work in all the
> POSIX shells):
>
> read_line() { read line || test -n "$line"; }
> printf '%s' "$input" | while read_line; do printf ' %s\n'; done
You missed -r in your read command, and "$line" in your second printf.
For bash scripts using this, I'd go a little bit fancier:
read_line() {
if (($# == 0)) || [[ ${!#} = -* ]]; then
declare -n _rl_lastvar=REPLY
else
declare -n _rl_lastvar=${!#}
fi
read -r "$@" || test -n "$_rl_lastvar"
}
This gives us a lot more flexibility:
printf %s "$input" | while read_line; do printf '<%s>\n' "$REPLY"; done
printf %s "$input" | while read_line -e; do printf '<%s>\n' "$REPLY"; done
printf %s "$input" | while read_line -r line; do printf '<%s>\n' "$line"; done
printf 'foo bar\nbaz quux' | while read_line a b; do printf '<%s> <%s>\n' "$a"
"$b"; done
printf 'foo&bar\nbaz&quux' | while IFS=\& read_line a b; do printf '<%s>
<%s>\n' "$a" "$b"; done
printf 'foo\0bar\0baz' | while read_line -rd '' line; do printf '<%s>\n'
"$line"; done
printf 'foo bar\nbaz quux' | while read_line -ra arr; do printf '<%s> '
"${arr[@]}"; echo; done
This intentionally skips a trailing incomplete line that has too few
fields, as in:
unicorn:~$ printf 'foo bar\nbaz' | while read_line a b; do printf '<%s> <%s>\n'
"$a" "$b"; done
The incomplete line here only has one field, so the "lastvar" (b) is
empty, and therefore the incomplete line isn't counted. I consider
this a feature, but others may not.
Of course, it's still not perfect, as it won't handle cases like
read_line -t 5
But I refuse to try to write that. The script-writer can simply change
it to read_line -t 5 REPLY or whatever alternative they want.
Re: current bash.git.devel segfaults on my code
* Alex fxmbsw7 Ratchev [2021-10-24] [gmane.comp.shells.bash.bugs]: > and whats the cmd to fetch or change a commit state instead of newest Use this tldr client to get all the git tips you need and much else besides: https://github.com/raylee/tldr-sh-client tldr: https://tldr.sh regards, s h e h u
Re: Option for read to handle incomplete last line
Hi Greg, hi *!
> For bash scripts using this, I'd go a little bit fancier:
>
> read_line() {
> if (($# == 0)) || [[ ${!#} = -* ]]; then
> declare -n _rl_lastvar=REPLY
> else
> declare -n _rl_lastvar=${!#}
> fi
> read -r "$@" || test -n "$_rl_lastvar"
> }
Great, thanks - this solution seems to solve more real world problems than it
introduces ;-)
> This intentionally skips a trailing incomplete line that has too few
> fields, as in:
>
> ...
>
> The incomplete line here only has one field, so the "lastvar" (b) is
> empty, and therefore the incomplete line isn't counted. I consider
> this a feature, but others may not.
Yes, one might discuss this...
Nevertheless, am I right that this solution relies on an undocumented feature?
Best regards
Martin
Re: Option for read to handle incomplete last line
On Sun, Oct 24, 2021 at 04:26:44PM +0200, Martin Schulte wrote:
> Hi Greg, hi *!
>
> > For bash scripts using this, I'd go a little bit fancier:
> >
> > read_line() {
> > if (($# == 0)) || [[ ${!#} = -* ]]; then
> > declare -n _rl_lastvar=REPLY
> > else
> > declare -n _rl_lastvar=${!#}
> > fi
> > read -r "$@" || test -n "$_rl_lastvar"
> > }
>
> Great, thanks - this solution seems to solve more real world problems than it
> introduces ;-)
>
> > This intentionally skips a trailing incomplete line that has too few
> > fields, as in:
> >
> > ...
> >
> > The incomplete line here only has one field, so the "lastvar" (b) is
> > empty, and therefore the incomplete line isn't counted. I consider
> > this a feature, but others may not.
>
> Yes, one might discuss this...
>
> Nevertheless, am I right that this solution relies on an undocumented feature?
Which "undocumented" feature did you have in mind? Most things are
documented, somewhere.
Are you thinking of -d '' signifying NUL as the delimiter? Chet told
us that he supports this "happy accident" and won't take it away.
Are you thinking of the equivalence of "$array" and "${array[0]}"? I
think that's documented somewhere, but I'm not going to dig for it right
now.
Re: Option for read to handle incomplete last line
Hi Greg!
> > Nevertheless, am I right that this solution relies on an undocumented
> > feature?
>
> Which "undocumented" feature did you have in mind? Most things are
> documented, somewhere.
>
> Are you thinking of -d '' signifying NUL as the delimiter? Chet told
> us that he supports this "happy accident" and won't take it away.
>
> Are you thinking of the equivalence of "$array" and "${array[0]}"? I
> think that's documented somewhere, but I'm not going to dig for it right
> now.
Before reading the source I would never have thought that read sets variables
although it returns FAILURE.
Best regards
Martin
Re: Option for read to handle incomplete last line
On 10/24/21 12:22 PM, Martin Schulte wrote: Before reading the source I would never have thought that read sets variables although it returns FAILURE. Think of them as orthogonal conditions. `read' reads until a newline, or EOF or other error condition (ignoring timeouts or reading N characters for a minute), storing the characters it reads into the variable(s). The variable(s) continue(s) to hold the characters read. `read' returns a status dependent on the status of the last read(2). If it gets EOF (or error), the return status is > 0, but the characters read are not lost. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: current bash.git.devel segfaults on my code
On Sun, Oct 24, 2021, at 7:28 AM, Shehu Dikko wrote: > Use this tldr client to get all the git tips you need and much else besides: > > https://github.com/raylee/tldr-sh-client > > tldr: https://tldr.sh There's also #git on irc.libera.chat, if you prefer. -- vq
Fixing coproc race condition bug
#!/usr/bin/env bash
set -e -u
coproc date
sleep 1
# In bash 5.0.17, this aborts with "COPROC[0]: unbound variable"
# if the coproc exits before the main process gets here.
read -u ${COPROC[0]} line
# Discussion:
#
# To prevent this race condition, Bash must not close the coproc output pipe
# when the coprocess exits, but keep it open until the main process has had
# a chance to read buffered data.
#
# However bash must prevent unbounded leaking file descriptors/pipes
# -- see ideas below.
#
# (The coproc *input* pipe can be closed when the coprocess exits because
# it can no longer be used -- a SIGPIPE error would occur if written to.)
#
# SUGGESTED STRATEGY:
#
# Close the coproc output pipe after the main process reads EOF from it.
# --This handles all cases where the coproc writes but does not read.
#
# Close both pipes before executing another 'coproc' command with the
# same effective NAME, i.e. which will overwrite the "COPROC" array.
# --This prevents leaking fds in loops or repetitive code.
#
# Close both pipes and undef COPROC if the main process reaps a coproc
# child with 'wait'.
# --This allows scripts to explicitly release coproc resources
#
# The above is probably good enough.
#
# Another tweak would be define a "no-op" coproc command to be a special
# case which does not create new pipes, but only closes any previous pipes
# associated with the same NAME (as described above).
# e.g.
# coproc; # implied NAME is "COPROC"
# coproc NAME {}; # explicit NAME
#
# However this would be unnecessary (i.e. redundant) if doing 'wait' on
# a coproc child implicitly closes it's pipes.
#
# Note: Ignore errors when closing pipes, as the user might have
manually closed
# the fds using 'exec {varname}>&-' or similar.
#
#(END)
Re: Fixing coproc race condition bug
On 10/24/21 1:44 PM, Jim Avera wrote:
# Note: Ignore errors when closing pipes, as the user might have
manually closed
# the fds using 'exec {varname}>&-' or similar.
Actually this isn't a good idea because the fd might have been re-used
for something else; so an error should be reported to make that
bug-condition noticeable so it can be fixed.
Instead, bash should search all active CPROC variables for any file
descriptors being closed explicitly, and undef the corresponding COPROC
slot; that will prevent bash from calling close() on a previously-closed
file descriptor.
-Jim
Re: current bash.git.devel segfaults on my code
On 10/24/21 4:00 AM, Alex fxmbsw7 Ratchev wrote: and whats the cmd to fetch or change a commit state instead of newest At some point, you have to be able to read through the things people send you. For instance: https://thoughtbot.com/blog/git-bisect gives you all the information you need. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: current bash.git.devel segfaults on my code
ill check the other more complete url but yours chet u posted twice doesnt say anything about an ending command, nor multiple commands beware im noob with git On Mon, Oct 25, 2021, 00:55 Chet Ramey wrote: > On 10/24/21 4:00 AM, Alex fxmbsw7 Ratchev wrote: > > and whats the cmd to fetch or change a commit state instead of newest > > At some point, you have to be able to read through the things people > send you. > > For instance: > > https://thoughtbot.com/blog/git-bisect > > gives you all the information you need. > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ > >
