Re: read command sometimes misses newline on timeout

[alex xmb sw ratchev]

well try remove the -n in echo

On Monday, October 7, 2024, Thomas Oettli  wrote:

> Sorry Alex, I don't understand exactly what you mean.
> Here is the Test-Script again with some comments:
>
> function reader() {
>   local buf line
>   while :; do
> read -t .01 buf  # try to read line to $buf with 
> timeout
> rc=$?
> if (( rc == 0 )); then   # got a full line or the rest of a 
> partial line, append $buf to $line
>   line+=$buf
> elif (( rc > 128 )); then# ran into timeout
>   line+=$buf # maybe received partial line, 
> append $buf to $line and continue reading
>   continue
> fi
>
> # at this point, the content of $line should always be "TEST"
>
> [[ $line != TEST ]] && echo Invalid line: $line && exit
> echo OK
> line=""  # prepare to read next line, set 
> $line to empty string
>   done
> }
> reader < <(
>   while :; do
> echo -n TEST
> sleep .00$(($RANDOM%10))
> echo
>   done
> )
>
>
> Could you please explain to me, where exactly the mistake is? I don't know
> if you ran the script by yourself, but on all of my machines it exits after
> just a few seconds
> with the following output:
>
> Invalid line: TESTTEST
> --
> *Von:* alex xmb sw ratchev 
> *Gesendet:* Montag, 7. Oktober 2024 16:41
> *An:* Thomas Oettli 
> *Cc:* bug-bash@gnu.org ; chet.ra...@case.edu <
> chet.ra...@case.edu>
> *Betreff:* [EXT] Re: read command sometimes misses newline on timeout
>
> CAUTION: This email originated from outside the SFS organization. Do not
> follow guidance, click links or open attachments unless you recognize the
> sender and know the content is safe.
> MAIL FROM:  <@fxmb...@gmail.com> Report as Spam / Check Mail
> 
>
> there is a case , u [[ $readreply ]] after read
>
> On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU
> Bourne Again SHell  wrote:
>
> I agree with you, but it should never happen that read returns timeout,
> also returns the full line and has already read the newline character.
> If that happens, there is no way for the script to decide what to do.
> Please see the provided test script, it showcases the error.
>
> If I did a mistake there, I would gladly change it. But I currently don't
> see any way how to handle this properly from the script side.
>
> Please also see the answer from Martin D Kealey, I think he is on to
> something:
> https://lists.gnu.org/archive/html/bug-bash/2024-10/msg7.html
>
>
>
>
> On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
> SHell wrote:
>
> > Bash Version: 5.2
> > Patch Level: 26
> > Release Status: release
> >
> > Description:
> > I have tried to write a bash script that asynchronously reads from a
> pipe (line by line) with the help of "read -t".
> > If the timeout occurs in just the right moment, read returns the full
> line, but the return code says timeout (rc > 128).
>
> If the read command times out, it always returns > 128, so if you have a
> return code in that range, you can assume read timed out and react
> accordingly.
>
> > Therefor it is not possible to know if a full line was returned or not.
>
> When read times out, it always returns what it read before the timeout in
> the buffer, so you don't lose any data. Whether or not that's a `full line'
> is up to timing, and it's up to the script to decide how to cope with it.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/
>
>

AW: Re: read command sometimes misses newline on timeout

[Thomas Oettli via Bug reports for the GNU Bourne Again SHell]

I agree with you, but it should never happen that read returns timeout, also 
returns the full line and has already read the newline character.
If that happens, there is no way for the script to decide what to do. Please 
see the provided test script, it showcases the error.

If I did a mistake there, I would gladly change it. But I currently don't see 
any way how to handle this properly from the script side.

Please also see the answer from Martin D Kealey, I think he is on to something:
https://lists.gnu.org/archive/html/bug-bash/2024-10/msg7.html




On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
SHell wrote:

> Bash Version: 5.2
> Patch Level: 26
> Release Status: release
>
> Description:
> I have tried to write a bash script that asynchronously reads from a pipe 
> (line by line) with the help of "read -t".
> If the timeout occurs in just the right moment, read returns the full line, 
> but the return code says timeout (rc > 128).

If the read command times out, it always returns > 128, so if you have a
return code in that range, you can assume read timed out and react
accordingly.

> Therefor it is not possible to know if a full line was returned or not.

When read times out, it always returns what it read before the timeout in
the buffer, so you don't lose any data. Whether or not that's a `full line'
is up to timing, and it's up to the script to decide how to cope with it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/

Re: read command sometimes misses newline on timeout

[alex xmb sw ratchev]

there is a case , u [[ $readreply ]] after read

On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU
Bourne Again SHell  wrote:

> I agree with you, but it should never happen that read returns timeout,
> also returns the full line and has already read the newline character.
> If that happens, there is no way for the script to decide what to do.
> Please see the provided test script, it showcases the error.
>
> If I did a mistake there, I would gladly change it. But I currently don't
> see any way how to handle this properly from the script side.
>
> Please also see the answer from Martin D Kealey, I think he is on to
> something:
> https://lists.gnu.org/archive/html/bug-bash/2024-10/msg7.html
>
>
>
>
> On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
> SHell wrote:
>
> > Bash Version: 5.2
> > Patch Level: 26
> > Release Status: release
> >
> > Description:
> > I have tried to write a bash script that asynchronously reads from a
> pipe (line by line) with the help of "read -t".
> > If the timeout occurs in just the right moment, read returns the full
> line, but the return code says timeout (rc > 128).
>
> If the read command times out, it always returns > 128, so if you have a
> return code in that range, you can assume read timed out and react
> accordingly.
>
> > Therefor it is not possible to know if a full line was returned or not.
>
> When read times out, it always returns what it read before the timeout in
> the buffer, so you don't lose any data. Whether or not that's a `full line'
> is up to timing, and it's up to the script to decide how to cope with it.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/
>
>

AW: Re: read command sometimes misses newline on timeout

[Thomas Oettli via Bug reports for the GNU Bourne Again SHell]

I know that it works in that case.
This is just an example that tries to force read into the same situatiation 
that I hit in a real world example in which I don't control the input.

Von: alex xmb sw ratchev 
Gesendet: Montag, 7. Oktober 2024 19:17
An: Thomas Oettli 
Cc: bug-bash@gnu.org ; chet.ra...@case.edu 

Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmb...@gmail.com>Report as Spam / Check 
Mail

well try remove the -n in echo

On Monday, October 7, 2024, Thomas Oettli 
mailto:thomas.oet...@sfs.com>> wrote:

Sorry Alex, I don't understand exactly what you mean.
Here is the Test-Script again with some comments:

function reader() {
  local buf line
  while :; do
read -t .01 buf  # try to read line to $buf with timeout
rc=$?
if (( rc == 0 )); then   # got a full line or the rest of a 
partial line, append $buf to $line
  line+=$buf
elif (( rc > 128 )); then# ran into timeout
  line+=$buf # maybe received partial line, append 
$buf to $line and continue reading
  continue
fi

# at this point, the content of $line should always be "TEST"

[[ $line != TEST ]] && echo Invalid line: $line && exit
echo OK
line=""  # prepare to read next line, set $line 
to empty string
  done
}
reader < <(
  while :; do
echo -n TEST
sleep .00$(($RANDOM%10))
echo
  done
)

Could you please explain to me, where exactly the mistake is? I don't know if 
you ran the script by yourself, but on all of my machines it exits after just a 
few seconds
with the following output:

Invalid line: TESTTEST

Von: alex xmb sw ratchev mailto:fxmb...@gmail.com>>
Gesendet: Montag, 7. Oktober 2024 16:41
An: Thomas Oettli mailto:thomas.oet...@sfs.com>>
Cc: bug-bash@gnu.org 
mailto:bug-bash@gnu.org>>; 
chet.ra...@case.edu 
mailto:chet.ra...@case.edu>>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmb...@gmail.com>Report as Spam / Check 
Mail

there is a case , u [[ $readreply ]] after read

On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU Bourne 
Again SHell mailto:bug-bash@gnu.org>> wrote:
I agree with you, but it should never happen that read returns timeout, also 
returns the full line and has already read the newline character.
If that happens, there is no way for the script to decide what to do. Please 
see the provided test script, it showcases the error.

If I did a mistake there, I would gladly change it. But I currently don't see 
any way how to handle this properly from the script side.

Please also see the answer from Martin D Kealey, I think he is on to something:
https://lists.gnu.org/archive/html/bug-bash/2024-10/msg7.html




On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
SHell wrote:

> Bash Version: 5.2
> Patch Level: 26
> Release Status: release
>
> Description:
> I have tried to write a bash script that asynchronously reads from a pipe 
> (line by line) with the help of "read -t".
> If the timeout occurs in just the right moment, read returns the full line, 
> but the return code says timeout (rc > 128).

If the read command times out, it always returns > 128, so if you have a
return code in that range, you can assume read timed out and react
accordingly.

> Therefor it is not possible to know if a full line was returned or not.

When read times out, it always returns what it read before the timeout in
the buffer, so you don't lose any data. Whether or not that's a `full line'
is up to timing, and it's up to the script to decide how to cope with it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU c...@case.edu 
http://tiswww.cwru.edu/~chet/

AW: Re: read command sometimes misses newline on timeout

[Thomas Oettli via Bug reports for the GNU Bourne Again SHell]

I guess I will wait for an answer from Martin D Kealey. His answer tells me 
that he understood what issue I have.
But thank you a lot, nevertheless.
BTW: I have already tried all kinds of combinations including read -t 0 and 
could not find a working solution, that is why I filled a bug report.

Von: alex xmb sw ratchev 
Gesendet: Montag, 7. Oktober 2024 21:50
An: Thomas Oettli ; bash list 
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmb...@gmail.com>Report as Spam / Check 
Mail

plz write to bash list

about ur case , i dont much get it
async read ?

just small question .. why do u ' read again , to get full line '

and a small note
there is -t 0 , that returns when data'd be available
u maybe can test this approach

On Monday, October 7, 2024, Thomas Oettli 
mailto:thomas.oet...@sfs.com>> wrote:
Sorry, English is not my mother tongue, so let me try to clear this.

I have a script that asynchronously reads from a pipe by using read with 
timeout (basically the "reader" function in the example). The separator could 
be any character, I just assume that it is "\n" in this case. I do not control 
the writer to the pipe, lags may occur at any moment.

Now this happens:


  1.
the script calls "read" on pipe
  2.
"read" hits timeout just after it has read "\n" from the pipe
  3.
"read" returns the full line, but exits with rc > 128
  4.
the script assumes a partial line, appends received data to a buffer and calls 
"read" again, but without timeout to get the rest of the line
  5.
because "\n" was already read at step 2, "read" returns the next line
  6.
script appends received data to the buffer to complete the line (due to wrong 
rc from step 3)

The result is two lines in the buffer (without "\n" in between). Do you 
understand now what I mean?

Von: alex xmb sw ratchev mailto:fxmb...@gmail.com>>
Gesendet: Montag, 7. Oktober 2024 20:02
An: Thomas Oettli mailto:thomas.oet...@sfs.com>>
Cc: bug-bash@gnu.org 
mailto:bug-bash@gnu.org>>; 
chet.ra...@case.edu 
mailto:chet.ra...@case.edu>>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmb...@gmail.com>Report as Spam / Check 
Mail

to know if read returned anything , even if its exit code is not 0 , check 
using [[ $reply ]]

i really didnt get ur english or issue

On Monday, October 7, 2024, Thomas Oettli 
mailto:thomas.oet...@sfs.com>> wrote:

I know that it works in that case.
This is just an example that tries to force read into the same situatiation 
that I hit in a real world example in which I don't control the input.

Von: alex xmb sw ratchev mailto:fxmb...@gmail.com>>
Gesendet: Montag, 7. Oktober 2024 19:17
An: Thomas Oettli mailto:thomas.oet...@sfs.com>>
Cc: bug-bash@gnu.org 
mailto:bug-bash@gnu.org>>; 
chet.ra...@case.edu 
mailto:chet.ra...@case.edu>>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmb...@gmail.com>Report as Spam / Check 
Mail

well try remove the -n in echo

On Monday, October 7, 2024, Thomas Oettli 
mailto:thomas.oet...@sfs.com>> wrote:

Sorry Alex, I don't understand exactly what you mean.
Here is the Test-Script again with some comments:

function reader() {
  local buf line
  while :; do
read -t .01 buf  # try to read line to $buf with timeout
rc=$?
if (( rc == 0 )); then   # got a full line or the rest of a 
partial line, append $buf to $line
  line+=$buf
elif (( rc > 128 )); then# ran into timeout
  line+=$buf # maybe received partial line, append 
$buf to $line and continue reading
  continue
fi

# at this point, the content of $line should always be "TEST"

[[ $line != TEST ]] && echo Invalid li

Re: read command sometimes misses newline on timeout

[Chet Ramey]

On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again 
SHell wrote:



Bash Version: 5.2
Patch Level: 26
Release Status: release

Description:
I have tried to write a bash script that asynchronously reads from a pipe (line 
by line) with the help of "read -t".
 If the timeout occurs in just the right moment, read returns the full line, 
but the return code says timeout (rc > 128).


If the read command times out, it always returns > 128, so if you have a
return code in that range, you can assume read timed out and react
accordingly.

 Therefor it is not possible to know if a full line was returned or not. 


When read times out, it always returns what it read before the timeout in
the buffer, so you don't lose any data. Whether or not that's a `full line'
is up to timing, and it's up to the script to decide how to cope with it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Bash debugging.

[Chet Ramey]

On 10/5/24 4:14 PM, David Shuman wrote:

I have been working on several items as I am using bash to configure
systems.

I started wanting to log the output of my scripts.  Then I added a prefixed
message construct so detailed logs could be summarized without extraneous
debugging information (I have written an extract program using c in the
past that could also probably be written in bash (readlng a line at a time
from the log file}. The code is expected to be modified to establish
project oriented bash environment settings for the execution of the
configuration scripts.   I have a function to print the bash reverse
function trace and the accumulated arguments table (BASH_ARGV).  The
function is user callable .It can also be issued as the first statement of
a function call (a function trace option).  Most importantly the reverse
trace is in theory (currently being tested as the initial processing for
trap statements.

Does this sound potentially like a section of the bash handbook?


It sounds like it could be a standalone document describing this
particular approach. I'd be glad to look at it.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: ${param@a} should bypass unbound variable check or be processed ahead of it.

[konsolebox]

Changing "c == '@'" to "c == '@' && !want_attributes" in line 10145 of
subst.c seems enough to fix it.


-- 
konsolebox
diff --git a/subst.c b/subst.c
index 251eafaf..9ee816e0 100644
--- a/subst.c
+++ b/subst.c
@@ -10142,7 +10142,7 @@ parameter_brace_expand (char *string, size_t *indexp, int quoted, int pflags, in
 
   /* All the cases where an expansion can possibly generate an unbound
  variable error. */
-  if (want_substring || want_patsub || want_casemod || c == '@' || c == '#' || c == '%' || c == RBRACE)
+  if (want_substring || want_patsub || want_casemod || c == '@' && !want_attributes || c == '#' || c == '%' || c == RBRACE)
 {
   if (var_is_set == 0 && unbound_vars_is_error && ((name[0] != '@' && name[0] != '*') || name[1]) && all_element_arrayref == 0)
 	{

AW: Re: read command sometimes misses newline on timeout

[Thomas Oettli via Bug reports for the GNU Bourne Again SHell]

Sorry Alex, I don't understand exactly what you mean.
Here is the Test-Script again with some comments:

function reader() {
  local buf line
  while :; do
read -t .01 buf  # try to read line to $buf with timeout
rc=$?
if (( rc == 0 )); then   # got a full line or the rest of a 
partial line, append $buf to $line
  line+=$buf
elif (( rc > 128 )); then# ran into timeout
  line+=$buf # maybe received partial line, append 
$buf to $line and continue reading
  continue
fi

# at this point, the content of $line should always be "TEST"

[[ $line != TEST ]] && echo Invalid line: $line && exit
echo OK
line=""  # prepare to read next line, set $line 
to empty string
  done
}
reader < <(
  while :; do
echo -n TEST
sleep .00$(($RANDOM%10))
echo
  done
)

Could you please explain to me, where exactly the mistake is? I don't know if 
you ran the script by yourself, but on all of my machines it exits after just a 
few seconds
with the following output:

Invalid line: TESTTEST

Von: alex xmb sw ratchev 
Gesendet: Montag, 7. Oktober 2024 16:41
An: Thomas Oettli 
Cc: bug-bash@gnu.org ; chet.ra...@case.edu 

Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmb...@gmail.com>Report as Spam / Check 
Mail

there is a case , u [[ $readreply ]] after read

On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU Bourne 
Again SHell mailto:bug-bash@gnu.org>> wrote:
I agree with you, but it should never happen that read returns timeout, also 
returns the full line and has already read the newline character.
If that happens, there is no way for the script to decide what to do. Please 
see the provided test script, it showcases the error.

If I did a mistake there, I would gladly change it. But I currently don't see 
any way how to handle this properly from the script side.

Please also see the answer from Martin D Kealey, I think he is on to something:
https://lists.gnu.org/archive/html/bug-bash/2024-10/msg7.html




On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
SHell wrote:

> Bash Version: 5.2
> Patch Level: 26
> Release Status: release
>
> Description:
> I have tried to write a bash script that asynchronously reads from a pipe 
> (line by line) with the help of "read -t".
> If the timeout occurs in just the right moment, read returns the full line, 
> but the return code says timeout (rc > 128).

If the read command times out, it always returns > 128, so if you have a
return code in that range, you can assume read timed out and react
accordingly.

> Therefor it is not possible to know if a full line was returned or not.

When read times out, it always returns what it read before the timeout in
the buffer, so you don't lose any data. Whether or not that's a `full line'
is up to timing, and it's up to the script to decide how to cope with it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU c...@case.edu 
http://tiswww.cwru.edu/~chet/

Re: read command sometimes misses newline on timeout

[alex xmb sw ratchev]

to know if read returned anything , even if its exit code is not 0 , check
using [[ $reply ]]

i really didnt get ur english or issue

On Monday, October 7, 2024, Thomas Oettli  wrote:

>
> I know that it works in that case.
> This is just an example that tries to force read into the same
> situatiation that I hit in a real world example in which I don't control
> the input.
> --
> *Von:* alex xmb sw ratchev 
> *Gesendet:* Montag, 7. Oktober 2024 19:17
> *An:* Thomas Oettli 
> *Cc:* bug-bash@gnu.org ; chet.ra...@case.edu <
> chet.ra...@case.edu>
> *Betreff:* [EXT] Re: read command sometimes misses newline on timeout
>
> CAUTION: This email originated from outside the SFS organization. Do not
> follow guidance, click links or open attachments unless you recognize the
> sender and know the content is safe.
> MAIL FROM:  <@fxmb...@gmail.com> Report as Spam / Check Mail
> 
>
> well try remove the -n in echo
>
> On Monday, October 7, 2024, Thomas Oettli  wrote:
>
> Sorry Alex, I don't understand exactly what you mean.
> Here is the Test-Script again with some comments:
>
> function reader() {
>   local buf line
>   while :; do
> read -t .01 buf  # try to read line to $buf with 
> timeout
> rc=$?
> if (( rc == 0 )); then   # got a full line or the rest of a 
> partial line, append $buf to $line
>   line+=$buf
> elif (( rc > 128 )); then# ran into timeout
>   line+=$buf # maybe received partial line, 
> append $buf to $line and continue reading
>   continue
> fi
>
> # at this point, the content of $line should always be "TEST"
>
> [[ $line != TEST ]] && echo Invalid line: $line && exit
> echo OK
> line=""  # prepare to read next line, set 
> $line to empty string
>   done
> }
> reader < <(
>   while :; do
> echo -n TEST
> sleep .00$(($RANDOM%10))
> echo
>   done
> )
>
>
> Could you please explain to me, where exactly the mistake is? I don't know
> if you ran the script by yourself, but on all of my machines it exits after
> just a few seconds
> with the following output:
>
> Invalid line: TESTTEST
> --
> *Von:* alex xmb sw ratchev 
> *Gesendet:* Montag, 7. Oktober 2024 16:41
> *An:* Thomas Oettli 
> *Cc:* bug-bash@gnu.org ; chet.ra...@case.edu <
> chet.ra...@case.edu>
> *Betreff:* [EXT] Re: read command sometimes misses newline on timeout
>
> CAUTION: This email originated from outside the SFS organization. Do not
> follow guidance, click links or open attachments unless you recognize the
> sender and know the content is safe.
> MAIL FROM:  <@fxmb...@gmail.com> Report as Spam / Check Mail
> 
>
> there is a case , u [[ $readreply ]] after read
>
> On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU
> Bourne Again SHell  wrote:
>
> I agree with you, but it should never happen that read returns timeout,
> also returns the full line and has already read the newline character.
> If that happens, there is no way for the script to decide what to do.
> Please see the provided test script, it showcases the error.
>
> If I did a mistake there, I would gladly change it. But I currently don't
> see any way how to handle this properly from the script side.
>
> Please also see the answer from Martin D Kealey, I think he is on to
> something:
> https://lists.gnu.org/archive/html/bug-bash/2024-10/msg7.html
>
>
>
>
> On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
> SHell wrote:
>
> > Bash Version: 5.2
> > Patch Level: 26
> > Release Status: release
> >
> > Description:
> > I have tried to write a bash script that asynchronously reads from a
> pipe (line by line) with the help of "read -t".
> > If the timeout occurs in just the right moment, read returns the full
> line, but the return code says timeout (rc > 128).
>
> If the read command times out, it always returns > 128, so if you have a
> return code in that range, you can assume read timed out and react
> accordingly.
>
> > Therefor it is not possible to know if a full line was returned or not.
>
> When read times out, it always returns what it read before the timeout in
> the buffer, so you don't lose any data. Whether or not that's a `full line'
> is up to timing, and it's up to the script to decide how to cope with it.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/
>
>

Re: Fwd: read command sometimes misses newline on timeout

[Greg Wooledge]

> -- Forwarded message --
> From: *Thomas Oettli* 

>1. the script calls "read" on pipe
>2. "read" hits timeout just after it has read "\n" from the pipe
>3. "read" returns the full line, but exits with rc > 128
>4. the script assumes a partial line, appends received data to a buffer
>and calls "read" again, but without timeout to get the rest of the line
>5. because "\n" was already read at step 2, "read" returns the next line
>6. script appends received data to the buffer to complete the line (due
>to wrong rc from step 3)
> 
> The result is two lines in the buffer (without "\n" in between). Do you
> understand now what I mean?

If bash is returning nonzero status after reading a full line, then
that may be a bug that should be fixed.  Martin may be looking into
that.

Meanwhile, your step 4 looks wrong to me.  If you're using an asynchronous
reader which expects and works around input delays, then suddenly deciding
to call read *without* the -t option seems strange.

If you believe you've got a partial line, then I would expect the next
call to read would be just like the first call.  You might get another
part of the line, without seeing the end of the line.  It might take
several calls before you get a full line.

Of course, fixing step 4 to use multiple calls with timeouts instead of
one blocking call won't help you if bash is in fact returning false
failure codes.

(Note for help-bash readers: this thread began on bug-bash, but a private
message was sent to one responder, who then forwarded it to help-bash,
perhaps by mistake.  I'm Cc'ing bug-bash on this, and I'd advise responders
to keep this on bug-bash if possible.)

Re: read command sometimes misses newline on timeout

[Martin D Kealey]

OK, running a similar test with instrumentation gets:

$ time (
  trap ' echo BANG ' SIGALRM ;
  while :; do printf TEST; sleep .00$((1+RANDOM%2)); echo; done |
  for ((r=10 ;r>0; --r)) do line= ; read -t .002 line; rc=$?; [[ $line
= TEST ]] ; echo "STATUS $rc $? $line" ; done
 ) |&
  sort | uniq -c

  1 +# Hit read timeout SIGALRM, run from line 699 in ./read.def #+
STATUS 142 1 TES
  1 +# Hit read timeout SIGALRM, run from line 72 in zread.c #+
STATUS 142 1
  2 +# Hit read timeout SIGALRM, run from line 865 in ./read.def #+
STATUS 142 1 T
  1 +# Hit read timeout SIGALRM, run from line 865 in ./read.def #+
STATUS 142 1 TES
 16 +# Hit read timeout SIGALRM, run from line 913 in ./read.def #+
STATUS 142 0 TEST
   1721 +# Hit read timeout SIGALRM, run from line 913 in ./read.def #+
STATUS 142 1
  3 STATUS 0 0 TEST
  32649 STATUS 0 1
  34367 STATUS 142 0 TEST
  31235 STATUS 142 1
  2 STATUS 142 1 EST
  2 STATUS 142 1 T

real3m21.411s
user1m14.054s
sys 1m46.493s

That's to say, that out of 10 reads:

   - 0.003% had both `read` returning 0 and filling var with TEST.
   - 34.4% had `read returning non-zero but filling var with TEST.
   - 32.6% had `read` returning 0 but leaving var empty (which is
   presumably the delayed bare newline after previously filling the var with
   TEST but returning 142).
   - 1.7% were handled by `check_read_timeout()`, and all had read
   returning a status of 142 (and leaving var empty).

I note that 1721+32649-34367=3 so I infer that the last 3 cases are
interconnected:

Of that 1.7% almost all were handled by this section of read.def:
  /* I don't think this clause ever tests true. */
  if (skip_ctlnul && saw_escape && i == 1 && input_string[0] == CTLNUL)
saw_escape = 0; /* Avoid dequoting bare CTLNUL */

  input_string[i] = '\0';
*  check_read_timeout ();*

#if defined (READLINE)
  if (edit)
free (rlbuf);
#endif

-Martin
PS: I've since run other tests that confirm the association I noted above,
but I didn't keep the output. I'll run them again if anyone wants to see a
more precise demonstration.

Re: Fwd: read command sometimes misses newline on timeout

[Martin D Kealey]

On Tue, 8 Oct 2024 at 06:26, Greg Wooledge  wrote:

> From: *Thomas Oettli* 
> > The result is two lines in the buffer (without "\n" in between). Do you
> understand now what I mean?
>
> If bash is returning nonzero status after reading a full line, then that
> may be a bug that should be fixed.
> Martin may be looking into that.
>

Indeed I am.

At this point I'm reminded of Zork's Introduction: "*You are in a maze of
twisty little passages, all alike. You are likely to be eaten by a Gru.*"
Trying to track longjmp() through four translation units is like trying to
hunt a Gru that's always trying to sneak up behind me. (I'm beginning to
wonder if a Gru is a Gnu with the lights turned off.  If you don't hear
from me in a year, you'll know what to write on my epitaph.)

Just to be clear, I replicated this with strictly positive delays in the
millisecond range, so that it wouldn't ever use `read -t0`. Nor have I used
non-timed reads. The pattern still emerges, where in a small proportion of
cases SIGALRM gets delivered while *not* inside the `read` syscall, when
the `read` has consumed just the newline.

I've inserted code that, when "check_signals()" is called, will print out
where it was called from. The slightest shift in how I write the test as a
script dramatically changes the ratio of which lines are most active.

*Some* of my earlier tests output "TESTTESTTESTTESTTEST" (five-fold or
longer), indicating multiple successive lost newlines, but I didn't record
those sessions so I can't show you right now.

I'm about to add more code to tell me what the code was doing when the
signal was raised, as I suspect that will be more instructive than what's
happening when it's detected. (Yay, introspection in C. AAARGHH!*)

-Martin