Re: [PATCH] Fix blocking read timeouts at a small probability

[felix]

On Mon, Feb 15, 2021 at 09:41:29AM -0500, Chet Ramey wrote:
> Just for testing, not for any real use case.
> 
> > He also provided me with his new test case
> > to see the interaction with SIGCHLD. The failure of this test case
> > seems to have also involved SIGCHLD in the previous devel Bash. Now in
> > the current devel, it is improved but still sometimes blocks.

Originally, this was a (my) bug in a multi-tasked script**, where timeout was
(wrongly) computed from previous step. When my script locked for unknown
reason, I wrote test script in order to reveal this.

> If you need a timeout on the order of a single microsecond in a shell
> script, I urge you to reconsider your choices.
I agree, for this kind of tricks, Python (or even Perl) is more apropriated,
but I like to see how a poor system (with only few binary libraries) could
do unattended things...

...And it's fun to see how far we can go with bash, today.

  ** https://f-hauri.ch/vrac/multiping.sh.txt
-- 
 Félix Hauri  --  http://www.f-hauri.ch

Re: syntax error while parsing a case command within `$(...)'

[Andreas Schwab]

On Feb 16 2021, Koichi Murase wrote:

> Because the ending keywords---}, fi, done, and esac---are allowed to
> be chained, which I think is undocumented though.

See 2.4 Reserved Words.

This recognition shall only occur when none of the characters is
quoted and when the word is used as:

* The first word of a command
* The first word following one of the reserved words other than
  case, for, or in
* The third word in a case command (only in is valid in this case)
* The third word in a for command (only in and do are valid in this
  case)

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月16日(火) 16:28 Andreas Schwab :
> See 2.4 Reserved Words.
>
> This recognition shall only occur when none of the characters is
> quoted and when the word is used as:

I think that section just describes the necessary condition that the
word is recognized as the reserved keyword: "This recognition shall
*only* occur...". It doesn't explain whether the reserved keyword can
be really used to construct the AST following the shell grammar. For
example, this alone doesn't explain why

$ if :; then echo A; fi if :; then echo A; fi

(i.e., the combination "fi if") is a syntax error.

Hmm, it seems we need to combine the Yacc-BNF notation at the end of
Sec. 2.10 to understand this? For example,

$ case x in (x) if :; then echo; fi esac

can be derived as

case_clause
-> Case WORD linebreak in linebreak case_list Esac
-> Case 'x' ε in ε case_list Esac
-> Case 'x' ε in ε case_item_ns Esac
-> Case 'x' ε in ε '(' pattern ')' compound_list Esac
-> Case 'x' ε in ε '(' 'x' ')' linebreak term Esac
-> Case 'x' ε in ε '(' 'x' ')' ε and_or Esac
-> Case 'x' ε in ε '(' 'x' ')' ε pipeline Esac
-> Case 'x' ε in ε '(' 'x' ')' ε pipe_sequence Esac
-> Case 'x' ε in ε '(' 'x' ')' ε command Esac
-> Case 'x' ε in ε '(' 'x' ')' ε compound_command Esac
-> Case 'x' ε in ε '(' 'x' ')' ε if_clause Esac
-> ...

where ε is an empty string. OK, now I understood this behavior is
actually required by the POSIX standard. Can we find any textual
explanation on this rule? Or maybe this behavior is intuitive enough
for those who understand the shell grammar so that they don't see the
necessity of an additional explanation...

Thank you for the comment!

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Oğuz]

16 Şubat 2021 Salı tarihinde Koichi Murase  yazdı:

> OK, now I understood this behavior is
> actually required by the POSIX standard. Can we find any textual
> explanation on this rule? Or maybe this behavior is intuitive enough
> for those who understand the shell grammar so that they don't see the
> necessity of an additional explanation...
>

XRAT Shell Grammar (
https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_10
) explains that giving almost the same examples as we posted in this thread.


-- 
Oğuz

Re: syntax error while parsing a case command within `$(...)'

[Chris Elvidge]

On 16/02/2021 10:19 am, Koichi Murase wrote:

2021年2月16日(火) 16:28 Andreas Schwab :

See 2.4 Reserved Words.

 This recognition shall only occur when none of the characters is
 quoted and when the word is used as:


I think that section just describes the necessary condition that the
word is recognized as the reserved keyword: "This recognition shall
*only* occur...". It doesn't explain whether the reserved keyword can
be really used to construct the AST following the shell grammar. For
example, this alone doesn't explain why

$ if :; then echo A; fi if :; then echo A; fi

(i.e., the combination "fi if") is a syntax error.


It seems to me that you're missing a semicolon between 'fi' and 'if' to 
delineate the two constructs.




Hmm, it seems we need to combine the Yacc-BNF notation at the end of
Sec. 2.10 to understand this? For example,

$ case x in (x) if :; then echo; fi esac

can be derived as

case_clause
-> Case WORD linebreak in linebreak case_list Esac
-> Case 'x' ε in ε case_list Esac
-> Case 'x' ε in ε case_item_ns Esac
-> Case 'x' ε in ε '(' pattern ')' compound_list Esac
-> Case 'x' ε in ε '(' 'x' ')' linebreak term Esac
-> Case 'x' ε in ε '(' 'x' ')' ε and_or Esac
-> Case 'x' ε in ε '(' 'x' ')' ε pipeline Esac
-> Case 'x' ε in ε '(' 'x' ')' ε pipe_sequence Esac
-> Case 'x' ε in ε '(' 'x' ')' ε command Esac
-> Case 'x' ε in ε '(' 'x' ')' ε compound_command Esac
-> Case 'x' ε in ε '(' 'x' ')' ε if_clause Esac
-> ...

where ε is an empty string. OK, now I understood this behavior is
actually required by the POSIX standard. Can we find any textual
explanation on this rule? Or maybe this behavior is intuitive enough
for those who understand the shell grammar so that they don't see the
necessity of an additional explanation...

Thank you for the comment!

--
Koichi





--
Chris Elvidge
England

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月16日(火) 19:29 Chris Elvidge :
> > I think that section just describes the necessary condition that the
> > word is recognized as the reserved keyword: "This recognition shall
> > *only* occur...". It doesn't explain whether the reserved keyword can
> > be really used to construct the AST following the shell grammar. For
> > example, this alone doesn't explain why
> >
> > $ if :; then echo A; fi if :; then echo A; fi
> >
> > (i.e., the combination "fi if") is a syntax error.
>
> It seems to me that you're missing a semicolon between 'fi' and 'if' to
> delineate the two constructs.

Yes, that's an example of a syntax error that I made by intentionally
removing the semicolon. If we just look at POSIX XCU 2.4 which Andreas
pointed to justify "fi fi", a similar combination "fi if" without
semicolon in between appears to be also justified, but that's false.
Thus there should be also other essential rules. That's what I wanted
to say.

By the way, in the case of nested compounds such as `case x in (x) if
:; then echo; fi esac' or `{ { echo; } }', the fact was actually
somewhat opposite. According to the formal grammar by BNF (without
caring XCU 2.4), actually, semicolonless commands like

$ case x in (x) if : then echo fi esac

or

$ { { echo } }

are valid. However, it is ambiguous and confusing, so this kind of
construct is afterward disallowed by the grammatical "Constraints"
specified in XCU 2.4.

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月16日(火) 19:24 Oğuz :
> XRAT Shell Grammar (  
> https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_10
>  ) explains that giving almost the same examples as we posted in this thread.

Thank you, interesting! So, if I understand correctly, the following
construct which Bash allows is finally disallowed by POSIX?

$ if (false) then (echo x) else (echo y) fi

To summarize, the following cases are required by the POSIX standard
to be correctly parsed?

$ bash -c ': $(case x in esac)'
$ bash -c ': $(case x in (x) if :; then echo a; fi esac)'
$ bash -c ': $(case x in (x) a() { echo a; } esac)'
$ bash -c ': $(case x in (x) { echo a; } esac)'
$ bash -c ': $(case x in (x) while false; do echo a; done esac)'
$ bash -c ': $(case x in (x) case y in (y) echo a;; esac esac)'

and this case

$ bash -c ': $(case x in (x) (echo a) esac)'

could be a Bash extension. But actually, it totally depends on what is
"the matching closing parenthesis" in the description of the command
substitution by the POSIX:

XCU 2.6.3 Command Substitution
> With the $(command) form, all characters following the open
> parenthesis to the matching closing parenthesis constitute the
> command.

I guess the rationale of the current Bash behavior is that "the
matching closing parenthesis" in the context of the command
substitution is actually not strictly defined in the standard.

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Oğuz]

16 Şubat 2021 Salı tarihinde Koichi Murase  yazdı:

> 2021年2月16日(火) 19:24 Oğuz :
> > XRAT Shell Grammar (  https://pubs.opengroup.org/
> onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_10 ) explains
> that giving almost the same examples as we posted in this thread.
>
> Thank you, interesting! So, if I understand correctly, the following
> construct which Bash allows is finally disallowed by POSIX?
>
> $ if (false) then (echo x) else (echo y) fi


No, it is allowed.


>
> To summarize, the following cases are required by the POSIX standard
> to be correctly parsed?
>
> $ bash -c ': $(case x in esac)'
> $ bash -c ': $(case x in (x) if :; then echo a; fi esac)'
> $ bash -c ': $(case x in (x) a() { echo a; } esac)'
> $ bash -c ': $(case x in (x) { echo a; } esac)'
> $ bash -c ': $(case x in (x) while false; do echo a; done esac)'
> $ bash -c ': $(case x in (x) case y in (y) echo a;; esac esac)'


Yes.


>
> and this case
>
> $ bash -c ': $(case x in (x) (echo a) esac)'
>
> could be a Bash extension.
>

No.


-- 
Oğuz

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月16日(火) 21:41 Oğuz :
>> > XRAT Shell Grammar (
>> > https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_10
>> > ) explains that giving almost the same examples as we posted in
>> > this thread.
>>
>> Thank you, interesting! So, if I understand correctly, the following
>> construct which Bash allows is finally disallowed by POSIX?
>>
>> $ if (false) then (echo x) else (echo y) fi
>
> No, it is allowed.

OK, I have again checked the page. I haven't read the latter half of
the section.  The special rule is

XCU 2.10.2/(1) Shell Grammar Rules [Command Name]
> When the TOKEN is exactly a reserved word, the token identifier for
> that reserved word shall result. Otherwise, the token WORD shall be
> returned. Also, if the parser is in any state where only a reserved
> word could be the next correct token, proceed as above.

It seems like XCU 2.4 is telling lie..., or at least some important
information is missing.  Is XCU 2.4 just informative texts?  Instead,
is XCU 2.10 solely the actual normative requirements for the shell
grammar?

The above special rule means that

  $ if [[ str ]] then [[ str ]] fi
  $ if ((expr)) then ((expr)) fi

should also be acceped as valid constructs.  Other shells parses them,
but Bash fails:

  $ bash -c 'if [[ str ]] then [[ str ]] fi'; echo $?
  bash: -c: line 0: syntax error near unexpected token `then'
  bash: -c: line 0: `if [[ str ]] then [[ str ]] fi'
  1
  $ zsh -c 'if [[ str ]] then [[ str ]] fi'; echo $?
  0
  $ ksh -c 'if [[ str ]] then [[ str ]] fi'; echo $?
  0
  $ yash -c 'if [[ str ]] then [[ str ]] fi'; echo $?
  0

  $ bash -c 'if ((expr)) then ((expr)) fi'; echo $?
  bash: -c: line 0: syntax error near unexpected token `then'
  bash: -c: line 0: `if ((expr)) then ((expr)) fi'
  1
  $ zsh -c 'if ((expr)) then ((expr)) fi'; echo $?
  0
  $ ksh -c 'if ((expr)) then ((expr)) fi'; echo $?
  0

  Note: yash doesn't support arithmetic commands.

Maybe Bash could insist that [[ ... ]] and ((...)) are not part of the
standard so that it can behave inconsistently.  But it seems to me
that there is really no reason to behave inconsistently here.

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Chet Ramey]

On 2/16/21 12:29 AM, Oğuz wrote:
16 Şubat 2021 Salı tarihinde Koichi Murase > yazdı:


bash -c ': $(case x in esac)'
bash -c ': $(case x in (x) if :; then echo a; fi esac)'
bash -c ': $(case x in (x) a() { echo a; } esac)'
bash -c ': $(case x in (x) for ((i=0;i<10;i++)) { echo a; } esac)'
bash -c ': $(case x in (x) while false; do echo a; done esac)'
bash -c ': $(case x in (x) case y in (y) echo a;; esac esac)'


I'm lost. Are these even supposed to work outside command substitution? Why?


The same principle that you articulated earlier: the last command-list
doesn't need to be delimited by a DSEMI as long as the esac is recognized
as a separate token, and then it goes on from there to the conditions
under which reserved words are recognized.

This is just academic. No sane person would write these without some kind
of delimiter.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

> XCU 2.10.2/(1) Shell Grammar Rules [Command Name]
> > When the TOKEN is exactly a reserved word, the token identifier for
> > that reserved word shall result. Otherwise, the token WORD shall be
> > returned. Also, if the parser is in any state where only a reserved
> > word could be the next correct token, proceed as above.
>
> [...]
>
> The above special rule means that
>
>   $ if [[ str ]] then [[ str ]] fi
>   $ if ((expr)) then ((expr)) fi
>
> should also be acceped as valid constructs.  Other shells parses them,
> but Bash fails:
>
> [...]
>
> Maybe Bash could insist that [[ ... ]] and ((...)) are not part of the
> standard so that it can behave inconsistently.  But it seems to me
> that there is really no reason to behave inconsistently here.

This is actually an easy fix as attached. I guess this change can be
safely applied because originally these constructs were a syntax
error, so no shell scripts rely on the current behavior.


0001-accept-reserved-words-after-arithmetic-condtional-co.patch
Description: Binary data

Re: syntax error while parsing a case command within `$(...)'

[Oğuz]

On Tue, Feb 16, 2021 at 5:34 PM Koichi Murase 
wrote:

> Maybe Bash could insist that [[ ... ]] and ((...)) are not part of the
> standard so that it can behave inconsistently.  But it seems to me
> that there is really no reason to behave inconsistently here.
>

They resemble `[ ... ]' (I know it's a simple command, but still), maybe
that's why. I think it'd seem more inconsistent to ordinary user if

  if [[ x ]] then [[ x ]] fi

worked but

  if [ x ] then [ x ] fi

didn't.

Re: syntax error while parsing a case command within `$(...)'

[Dale R. Worley]

Oğuz  writes:
>> Before we worry about what to change, I want to note that the original
>> example is syntactically incorrect.  The example is
>>
>>   $ bash -c ': $(case x in x) esac)'
>>
>> But the manual page makes it clear that each case must be ended with
>> ";;".
>
> `;;' is optional for the last case item.

The manual page (for my version) says it's required.  If, in some
certain circumstances, it works without, that's nice.  But there's no
commitment that it will work now, or in future releases.

> `case x in esac' (without the linebreak) works fine outside the command
> substitution.

The manual page (for my version) says that "esac" will be recognized in
positions where a simple command may appear.  If, in some other
circumstances, it works, that's nice.  But there's no commitment that it
will work now, or in future releases.

Now, if you want to advocate that it *should* always work, go ahead.
But that's a feature request, not a bug report.

Dale

Re: syntax error while parsing a case command within `$(...)'

[Lawrence Velázquez]

> On Feb 16, 2021, at 10:42 PM, Dale R. Worley  wrote:
> 
>> Oğuz  writes:
>> 
>> `;;' is optional for the last case item.
> 
> The manual page (for my version) says it's required. If, in some
> certain circumstances, it works without, that's nice.


It's also required by POSIX.

> But there's no commitment that it will work now, or in future
> releases.

The commitment is bash's claim to POSIX compliance. Outside of
command substitutions, the referenced construct currently works,
as POSIX requires. I expect this is by design. If it ceases to be
recognized in the future, then bash will have intentionally become
less compliant with POSIX, for no good reason.

>> `case x in esac' (without the linebreak) works fine outside the
>> command substitution.
> 
> The manual page (for my version) says that "esac" will be recognized
> in positions where a simple command may appear.

The 5.1 man page doesn't say that, or I can't find it, but it doesn't
matter because a simple command can't go immediately after that "in".

> If, in some other circumstances, it works, that's nice.


Recognition of "case x in esac" is also required by POSIX.

> But there's no commitment that it will work now, or in future
> releases.

See above.

> Now, if you want to advocate that it *should* always work, go ahead.
> But that's a feature request, not a bug report.

TO: Chet
SUBJECT: Feature request

Please do not make bash less POSIX-compliant for no good reason!

thx
vq