Re: syntax error while parsing a case command within `$(...)'

[Robert Elz]

Date:Tue, 16 Feb 2021 18:19:48 +0800
From:Koichi Murase 
Message-ID:  


  | For example, this alone doesn't explain why
  |
  | $ if :; then echo A; fi if :; then echo A; fi
  |
  | (i.e., the combination "fi if") is a syntax error.

That one is quite a different issue, if you look at the grammar
you'll find there's no way to have 2 consecutive commands (no matter
what kind of commands they are) without some kind of separator
between them.   The second "if" there is a total red herring, it makes
no difference what word is there (except one of the legal operators in
that location) - it is going to be a syntax error.

The other examples given in earlier messages have one command
embedded in another, there's no requirement for separators in those
(and they are separators, not terminators).

Your "case x in (x) if :; then echo; fi esac" is the same.

kre

ps: many of these strange rules all originate in the original Bourne
shell's recursive descent parser, which was crammed into a very limited
space, and so didn't always enforce sanity - it made sure that what
needed to work worked, but cared much less about prohibiting things
that are useless, and could easily have been rejected (like case statements
with no patterns) had it been considered worthwhile to spend the code
space needed to make that happen.Since then everyone copies so as
to maintain backward compatibility ("some idiot might have written code
like that, we cannot reject it"), and consequently, POSIX, which documents
what actually exists, specifies it as well.

Re: syntax error while parsing a case command within `$(...)'

[Oğuz]

16 Şubat 2021 Salı tarihinde Chet Ramey  yazdı:
>
> This is just academic. No sane person would write these without some kind
> of delimiter.
>

Yeah, `parse_comsub' really needs some work anyway.

  $ ./bash -c '$(case x in x|esac) foo;; esac)'
  ./bash: -c: line 1: syntax error near unexpected token `;;'
  ./bash: -c: line 1: `$(case x in x|esac) foo;; esac)'


-- 
Oğuz

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月17日(水) 21:56 Robert Elz :
>   | For example, this alone doesn't explain why
>   |
>   | $ if :; then echo A; fi if :; then echo A; fi
>   |
>   | (i.e., the combination "fi if") is a syntax error.
>
> That one is quite a different issue,

Yes, that's what I wanted to claim by the above example, that is, XCU
2.4 isn't the (most) essential point for the reason that `fi esac' or
`fi fi` is a valid construct. The above example is just a
counter-example against explaining `fi esac` by solely XCU 2.4. I used
`fi if' as an example because I did know that `fi if' isn't allowed by
the grammar, so I have referred to the grammar in the next paragraph
in my previous reply. Instead, as you have explained in your next
paragraph, the essential reason why `fi esac' can appear in a valid
command is that the grammatical structure allows the combination of
`fi esac' as a result of some embedded compound command in another.

> ps: many of these strange rules all originate in the original Bourne
> shell's recursive descent parser,

I was also thinking these apparent rules are strange before carefully
reading the BNF (in yacc style) specified in XCU 2.10, but now I
somehow feel these rules are understandable (but still not so natural
of course). I admit that it is confusing to most users, but at least,
I wouldn't say it's ``insane'' now.

> [...] and consequently, POSIX, which documents
> what actually exists, specifies it as well.

Thank you for the explanation of the historical backgrounds. That's
actually exactly what I guessed when I looked at the BNF.

Anyway, even if the POSIX specifies some inconsistent grammar, I feel
that cannot be the reason that Bash behaves in a way even more
inconsistent than what POSIX specifies.

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月17日(水) 1:52 Oğuz :
> They resemble `[ ... ]' (I know it's a simple command, but still), maybe 
> that's why. I think it'd seem more inconsistent to ordinary user if
>
>   if [[ x ]] then [[ x ]] fi
>
> worked but
>
>   if [ x ] then [ x ] fi
>
> didn't.

``Ordinary users'' may consider it being inconsistent, but what would
you think of this?

Actually, as you have written, [ ... ] is a simple command, so there
are already many differences that ``ordinary users'' would think
inconsistent. For example, the word splitting and pathname expansion
rules are different. For another example, `tempenv=1 [[ -v tempenv ]]'
is disallowed while `tempenv=1 [ -v tempenv ]' is allowed. and more...
Letting [[ ... ]] behave differently from either of [ ... ] and ( ...
) means that we create the third category of command which
grammatically behaves like neither the simple command nor the compound
command. I actually feel it is more consistent to allow `if [[ ... ]]
then' as far as we accept `if ( ... ) then'. That would be the only
neat thing under the POSIX constraints.

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Andreas Schwab]

On Feb 17 2021, Koichi Murase wrote:

> Yes, that's what I wanted to claim by the above example, that is, XCU
> 2.4 isn't the (most) essential point for the reason that `fi esac' or
> `fi fi` is a valid construct. The above example is just a
> counter-example against explaining `fi esac` by solely XCU 2.4.

XCU 2.4 only explains when words are to be recognized as reserved words
(the lexer part of parsing).  Only because of these rules the `fi esac'
example contains two consecutive reserved words.  The grammar depends on
having these rules in place, and then applies more rules on the
placement of tokens.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: syntax error while parsing a case command within `$(...)'

[Oğuz]

17 Şubat 2021 Çarşamba tarihinde Koichi Murase 
yazdı:

> 2021年2月17日(水) 1:52 Oğuz :
> > They resemble `[ ... ]' (I know it's a simple command, but still), maybe
> that's why. I think it'd seem more inconsistent to ordinary user if
> >
> >   if [[ x ]] then [[ x ]] fi
> >
> > worked but
> >
> >   if [ x ] then [ x ] fi
> >
> > didn't.
>
> ``Ordinary users'' may consider it being inconsistent, but what would
> you think of this?


I wouldn't mind. I always put a semicolon before `then' anyway.


>
> Actually, as you have written, [ ... ] is a simple command, so there
> are already many differences that ``ordinary users'' would think
> inconsistent. For example, the word splitting and pathname expansion
> rules are different. For another example, `tempenv=1 [[ -v tempenv ]]'
> is disallowed while `tempenv=1 [ -v tempenv ]' is allowed. and more...
> Letting [[ ... ]] behave differently from either of [ ... ] and ( ...
> ) means that we create the third category of command which
> grammatically behaves like neither the simple command nor the compound
> command.


Yes, this makes sense. You obviously put more thought into this than I did,
my earlier post was just the first thing I could think of that is against
what you suggested.


-- 
Oğuz

Re: syntax error while parsing a case command within `$(...)'

[Robert Elz]

Date:Wed, 17 Feb 2021 23:39:07 +0800
From:Koichi Murase 
Message-ID:  


Andreas answered the technical part of your message, and I agree
with that.

  | Anyway, even if the POSIX specifies some inconsistent grammar, I feel
  | that cannot be the reason that Bash behaves in a way even more
  | inconsistent than what POSIX specifies.

For a long time it was believed (incorrectly) that () counting & matching
was sufficient to find the end of a $() command substitution - just as the
next unescaped ` finds the end of a `` cmdsub.   Parsing these things is
a bit tricky, as they occur in the middle of a word, which is nominally a 
lexical token (an indivisible unit to the grammar) - and so needs either
fairly horrid tricks, or a parser generator which allows suspending its
current parse and running a new one, so the end of the cmdsub can be
correctly found (for the $() case).   Because that's so hard, and because
() counting was assumed to work, some shells (I believe including bash
initially, though I have never examined its code) used that technique.

That doesn't work however, some kind of parser is required, not just character
scanning for parentheses (and quoting).And it all has to be recursive,
as $() cmdsubs can occur in the middle of words in the middle of other cndsubs.

Chet can explain more if he feels it is warranted, but I believe that even
today, bash uses a very heuristic type of almost pretend to be a parser
to deal with these things - not really parsing them fully (not until they're
to be executed, where the issues all go away, as at that point, nothing
else is being parsed) but attempting to handle most cases so things work
for real code.

This is why commands that parse file and work properly outside cmdsubs
don't always parse correctly inside (while parsing the command that
contains them) - they're being done by two completely different parsers,
one that is correct, and works, and the other than is just mostly OK.

None of the (current anyway) problems represent code that anyone trying
to solve a real problem (as distinct from stressing the parser) would
ever write, so one can understand a certain reluctance to spending a lot
of time fixing things that only even happen in stress tests.

Really, who ever writes a real case statement with no patterns in it?

kre

Re: syntax error while parsing a case command within `$(...)'

[Chet Ramey]

On 2/17/21 9:53 AM, Oğuz wrote:
16 Şubat 2021 Salı tarihinde Chet Ramey > yazdı:


This is just academic. No sane person would write these without some kind
of delimiter.


Yeah, `parse_comsub' really needs some work anyway.


Don't worry.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: syntax error while parsing a case command within `$(...)'

[Chet Ramey]

On 2/17/21 11:56 AM, Robert Elz wrote:


Chet can explain more if he feels it is warranted, but I believe that even
today, bash uses a very heuristic type of almost pretend to be a parser
to deal with these things - not really parsing them fully (not until they're
to be executed, where the issues all go away, as at that point, nothing
else is being parsed) but attempting to handle most cases so things work
for real code.


I'd add that the reason I did this back in 2008 was that it was just
about impossible to have a re-entrant bison parser (one that calls
yyparse() recursively) and completely impossible to do portably with
yacc. The situation with bison may have improved.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

 2021年2月18日(木) 0:56 Robert Elz :
> For a long time [...]
>
> [...] And it all has to be recursive,
> as $() cmdsubs can occur in the middle of words in the middle of other 
> cndsubs.

Thank you for the historical story.

> Chet can explain more if he feels it is warranted, but I believe that even
> today, bash uses a very heuristic type of almost pretend to be a parser
> to deal with these things

Yes, that is done in `parse_comsub' in `parse.y' which I mentioned in
some earlier reply in this tree.

> None of the (current anyway) problems represent code that anyone trying
> to solve a real problem (as distinct from stressing the parser) would
> ever write, so one can understand a certain reluctance to spending a lot
> of time fixing things that only even happen in stress tests.

I actually doubt that it takes so much time to fix them... Have you
ever written a shell parser? Some time ago I have written a simple
shell parser mimicking the Bash behavior without thinking the details,
but it correctly parses all the command substitutions reported here.

Re: syntax error while parsing a case command within `$(...)'

[Robert Elz]

Date:Thu, 18 Feb 2021 01:34:59 +0800
From:Koichi Murase 
Message-ID:  


  | ever written a shell parser?

From scratch, no, but I maintain one.

With all the details

It isn't all that hard, but isn't trivial
either, and amounts to quite a bit of code
(and parsing all the bash extensions the
way it does would make it harder and bigger)

Like everything, it becomes a cost/benefit
tradeoff, are the costs worth the benefits?

That one is not for me (or you) to answer.

kre

ps:  I am sure Chet will confirm that I am not
shy about pointing out bash bugs/defects that
shoukd be fixed when I come accross one.

Re: syntax error while parsing a case command within `$(...)'

[Koichi Murase]

2021年2月18日(木) 7:43 Robert Elz :
> From scratch, no, but I maintain one.
>
> With all the details
>
> It isn't all that hard, but isn't trivial
> either, and amounts to quite a bit of code
> (and parsing all the bash extensions the
> way it does would make it harder and bigger)
>
> Like everything, it becomes a cost/benefit
> tradeoff, are the costs worth the benefits?

Yeah, right. I was so sleepy last night (in my timezone) that I have
written things that just came to my mind without thinking deeply,
sorry. The fixing cost actually depends on many factors such as how it
is implemented currently, how busy the maintainer is, ...

> That one is not for me (or you) to answer.

Now, they are fixed by Chet in the latest devel branch. Chet, I'm
sorry that it seems I have pushed you, and thank you very much for
your hard work.

--
Koichi

Re: syntax error while parsing a case command within `$(...)'

[Dale R. Worley]

Lawrence Velázquez  writes:
>>> `;;' is optional for the last case item.
>> 
>> The manual page (for my version) says it's required. If, in some
>> certain circumstances, it works without, that's nice.
>
> It's also required by POSIX.

Ah, now that's different.  Has the current man page been updated to
match?

Dale

Re: syntax error while parsing a case command within `$(...)'

[Lawrence Velázquez]

> On Feb 17, 2021, at 10:27 PM, Dale R. Worley  wrote:
> 
> Lawrence Velázquez  writes:
 `;;' is optional for the last case item.
>>> 
>>> The manual page (for my version) says it's required. If, in some
>>> certain circumstances, it works without, that's nice.
>> 
>> It's also required by POSIX.
> 
> Ah, now that's different.  Has the current man page been updated to
> match?

No.

https://www.gnu.org/software/bash/manual/html_node/Conditional-Constructs.html#index-case

Bash has permitted omitting the final ;; since the beginning (more
or less). I assume the documentation omits that detail for a reason
-- perhaps to discourage its use.

vq