Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: netbsd
Compiler: gcc
Compilation CFLAGS: -O2 -D_FORTIFY_SOURCE=2 -I/usr/include
-Wno-parentheses -Wno-format-security
uname output: NetBSD cq60-615dx.blilly.net 9.0 NetBSD 9.0 (GENERIC)
#0: Fri Feb 14 00:06:28 UTC 2020
mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
amd64
Machine Type: x86_64--netbsd
Bash Version: 5.0
Patch Level: 17
Release Status: release
Description:
Bash parameter expansion (remove largest trailing match, remove
largest leading match, pattern replacement) does not work
Tested on OpenSUSE Leap 15.2, bash version 4.4.2.3(1)-release
(x86_64-suse-linux-gnu)
OpenBSD 6.7 bash version 5.0.17(1)-release (x86_64-unknown-openbsd6.7)
NetBSD 9.0 bash version 5.0.17(1)-release (x86_64--netbsd)
FreeBSD 12.1-STABLE bash version 5.0.18(2)-release (amd64-portbld-freebsd12.1)
Same results in all cases; this report posted from NetBSD 9.0.
As an example, consider constructing a path from components; the full
path should
have a single separator (slash), so any stray slashes at the tail of
the first part or the
start of the last part should be elided. There are many ways that
that can be done,
the "Repeat By" section demonstrates only a few examples.
Relevant manual sections are:
https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion
https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Pattern-Matching
Repeat-By:
$ bash shellbug
Running bash from /usr/pkg/bin/bash
Bash version 5.0.17(1) release GNU
dir=/foo/bar/baz/// separator1=/ separator2(octal escape)=\057
number0=* number1=+ pattern1=${number1}(${separator1})=+(/)
pattern2=${number0}([${separator2}])=*([\057]) base=///grimble/pritz
path=${dir%%${pattern1}}${separator1}${base##${pattern2}}=${dir%%+(/)}/${base##*([\057])}=/foo/bar/baz///grimble/pritz
path=${dir/%${pattern1}/${separator1}}${base/#${pattern2}/}=${dir/%+(/)//}${base/#*([\057])/}=/foo/bar/baz//grimble/pritz
Fix:
(demonstrates how a functioning shell handles exactly the same cases):
$ ksh93 shellbug
Running ksh93 from /usr/pkg/bin/ksh93
$Id: Version AJM 93u+ 2012-08-01 $
dir=/foo/bar/baz/// separator1=/ separator2(octal escape)=\057
number0=* number1=+ pattern1=${number1}(${separator1})=+(/)
pattern2=${number0}([${separator2}])=*([\057]) base=///grimble/pritz
path=${dir%%${pattern1}}${separator1}${base##${pattern2}}=${dir%%+(/)}/${base##*([\057])}=/foo/bar/baz/grimble/pritz
path=${dir/%${pattern1}/${separator1}}${base/#${pattern2}/}=${dir/%+(/)//}${base/#*([\057])/}=/foo/bar/baz/grimble/pritz
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
Robert, Did you look at the relevant bash manual pages linked in the bug report? > >
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
On Sat, Aug 29, 2020, 13:48 Robert Elz wrote: > Date:Sat, 29 Aug 2020 13:33:37 -0400 > From: Bruce Lilly > Message-ID: n...@mail.gmail.com> > > | Did you look at the relevant bash manual pages linked in the bug > report? > > Enough to know that [\057] doesn't mean what you seem to believe it should. > > kre > Evidently not enough to see the specifications for pattern matching... > >
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
Please don't assume that something output by printf (without quoting) for clarity is representative of actual expansion by the shell when properly quoted. On Sat, Aug 29, 2020, 14:52 Koichi Murase wrote: > 2020-08-29 23:47 Bruce Lilly : > > Description: > > Bash parameter expansion (remove largest trailing match, > > remove largest leading match, pattern replacement) does not work > > In short, (1) ``octal escape \057'' doesn't have special meaning in > Bash glob patterns. `[\057]' means just one of a literal backslash or > digits 0, 5, or 7. (2) You need to set `shopt -s extglob` to use > extended glob patterns like `+(...)' and `*(...)'. Please look at the > paragraph above the list of extended patterns in the manual. > > -- > Koichi >
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
Thanks for the pointer to "shopt"; I'll check when I get a chance. On Sat, Aug 29, 2020, 14:52 Koichi Murase wrote: > 2020-08-29 23:47 Bruce Lilly : > > Description: > > Bash parameter expansion (remove largest trailing match, > > remove largest leading match, pattern replacement) does not work > > In short, (1) ``octal escape \057'' doesn't have special meaning in > Bash glob patterns. `[\057]' means just one of a literal backslash or > digits 0, 5, or 7. (2) You need to set `shopt -s extglob` to use > extended glob patterns like `+(...)' and `*(...)'. Please look at the > paragraph above the list of extended patterns in the manual. > > -- > Koichi >
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
Unfortunately, because bash is GPL, I can't post the copyrighted script which is covered by a non-GPL license. On Sat, Aug 29, 2020, 15:22 Ilkka Virta wrote: > On Sat, Aug 29, 2020 at 9:56 PM Bruce Lilly wrote: > >> Please don't assume that something output by printf (without quoting) for >> clarity is representative of actual expansion by the shell when properly >> quoted. >> > > If you don't want people to assume (and you shouldn't, if you want them to > help you), > you'd better post the actual script you use, and not just something output > by printf. > Without that, it's impossible to check what quoting you used, or to > reproduce the issue. > >
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
On Sat, Aug 29, 2020, 15:40 Koichi Murase wrote: > Don't worry. In this case, the GPL doesn't apply. Please read the > following Q&A. > > https://www.gnu.org/licenses/gpl-faq.en.html#IfInterpreterIsGPL > > Even if your original `shellbug' is licensed under something other, > and it prohibits to make it public, you can still create another > script to reproduce the same issue. [...] > It's a bit more complicated than that; if, for example, some excerpt ended up in regression tests, there would be a question about whether or not there was a copyright violation. As I understand the GPL (IANAL), it requires all parts of a "work" to be GPL'd, and that wouldn't be possible for any parts of the script that ended up in bash regression tests.
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
On Sat, Aug 29, 2020, 15:27 Koichi Murase wrote:
> I assumed that you have written like
>
> separator2='\057'
> pattern1="${number1}(${separator1})"
>
> because otherwise, it doesn't work with ksh93 either.
>
You are correct.
To explain it in more detail, first, these parameter expansions work
> for me with separator1='/'. The pattern that you constructed from
> separator2='\057' is just not supported by Bash.
That's surprising, as octal and hexadecimal escapes are fairly common.
Next, you specify
> exactly the same script `shellbug' to both Bash and ksh93, so I could
> assume that you are missing the Bash-specific `shopt -s extglob' in
> the script because otherwise, ksh93 would have complained it. That is
> all what I can guess from your original post.
>
Yes, I'm still looking into that (along with updating a couple of FreeBSD
machines, eating lunch, and monitoring a couple of downloads). I'll
obviously have to wrap the "shopt" bit in a wrapper; is there some minimum
bash version that supports it?
>
>
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
On Sat, Aug 29, 2020 at 4:53 PM Koichi Murase wrote:
[...]
> That's an interesting discussion. I don't know how you define the
> "work", but basically GPL only affects the derivative
> programs/software but not all the "work" including the output of the
> programs or the knowledge obtained in running/developing the code.
I'm using it in the sense of the GPL, as in the part of the preamble:
"software and other kinds of works".
It's not clearly defined; vagueness is one of the problems with the GPL.
I'm assuming that for bash it basically means everything that's included
in the distributed package. "Everything" is basically what GPL 3 section
5c says must be covered by the GPL.
[...]
> Otherwise, anyone who
> read a GPL code in the past cannot write any non-GPL programs because
> one cannot prove the experience of reading the GPL code doesn't affect
> any code that he/she writes thereafter. I sometimes hear that someone
> avoids hiring programmers who have read a GPL code in the past for
> defensive purposes, but I believe it's a matter of degree.
That's basically the issue, and it goes both ways (i.e. also people
writing GPL'ed code based on non-GPL source (or slapping a GPL
license on non-GPL code); there's a rather famous case
http://undeadly.org/cgi?action=article&sid=20070913014315 ).
> 2020-08-30 5:07 Bruce Lilly :
> > That's surprising, as octal and hexadecimal escapes are fairly
> > common.
>
> Yes, I know that it is confusing to those who are familiar with modern
> Perl-style regular expressions. But historically, POSIX regular
> expressions do not support the backslash escape sequences in bracket
> expressions `[...]'. The backslash escape sequences in bracket
> expressions were the extension historically. As far as I know, in
> POSIX, only awk supports backslash sequences in regular expressions.
Actually, it works (portably) with
separator2=$'\057'
> Bash 2.02 supports `shopt -s extglob', so you can assume every Bash
> has the support. If you are still failing to get an expected
> behavior, you can just put the line `shopt -s extglob' in the
> beginning of the script. In the case of the above mentioned reduced
> case, you can write like this:
The magic "shopt" incantation was indeed the key: thank you.
I guess this can be considered closed.
It would be nice, however, if there were an environment variable that
could cause bash to default to extglob. My concern is some variants of
make that send recipes line-by-line to a shell, in which case each line
might need to be prefixed by a test for ${BASH_VERSION} and the
shopt builtin.
Re: Bash parameter expansion (remove largest trailing match, remove largest leading match, pattern replacement) does not work
On Sat, Aug 29, 2020 at 9:12 PM Lawrence Velázquez wrote: > (a) $'...' is not POSIX. For instance, dash does not recognize it. dash also doesn't have adequate pattern matching for the example task (building a path while ensuring no empty components); it has no way to specify one-or-more (or zero-or-more) occurrences of a pattern such as a slash.
