Re: string substitution broken since 5.2
Le 03/11/2022 à 19:50, Chet Ramey écrivait : The option is enabled by default. If you want to restore the previous behavior, add `shopt -u patsub_replacement'. Having it enabled by default is not good, because it introduces side-effects for existing scripts. Shell has historically perpetuated legacy features to preserve the function of those no-longer maintained systems and associated scripts. Are there enough reasons to break this trend and stop preserving backward-compatibility with older scripts; by enabling new features that can affect the behaviour of previous code with side-effects? -- Léa Gris
Re: string substitution broken since 5.2
On Fri, Nov 4, 2022, 08:56 Léa Gris wrote: > Le 03/11/2022 à 19:50, Chet Ramey écrivait : > > The option is enabled by default. If you want to restore the previous > > behavior, add `shopt -u patsub_replacement'. > > Having it enabled by default is not good, because it introduces > side-effects for existing scripts. > there is , imo , no sense to comply to outdated specs to not add few more of the infinite base features nevermind .. > > Shell has historically perpetuated legacy features to preserve the > function of those no-longer maintained systems and associated scripts. > > Are there enough reasons to break this trend and stop preserving > backward-compatibility with older scripts; by enabling new features that > can affect the behaviour of previous code with side-effects? > > -- > Léa Gris > >
Re: string substitution broken since 5.2
On Fri, Nov 04, 2022 at 09:50:03AM +0100, Alex fxmbsw7 Ratchev wrote:
> On Fri, Nov 4, 2022, 08:56 Léa Gris wrote:
>
> > Le 03/11/2022 à 19:50, Chet Ramey écrivait :
> > > The option is enabled by default. If you want to restore the previous
> > > behavior, add `shopt -u patsub_replacement'.
> >
> > Having it enabled by default is not good, because it introduces
> > side-effects for existing scripts.
> >
>
> there is , imo , no sense to comply to outdated specs to not add few more
> of the infinite base features
>
> nevermind ..
There is no "spec" for this. It's not a POSIX feature. It's a bashism.
To be clear, we're talking about:
somecmd "${var//search/replace}"
where "replace" might be a string contained in a variable.
The reason this is an issue is because bash 5.2 has changed the default
behavior of this expansion in a way that not only changes the behavior of
existing scripts (possibly "breaking" them), but also makes it extremely
difficult to write one script that works across all bash versions.
In order to write a version-portable script that uses this feature,
now you have to choose between these two workarounds:
1) Put something like "shopt -u patsub_replacement 2>/dev/null || true"
at the top of your script.
2) Assign the result of the expansion to a temporary variable, and pass
the temp var to somecmd. Every. Single. Time.
Re: string substitution broken since 5.2
On Fri, Nov 4, 2022, 12:33 Greg Wooledge wrote:
> On Fri, Nov 04, 2022 at 09:50:03AM +0100, Alex fxmbsw7 Ratchev wrote:
> > On Fri, Nov 4, 2022, 08:56 Léa Gris wrote:
> >
> > > Le 03/11/2022 à 19:50, Chet Ramey écrivait :
> > > > The option is enabled by default. If you want to restore the previous
> > > > behavior, add `shopt -u patsub_replacement'.
> > >
> > > Having it enabled by default is not good, because it introduces
> > > side-effects for existing scripts.
> > >
> >
> > there is , imo , no sense to comply to outdated specs to not add few more
> > of the infinite base features
> >
> > nevermind ..
>
> There is no "spec" for this. It's not a POSIX feature. It's a bashism.
>
its a 'basic' to me .. viral .. essential
i am talking mr chet bash , sticks much to , draft specs , some , and much
on backward compatibility
its hard to add features like this
i.. would care ppl get informed of bugs they use cause , very old versions
in use , and so ly u can even assure future updates
for them , its 'it works for me'
to me , its like non acceptable
To be clear, we're talking about:
>
> somecmd "${var//search/replace}"
>
> where "replace" might be a string contained in a variable.
>
> The reason this is an issue is because bash 5.2 has changed the default
> behavior of this expansion in a way that not only changes the behavior of
> existing scripts (possibly "breaking" them), but also makes it extremely
> difficult to write one script that works across all bash versions.
>
> In order to write a version-portable script that uses this feature,
> now you have to choose between these two workarounds:
>
i mean yea , for those who 'portable'
my, 'posix++' specs are even less portable
not of the logic
of the diswantings of the parties there
in future , specs drafts will grow and addments done..
in future.. we are far away from it .. :(
> 1) Put something like "shopt -u patsub_replacement 2>/dev/null || true"
>at the top of your script.
>
there d be many such senselessnesses
>
> 2) Assign the result of the expansion to a temporary variable, and pass
>the temp var to somecmd. Every. Single. Time.
>
? i dont get that
i just say its for me painful to have more keep-code-old ppl than -updated-
.. as an end user
sorry , greets , peace
>
Re: string substitution broken since 5.2
On 11/4/22 3:56 AM, Léa Gris wrote: Le 03/11/2022 à 19:50, Chet Ramey écrivait : The option is enabled by default. If you want to restore the previous behavior, add `shopt -u patsub_replacement'. Having it enabled by default is not good, because it introduces side-effects for existing scripts. In my experience, it doesn't matter when you flip that switch. As soon as you change something, people will complain. The question is what should have higher priority: useful new features or backwards compatibility. In this case, I took the former. Shell has historically perpetuated legacy features to preserve the function of those no-longer maintained systems and associated scripts. Why are "no-longer-maintained systems" upgrading software? They're essentially frozen in amber. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: string substitution broken since 5.2
I now very much regret not commenting on this proposal during the pre-release period, and I apologise for not having done so. On Fri, 4 Nov 2022, 05:50 Chet Ramey, wrote: > > The option is enabled by default. If you want to restore the previous > behavior, add `shopt -u patsub_replacement'. > I am very disquieted by this pattern of creating breaking changes and then saying "just add shopt blahblah to go back to the old behaviour". This might be justifiable to a fix for a security problem (e.g. array subscripting), but *this* is just a nice-to-have feature. Suggesting that an existing script should be modified implies that scripts must necessarily have "maintainers" whose job includes tracking changes to the language provided by Bash, or that every piece of software installed on a system comes from a single source who can test combinations for compatibility. That is not the world that we live in. The average user out there has almost zero shell scripting experience, and even less competence to diagnose faults in their system. They rely on a package manager to install versions of basic utilities like Bash, and diligently apply updates to keep on top of security. They also get other software from sources who know nothing about them or their package manager. In this context the reasonable thing to do is to aim for maximal backwards compatibility, and only enable breaking changes when a special option or command is used. Can we please have an immediate point release turning this feature off by default, and then let's take another look at how this can be done without gratuitous breakage of existing code. -Martin PS: I'm puzzled why a bare '&' was chosen in the first place; that seems to provide the maximal likelihood of conflict with existing code. As an alternative that wouldn't need an enabling option, how about using "$BASH_MATCH" or "$&"? (I suspect the latter would be easier to use as a marker for a deferred expansion.) Rather than inserting a modal command or option, one possibility might be to use the same mechanism as handles POSIX "sh", and extend it to look for a trailing version number, and use that to enable new features.
Re: string substitution broken since 5.2
2022年11月4日(金) 23:26 Martin D Kealey : > I now very much regret not commenting on this proposal during the > pre-release period, and I apologise for not having done so. I actually raised the concern in [1], but Chet's reply [2] at the time was > I generally make new functionality that's controlled by a separate option > off by default in releases. I'll do the same here. Chet seems to have changed his mind after that (as Chet explained in another reply in this reply tree), and it was not made off in releases. [1] https://lists.gnu.org/archive/html/bug-bash/2022-01/msg00036.html [2] https://lists.gnu.org/archive/html/bug-bash/2022-01/msg00043.html
Re: string substitution broken since 5.2
that u word comply to very outdated specs i did word bye On Fri, Nov 4, 2022, 15:26 Martin D Kealey wrote: > I now very much regret not commenting on this proposal during the > pre-release period, and I apologise for not having done so. > > On Fri, 4 Nov 2022, 05:50 Chet Ramey, wrote: > > > > > The option is enabled by default. If you want to restore the previous > > behavior, add `shopt -u patsub_replacement'. > > > > I am very disquieted by this pattern of creating breaking changes and then > saying "just add shopt blahblah to go back to the old behaviour". > > This might be justifiable to a fix for a security problem (e.g. array > subscripting), but *this* is just a nice-to-have feature. > > Suggesting that an existing script should be modified implies that scripts > must necessarily have "maintainers" whose job includes tracking changes to > the language provided by Bash, or that every piece of software installed on > a system comes from a single source who can test combinations for > compatibility. > > That is not the world that we live in. > > The average user out there has almost zero shell scripting experience, and > even less competence to diagnose faults in their system. They rely on a > package manager to install versions of basic utilities like Bash, and > diligently apply updates to keep on top of security. They also get other > software from sources who know nothing about them or their package manager. > > In this context the reasonable thing to do is to aim for maximal backwards > compatibility, and only enable breaking changes when a special option or > command is used. > > Can we please have an immediate point release turning this feature off by > default, and then let's take another look at how this can be done without > gratuitous breakage of existing code. > > -Martin > > PS: > I'm puzzled why a bare '&' was chosen in the first place; that seems to > provide the maximal likelihood of conflict with existing code. > > As an alternative that wouldn't need an enabling option, how about using > "$BASH_MATCH" or "$&"? (I suspect the latter would be easier to use as a > marker for a deferred expansion.) > > Rather than inserting a modal command or option, one possibility might be > to use the same mechanism as handles POSIX "sh", and extend it to look for > a trailing version number, and use that to enable new features. >
Re: string substitution broken since 5.2
On Fri, Nov 04, 2022 at 01:30:16PM +0100, Alex fxmbsw7 Ratchev wrote:
> > 1) Put something like "shopt -u patsub_replacement 2>/dev/null || true"
> >at the top of your script.
> >
>
> there d be many such senselessnesses
>
> >
> > 2) Assign the result of the expansion to a temporary variable, and pass
> >the temp var to somecmd. Every. Single. Time.
> >
>
> ? i dont get that
OK. Let me offer a quick example script. It works as expected in
bash-5.1:
unicorn:~$ cat foo
#!/bin/bash
template='The best candy is clearly @CANDY@!'
candy='M&Ms'
printf '%s\n' "${template/@CANDY@/"$candy"}"
unicorn:~$ ./foo
The best candy is clearly M&Ms!
Now, let's run this script under bash-4.2:
unicorn:~$ bash-4.2 ./foo
The best candy is clearly "M&Ms"!
Uh oh! The quotes are wrong for bash-4.2. Bug #1 is filed for this issue.
As the maintainer of the script, I test a few things between bash-4.2
and bash-5.1 and I come up with this workaround:
unicorn:~$ cat foo
#!/bin/bash
template='The best candy is clearly @CANDY@!'
candy='M&Ms'
printf '%s\n' "${template/@CANDY@/$candy}"# unquote $candy to fix bug #1
unicorn:~$ bash-4.2 ./foo
The best candy is clearly M&Ms!
unicorn:~$ bash-5.1 ./foo
The best candy is clearly M&Ms!
Now it works on older systems too. Everything's fine... until bash-5.2.
unicorn:~$ bash-5.2 ./foo
The best candy is clearly M@CANDY@Ms!
The workaround for bug #1 causes bug #2 on bash-5.2. To make the script
work on all three versions of bash, we need a different workaround:
unicorn:~$ cat foo
#!/bin/bash
template='The best candy is clearly @CANDY@!'
candy='M&Ms'
message=${template/@CANDY@/"$candy"} # Work around bug #1 and #2.
printf '%s\n' "$message"
unicorn:~$ bash-4.2 ./foo
The best candy is clearly M&Ms!
unicorn:~$ bash-5.1 ./foo
The best candy is clearly M&Ms!
unicorn:~$ bash-5.2 ./foo
The best candy is clearly M&Ms!
And there you have it. You're allowed to quote "$candy" in a variable
assignment, as long as the whole parameter expansion ${...} remains
unquoted, and it'll work properly in all 3 versions. I'm using "message"
as a temporary variable, whose sole purpose is to work around this issue.
Re: string substitution broken since 5.2
On Fri, Nov 4, 2022, 17:03 Greg Wooledge wrote:
> On Fri, Nov 04, 2022 at 01:30:16PM +0100, Alex fxmbsw7 Ratchev wrote:
> > > 1) Put something like "shopt -u patsub_replacement 2>/dev/null || true"
> > >at the top of your script.
> > >
> >
> > there d be many such senselessnesses
> >
> > >
> > > 2) Assign the result of the expansion to a temporary variable, and pass
> > >the temp var to somecmd. Every. Single. Time.
> > >
> >
> > ? i dont get that
>
> OK. Let me offer a quick example script. It works as expected in
> bash-5.1:
>
> unicorn:~$ cat foo
> #!/bin/bash
> template='The best candy is clearly @CANDY@!'
> candy='M&Ms'
> printf '%s\n' "${template/@CANDY@/"$candy"}"
>
> unicorn:~$ ./foo
> The best candy is clearly M&Ms!
>
> Now, let's run this script under bash-4.2:
>
> unicorn:~$ bash-4.2 ./foo
> The best candy is clearly "M&Ms"!
>
> Uh oh! The quotes are wrong for bash-4.2. Bug #1 is filed for this issue.
> As the maintainer of the script, I test a few things between bash-4.2
> and bash-5.1 and I come up with this workaround:
>
> unicorn:~$ cat foo
> #!/bin/bash
> template='The best candy is clearly @CANDY@!'
> candy='M&Ms'
> printf '%s\n' "${template/@CANDY@/$candy}"# unquote $candy to fix bug
> #1
>
> unicorn:~$ bash-4.2 ./foo
> The best candy is clearly M&Ms!
> unicorn:~$ bash-5.1 ./foo
> The best candy is clearly M&Ms!
>
> Now it works on older systems too. Everything's fine... until bash-5.2.
>
> unicorn:~$ bash-5.2 ./foo
> The best candy is clearly M@CANDY@Ms!
>
> The workaround for bug #1 causes bug #2 on bash-5.2. To make the script
> work on all three versions of bash, we need a different workaround:
>
> unicorn:~$ cat foo
> #!/bin/bash
> template='The best candy is clearly @CANDY@!'
> candy='M&Ms'
> message=${template/@CANDY@/"$candy"} # Work around bug #1 and #2.
> printf '%s\n' "$message"
>
> unicorn:~$ bash-4.2 ./foo
> The best candy is clearly M&Ms!
> unicorn:~$ bash-5.1 ./foo
> The best candy is clearly M&Ms!
> unicorn:~$ bash-5.2 ./foo
> The best candy is clearly M&Ms!
>
> And there you have it. You're allowed to quote "$candy" in a variable
> assignment, as long as the whole parameter expansion ${...} remains
> unquoted, and it'll work properly in all 3 versions. I'm using "message"
> as a temporary variable, whose sole purpose is to work around this issue.
>
mate , thank you for investing time in explainmennts
what on yours i ask , o i see now , 4.2 ..
i quote right in ${ // - to me , preserved quotes are • an invalid youth
bug to be replaced with better by bash.c changes
which it seemfully did
is quotes preserveness the only bug here i see ?
root@localhost:~# a=i_am_foo b=_ c=${b}am$b d=${b}be$b e=${a//"$c"/"$d"} ;
printf ' --> %s\n' "$e" "${a//"$c"/"$d"}"
--> i_be_foo
--> i_be_foo
for me , trying to support such wrongness , resulted ( along other same
logics.. ) in a 'best is newest upstream version' rule
>
Re: string substitution broken since 5.2
On 11/4/22 12:22 PM, Alex fxmbsw7 Ratchev wrote:
what on yours i ask , o i see now , 4.2 ..
i quote right in ${ // - to me , preserved quotes are • an invalid youth
bug to be replaced with better by bash.c changes
which it seemfully did
The question is, as always, tradeoffs. Do you fix bugs or not? Do you make
the behavior more consistent across operations, even if incompatibilities
(as above) result? Do you tighten up behavior where previous versions were
too permissive, even if people will have to change their code as a result,
or not? Do you introduce new features, or not? Do you make things opt-in,
or opt-out? The answers are not always the same.
Greg, for example, puts a lot of value in being able to use the same code
from bash-4.2, which was released in 2010, through the current version.
That's the tradeoff he's willing to make when answering these questions.
This was the entire rationale for the compatibility mode in the first
place: if you want things as they were before, set the appropriate
compatibility variable. The same is true of features controlled by shell
options.
Backwards compatibility is important, and I put a lot of effort into it.
But I nudge things forward when I think it's warranted. As you've seen,
there's always spirited disagreement.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: string substitution broken since 5.2
addition to my greycat-answer mail
.. i didnt use & there at all , my fault .. but with & its no else
root@localhost:~# a=i_am_foo b=_ c=${b}am$b d=$b\&$b e=${a//"$c"/"$d"} ;
printf ' --> %s\n' "$e" "${a//"$c"/"$d"}"
--> i_&_foo
--> i_&_foo
sorry
short imo , the ' " not parsing is outdated , and got fixed later
u may want bash v<1 compatibility ?
i just say its two horsrs , one about old and keep , the other about new
and updated
On Fri, Nov 4, 2022, 17:22 Alex fxmbsw7 Ratchev wrote:
>
>
> On Fri, Nov 4, 2022, 17:03 Greg Wooledge wrote:
>
>> On Fri, Nov 04, 2022 at 01:30:16PM +0100, Alex fxmbsw7 Ratchev wrote:
>> > > 1) Put something like "shopt -u patsub_replacement 2>/dev/null ||
>> true"
>> > >at the top of your script.
>> > >
>> >
>> > there d be many such senselessnesses
>> >
>> > >
>> > > 2) Assign the result of the expansion to a temporary variable, and
>> pass
>> > >the temp var to somecmd. Every. Single. Time.
>> > >
>> >
>> > ? i dont get that
>>
>> OK. Let me offer a quick example script. It works as expected in
>> bash-5.1:
>>
>> unicorn:~$ cat foo
>> #!/bin/bash
>> template='The best candy is clearly @CANDY@!'
>> candy='M&Ms'
>> printf '%s\n' "${template/@CANDY@/"$candy"}"
>>
>> unicorn:~$ ./foo
>> The best candy is clearly M&Ms!
>>
>> Now, let's run this script under bash-4.2:
>>
>> unicorn:~$ bash-4.2 ./foo
>> The best candy is clearly "M&Ms"!
>>
>> Uh oh! The quotes are wrong for bash-4.2. Bug #1 is filed for this
>> issue.
>> As the maintainer of the script, I test a few things between bash-4.2
>> and bash-5.1 and I come up with this workaround:
>>
>> unicorn:~$ cat foo
>> #!/bin/bash
>> template='The best candy is clearly @CANDY@!'
>> candy='M&Ms'
>> printf '%s\n' "${template/@CANDY@/$candy}"# unquote $candy to fix
>> bug #1
>>
>> unicorn:~$ bash-4.2 ./foo
>> The best candy is clearly M&Ms!
>> unicorn:~$ bash-5.1 ./foo
>> The best candy is clearly M&Ms!
>>
>> Now it works on older systems too. Everything's fine... until bash-5.2.
>>
>> unicorn:~$ bash-5.2 ./foo
>> The best candy is clearly M@CANDY@Ms!
>>
>> The workaround for bug #1 causes bug #2 on bash-5.2. To make the script
>> work on all three versions of bash, we need a different workaround:
>>
>> unicorn:~$ cat foo
>> #!/bin/bash
>> template='The best candy is clearly @CANDY@!'
>> candy='M&Ms'
>> message=${template/@CANDY@/"$candy"} # Work around bug #1 and
>> #2.
>> printf '%s\n' "$message"
>>
>> unicorn:~$ bash-4.2 ./foo
>> The best candy is clearly M&Ms!
>> unicorn:~$ bash-5.1 ./foo
>> The best candy is clearly M&Ms!
>> unicorn:~$ bash-5.2 ./foo
>> The best candy is clearly M&Ms!
>>
>> And there you have it. You're allowed to quote "$candy" in a variable
>> assignment, as long as the whole parameter expansion ${...} remains
>> unquoted, and it'll work properly in all 3 versions. I'm using "message"
>> as a temporary variable, whose sole purpose is to work around this issue.
>>
>
> mate , thank you for investing time in explainmennts
>
> what on yours i ask , o i see now , 4.2 ..
> i quote right in ${ // - to me , preserved quotes are • an invalid youth
> bug to be replaced with better by bash.c changes
> which it seemfully did
>
> is quotes preserveness the only bug here i see ?
>
> root@localhost:~# a=i_am_foo b=_ c=${b}am$b d=${b}be$b e=${a//"$c"/"$d"}
> ; printf ' --> %s\n' "$e" "${a//"$c"/"$d"}"
> --> i_be_foo
> --> i_be_foo
>
> for me , trying to support such wrongness , resulted ( along other same
> logics.. ) in a 'best is newest upstream version' rule
>
>>
Re: string substitution broken since 5.2
On Fri, Nov 4, 2022, 19:42 Chet Ramey wrote:
> On 11/4/22 12:22 PM, Alex fxmbsw7 Ratchev wrote:
>
> > what on yours i ask , o i see now , 4.2 ..
> > i quote right in ${ // - to me , preserved quotes are • an invalid youth
> > bug to be replaced with better by bash.c changes
> > which it seemfully did
>
> The question is, as always, tradeoffs. Do you fix bugs or not? Do you make
> the behavior more consistent across operations, even if incompatibilities
> (as above) result? Do you tighten up behavior where previous versions were
> too permissive, even if people will have to change their code as a result,
> or not? Do you introduce new features, or not? Do you make things opt-in,
> or opt-out? The answers are not always the same.
>
i d ask me one other .. asking where 'the specific' s place is , to code it
there , where it belongs
Greg, for example, puts a lot of value in being able to use the same code
> from bash-4.2, which was released in 2010, through the current version.
> That's the tradeoff he's willing to make when answering these questions.
>
im a poor invalid-rent in swiss assed , tech++ privateer
to me only newest versions count ( preferabily containing feature addments )
to me as pro bash coder ( i sitted and learned and teachen further by
interactive computer chats ( mostly irc ) ) for years
.. to me , knowing all the bugs bash has over some time , makes me have big
anti reasons to use old bashes
i mean its one to support old
but , why does that old exist
redhat 5 6 7
they cant update ? or use extra , old softwre , to exploit flaws in sw
.. to support such .. no me man , not for me
i mean i got clauses here and there , in my coding freelance ing , one is
'pref. updated debian to run' - that is debian all public short tree s ,
not just useless 'stable' - and i d put it on as sometimes
i try to say , .. stuff is incomplete , and if fixes dont come , just as no
good as always it is
just as ppl support 4.2 i 'only newest'
greg did great examples btw
This was the entire rationale for the compatibility mode in the first
> place: if you want things as they were before, set the appropriate
> compatibility variable. The same is true of features controlled by shell
> options.
Backwards compatibility is important, and I put a lot of effort into it.
> But I nudge things forward when I think it's warranted. As you've seen,
> there's always spirited disagreement.
>
it s a vital work for you , ye
i can just say , changing quotes to be interpreted is imo vital too :p i
mean done correctly better now than before
when , you have smth , like sh or bash , or non computer stuff
it takes , its known-to-be , material-beeing
which will in time need changes to be done to it
hm hm , greets , x
--
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>
