Re: GROUPS

[Štěpán Němec]

On Tue, 10 Aug 2021 23:39:47 +0700
Robert Elz wrote:

> Date:Tue, 10 Aug 2021 10:22:29 -0400
> From:Chet Ramey 
> Message-ID:  <731876fc-39c0-4388-0c9e-bf560921b...@case.edu>
>
>   | In this case, you are using features outside what POSIX specifies.
>
> Using a variable name that's outside what POSIX specifies is hardly
> using a feature that's outside POSIX - if it were then there would be
> no safe non-trivial scripts, since any variable name might be made magic
> by some shell or other (and no, there's nothing special about all upper
> case variable names).Only those defined by POSIX to have some special
> meaning mean something, and even those can be re-used for another purpose
> as long as one doesn't expect the special behaviour to function correctly.

Quoting POSIX.1-2017 on environment variables [1]:

  Environment variable names used by the utilities in the Shell and
  Utilities volume of POSIX.1-2017 consist solely of uppercase letters,
  digits, and the  ( '_' ) from the characters defined in
  Portable Character Set and do not begin with a digit. Other characters
  may be permitted by an implementation; applications shall tolerate the
  presence of such names. Uppercase and lowercase letters shall retain
  their unique identities and shall not be folded together. The name space
  of environment variable names containing lowercase letters is reserved
  for applications. Applications can define any environment variables with
  names from this name space without modifying the behavior of the
  standard utilities.

So, no matter if "special" or not by your standards, there is definitely
a significant difference between "all upper case" variable names and the
rest, which IMO in itself constitutes a sufficient argument for sticking
to non-all-caps variable names for custom functionality (AKA
"applications" above or one's own shell scripts).

[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html

-- 
Štěpán

Re: GROUPS

[Andreas Schwab]

On Aug 11 2021, Štěpán Němec wrote:

> Quoting POSIX.1-2017 on environment variables [1]:

Note that GROUPS is not an environment variable in bash, it is not
exported.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: GROUPS

[Chet Ramey]

On 8/10/21 12:39 PM, Robert Elz wrote:


   | In this case, you are using features outside what POSIX specifies.

Using a variable name that's outside what POSIX specifies is hardly
using a feature that's outside POSIX - if it were then there would be
no safe non-trivial scripts, since any variable name might be made magic
by some shell or other (and no, there's nothing special about all upper
case variable names).Only those defined by POSIX to have some special
meaning mean something, and even those can be re-used for another purpose
as long as one doesn't expect the special behaviour to function correctly.


As long as POSIX doesn't define a variable to have some special meaning, it
doesn't have anything to say about how a shell chooses to use it. It's not
a POSIX issue, period.

In this case, as is the case with just about all of the bash special
variables, you can unset it and it will lose its special meaning.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: GROUPS

[Chet Ramey]

On 8/10/21 8:56 PM, Franklin, Jason wrote:


What surprised me was that Bash-specific "magic" variables did not lose
their "magic" qualities when Bash was invoked in a POSIX-compliant mode
of execution.  


That's not what bash posix mode is for. POSIX does not prohibit extensions.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: gettext feature request

[Chet Ramey]

On 8/10/21 2:56 PM, Mike Jonkmans wrote:


noexpand_translation
If  set,  bash encloses the translated results of $"..."
quoting in single quotes instead of double  quotes.   If
the string is not translated, this has no effect.

It will be in the next devel branch push.


So noexpand_translation is off by default.


Yes. The option changes the existing historical behavior.


I suggest to use the positive version: expand_translation
As that circumvents the double negation.


I understand that, and I considered it. I chose to have the option reflect
its purpose: to alter the default behavior.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: RFE: new option affecting * expansion

[Chet Ramey]

On 8/10/21 5:08 PM, Chris F.A. Johnson wrote:


It would be nice if there were an option to allow * to expand sorted
by timestamp rather than aphabetically.


When you say `timestamp' I assume you mean by last modification time.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: RFE: new option affecting * expansion

[Greg Wooledge]

On Wed, Aug 11, 2021 at 10:43:12AM -0400, Chet Ramey wrote:
> On 8/10/21 5:08 PM, Chris F.A. Johnson wrote:
> > 
> > It would be nice if there were an option to allow * to expand sorted
> > by timestamp rather than aphabetically.
> 
> When you say `timestamp' I assume you mean by last modification time.

That was my interpretation too.

Re: GROUPS

[Robert Elz]

Date:Wed, 11 Aug 2021 10:16:42 -0400
From:Chet Ramey 
Message-ID:  <26e01365-d7f0-448d-dc4d-83f244bd0...@case.edu>

  | As long as POSIX doesn't define a variable to have some special meaning, it
  | doesn't have anything to say about how a shell chooses to use it. It's not
  | a POSIX issue, period.

Perhaps - but that's true of all variables.  How would you ever safely write
a script using variables, if any of them might be usurped by the shell for
some special purpose?   Much better for the shell (as much as is possible)
to get out of the way for variables the user initialises, and retain
special meaning for only variables that are used without being set.

  | In this case, as is the case with just about all of the bash special
  | variables, you can unset it and it will lose its special meaning.

Sure, but to do that the script writer must either unset every variable
the script is going to use before using it (in which case the possibility
of passing in values via the environment is lost) or somehow (via magic)
know which variable names are magic in every possible shell that might
ever exist and be used to run the script.

Even then (using either solution), what does a script do if it wants to
be invoked as:

GROUPS='abelian ...' script [options]

?   That works with most shells (the script just accesses ${GROUPS}
or perhaps first includes ": ${GROUPS=a b c}" to set the default value
in case one is not provided.   I write scripts like that quite often
(not using GROUPS as the variable, but some of the ones I do use
could easily be made magic in some shell or other - a few have names
that make that entirely plausible (like MINS)).

My point was that where the bash doc says:

   Assignments to MAGIC have no effect.  If MAGIC is unset,
   it loses its special properties, even if it is subsequently
   reset.

which (with s/MAGIC/whatever/) appears in the descriptions of
several variables, including GROUPS, it could just as easily
say (assuming the implementation were to make this happen)

   If MAGIC is set or unset it loses its special properties

followed by either "which cannot be regained for the lifetime of
the shell" or "which can be returned by ..." if some mechanism
is provided to allow that to happen.

Further this can safely be changed, as because of "Assignments
to MAGIC have no effect" no bash script can be depending on being
able to do "MAGIC=abracadabra" with any useful purpose, thus we
can assume that none do so (or not without "unset MAGIC" first).
Since scripts written for other shells do not know they need to
do "unset MAGIC" to make "MAGIC=abracadabra" work (it simply does)
they can't be expected to do that, and requiring scripts to be
modified to work with bash doesn't seem like a productive design.

So, if the change above were made, it would break nothing, but
make things work much better.

kre

Re: GROUPS

[Chet Ramey]

On 8/11/21 2:06 PM, Robert Elz wrote:

 Date:Wed, 11 Aug 2021 10:16:42 -0400
 From:Chet Ramey 
 Message-ID:  <26e01365-d7f0-448d-dc4d-83f244bd0...@case.edu>

   | As long as POSIX doesn't define a variable to have some special meaning, it
   | doesn't have anything to say about how a shell chooses to use it. It's not
   | a POSIX issue, period.

Perhaps - but that's true of all variables.  How would you ever safely write
a script using variables, if any of them might be usurped by the shell for
some special purpose?   Much better for the shell (as much as is possible)
to get out of the way for variables the user initialises, and retain
special meaning for only variables that are used without being set.


I believe I'd rather have variables behave as they're documented. It's more
predictable. Bash allows variables in the environment to override special
variables -- for example, UID and EUID -- and I've gotten bug reports about
that behavior for forever. Here's a relatively recent one:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900564



   | In this case, as is the case with just about all of the bash special
   | variables, you can unset it and it will lose its special meaning.

Sure, but to do that the script writer must either unset every variable
the script is going to use before using it (in which case the possibility
of passing in values via the environment is lost) or somehow (via magic)
know which variable names are magic in every possible shell that might
ever exist and be used to run the script.


That's one of the problems here: the author never intended this script to
run in any shell other that Debian's dash. If they had, maybe they would
have tested it on other shells, even other shells that Debian includes.

The other problem is the OP assuming that posix mode would change it, when
posix mode means something else.



Even then (using either solution), what does a script do if it wants to
be invoked as:

GROUPS='abelian ...' script [options]

?   


That case works, see above. Maybe it shouldn't, but it does.

That works with most shells (the script just accesses ${GROUPS}

or perhaps first includes ": ${GROUPS=a b c}" to set the default value
in case one is not provided.   I write scripts like that quite often
(not using GROUPS as the variable, but some of the ones I do use
could easily be made magic in some shell or other - a few have names
that make that entirely plausible (like MINS)).

My point was that where the bash doc says:

Assignments to MAGIC have no effect.  If MAGIC is unset,
it loses its special properties, even if it is subsequently
reset.

which (with s/MAGIC/whatever/) appears in the descriptions of
several variables, including GROUPS, it could just as easily
say (assuming the implementation were to make this happen)

If MAGIC is set or unset it loses its special properties


It could have, but I'm not seeing a good enough reason to change it
25-30 years in.

The original implementation was read-only, but that caused bug reports
because trying to unset a readonly variable is a fatal error in posix
mode.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: RFE: new option affecting * expansion

[Chris F.A. Johnson]

On Wed, 11 Aug 2021, Chet Ramey wrote:


On 8/10/21 5:08 PM, Chris F.A. Johnson wrote:


It would be nice if there were an option to allow * to expand sorted
by timestamp rather than aphabetically.


When you say `timestamp' I assume you mean by last modification time.


Yes, that's what I meant.

--
   Chris F.A. Johnson 
   === Author: ===
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
   Pro Bash Programming: Scripting the GNU/Linux shell (2009, Apress)

Re: gettext feature request

[Jean-Jacques Brucker]

Le 10/08/2021 à 17:22, Chet Ramey a écrit :

This is the current description:

noexpand_translation
If  set,  bash encloses the translated results of $"..."
quoting in single quotes instead of double  quotes.   If
the string is not translated, this has no effect.

It will be in the next devel branch push.



Thank a lot Chet.

I still think it would have been more consistent to extend the $'...' 
syntax.


But this change is already a great feature, and I hope the gettext 
developers will warmly welcome it and then update their documentation.


Finally, this functionality which already reinforces the security, does 
not prevent another option which one could call 'translate_c_string' to 
translate the $'strings' (without any expansion).


(my idea of an additional variable like 'TEXTCDOMAIN' was indeed not 
that great)


If both options existed ('noexpand_translation' AND 
'translate_c_string'), I would prefer to use the second which is even 
more secure (either my strings are translated or they are not, but they 
can never be expanded).


But already I'm delighted to not plan horrible workarounds anymore, and 
if only 'noexpand_translation' is released, I'll be happy to use it. :-)


Cheers,
Jean-Jacques.



OpenPGP_0xA3983A40D1458443.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: GROUPS

[Franklin, Jason]

Chet:

My apologies in advance for not responding in thread.  The bug-bash
archive interface doesn't expose the "Message-ID" header anywhere I can
find, and I am not a subscriber.  I suppose I should become one.  :)

> I believe I'd rather have variables behave as they're documented. It's more
> predictable. 

Is it?

The whole point of a standard like POSIX is that I should be able to
write my code in accordance with the standard and have it run on
compliant systems that I may not even be aware of or that might not have
been developed yet.

The standard determines what is "predictable" behavior, not the Bash
documentation.

> That's one of the problems here: the author never intended this script to
> run in any shell other that Debian's dash. If they had, maybe they would
> have tested it on other shells, even other shells that Debian includes.

The author intended for portability and POSIX compliance.  Otherwise,
the script header would not have read "#! /bin/sh".  It runs as expected
on dash and ksh.

POSIX is supposed to enable portability.  Compliant code should run the
same on any compliant interpreter or compiler.  How can I possibly test
on all systems that try to be compliant?  I may not be aware of all of
them or one may not even exist yet.

Requiring someone to test on all possible current and future shells that
claim POSIX-compliance defeats the purpose of a reference.

As an aside:  I believe in thorough testing in as many environments as
feasible!

> Even then (using either solution), what does a script do if it wants to
> be invoked as:
> 
> GROUPS='abelian ...' script [options]
> 
> ? 
> 
> That case works, see above. Maybe it shouldn't, but it does.

This doesn't work unless it was recently fixed.  A variation does...

bash-5.0$ echo $BASH_VERSION
5.0.17(1)-release
bash-5.0$ GROUPS=FOO bash -c 'echo $GROUPS'
1000
bash-5.0$ GROUPS=FOO bash --posix -c 'echo $GROUPS'
1000
bash-5.0$ env GROUPS=FOO bash -c 'echo $GROUPS'
FOO
bash-5.0$ env GROUPS=FOO bash --posix -c 'echo $GROUPS'
FOO

The same is the case for UID...

bash-5.0$ UID=x bash -c 'echo $UID'
bash: UID: readonly variable
1000

I could be wrong here if this was recently fixed.

> It could have, but I'm not seeing a good enough reason to change it
> 25-30 years in.

I think the result is that this report will surface at regular intervals
for decades to come.

-- 
Jason Franklin

Re: GROUPS

[Greg Wooledge]

On Wed, Aug 11, 2021 at 08:00:12PM -0400, Franklin, Jason wrote:
> This doesn't work unless it was recently fixed.  A variation does...
> 
> bash-5.0$ echo $BASH_VERSION
> 5.0.17(1)-release
> bash-5.0$ GROUPS=FOO bash -c 'echo $GROUPS'
> 1000
> bash-5.0$ GROUPS=FOO bash --posix -c 'echo $GROUPS'
> 1000
> bash-5.0$ env GROUPS=FOO bash -c 'echo $GROUPS'
> FOO
> bash-5.0$ env GROUPS=FOO bash --posix -c 'echo $GROUPS'
> FOO

You can't overwrite GROUPS while you're inside of bash, apparently not
even in the temporary assignment context before a simple command.  So,
in order for the first form to work as expected, you'd have to be in a
shell other than bash.

unicorn:~$ ksh
$ GROUPS=FOO bash -c 'echo "$GROUPS"'
FOO

And so on.  Either that, or unset GROUPS first.

Re: GROUPS

[Franklin, Jason]

On Wed, 2021-08-11 at 20:36 -0400, Greg Wooledge wrote:
> On Wed, Aug 11, 2021 at 08:00:12PM -0400, Franklin, Jason wrote:
> > This doesn't work unless it was recently fixed.  A variation does...
> > 
> > bash-5.0$ echo $BASH_VERSION
> > 5.0.17(1)-release
> > bash-5.0$ GROUPS=FOO bash -c 'echo $GROUPS'
> > 1000
> > bash-5.0$ GROUPS=FOO bash --posix -c 'echo $GROUPS'
> > 1000
> > bash-5.0$ env GROUPS=FOO bash -c 'echo $GROUPS'
> > FOO
> > bash-5.0$ env GROUPS=FOO bash --posix -c 'echo $GROUPS'
> > FOO
> 
> You can't overwrite GROUPS while you're inside of bash, apparently not
> even in the temporary assignment context before a simple command.  So,
> in order for the first form to work as expected, you'd have to be in a
> shell other than bash.

Exactly. I had thought that it was partially KRE's point.

bash-5.0$ x=foo
bash-5.0$ x=bar bash -c 'echo $x'
bar
bash-5.0$ echo $x
foo

The above is a reasonable expected behavior that works with "x" but
fails with "GROUPS" and other such magic variables.

The trick of "unset GROUPS" is the only way out.

-- 
Jason Franklin