5.0: documentation for 'wait -f'

[Boruch Baum]

Subject: 5.0: missing documentation for 'wait -f'

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc

Compilation CFLAGS: -g -O2
-fdebug-prefix-map=/build/bash-RVqIBY/bash-5.0=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall
-Wno-parentheses -Wno-format-security

uname output: Linux E15-2016 4.14.0-3-amd64 #1 SMP Debian 4.14.17-1
(2018-02-14) x86_64 GNU/Linux

Machine Type: x86_64-pc-linux-gnu

Bash Version: 5.0
Patch Level: 0
Release Status: release

Description:
  The CHANGES.gz file documents the addition of an '-f' option to the
  'wait' builtin, and the feature is documented in the bash man page;
  however, it is absent from the documentation returned when using the
  'help' command, ie. 'help wait'.


Repeat-By:

  help wait

Fix:

  cut and paste text?


BTW: sneaking in another report here... The bashbug output didn't
respect the email convention of breaking lines at 72-80 characters, so I
manually auto-filled the lines for 'compilation CFLAGS' and 'uname' and
inserted blank line delimiters after each.


-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Rawiri Blundell]

> OK, this is a reasonable approach. Since /dev/urandom just generates
> random bytes, there's a lot of flexibility and we're not subject to
> any kind of backwards compatibility constraints, especially not the
> 16-bit limit. What do you think would be the best way to present that
> to a user? As a 32-bit random number? A character string you can use to
> create filenames? Some other form?

Hi Chet, et al,
I really like this suggestion.  All recent variants of /dev/urandom
are using something like ChaCha20, Yarrow or Fortuna, and older
implementations will still be reasonably secure - using something like
Yarrow, Yarrow-like or arc4.  Ole's concerns should be well covered by
this approach.

Having said that, there appear to be some gotchas, as covered here:

* 
https://stackoverflow.com/questions/2572366/how-to-use-dev-random-or-urandom-in-c
* 
http://insanecoding.blogspot.com/2014/05/a-good-idea-with-bad-usage-devurandom.html

So it might be a case of restricting the usability of this change to
newer kernels that have dedicated calls like getrandom() or
getentropy(), and having to handle detecting/selecting those?

So if this is an exercise that you're happy to entertain, and without
wanting to feature-creep too much, why not something like this?

* $RANDOM = exactly as it is (or with an improved RNG... probably a
separate discussion now)
* $RANDINT = 32 bit integer from RANDOM's RNG i.e. this is seed-able
* $OS_RANDINT = 32 bit random integer from /dev/urandom
* $OS_RANDSTR = random character string from /dev/urandom

I suggest OS_ because it's less typing than, say, SECURE_ or SYSTEM_,
vaguely conveys the source of randomness and it might have some
familiarity with other languages. Python's os.urandom comes to mind.
SYS_ might be a good alternative, too.

Certain characters such as '!' may need to be omitted from
OS_RANDSTR's output, possibly that's a worthwhile thing to keep in
mind?

As an aside, I can confirm the findings of a performance difference
between 4.4 and 5.0 when running the script provided earlier in the
discussion. At first glance it seems to be due to the switch from the
old LCG to the current MINSTD RNG, but I've compiled 5.0 with the old
LCG code and on my hardware that performs roughly the same as the
MINSTD RNG and Ole's ChaCha20 patch. So I thought it might be
something to do with how loops are handled, but testing other tasks
shows no great difference there between 4.4 and 5.0. That's as far as
I got.

Cheers

Rawiri

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Martijn Dekker]

Op 19-01-19 om 23:10 schreef Chet Ramey:
> On 1/19/19 2:45 PM, Martijn Dekker wrote:
>> Op 16-01-19 om 02:21 schreef Quentin:
>>> If you really need some quality CSPRNG values, I'd suggest adding a
>>> $SECURE_RANDOM variable that just reads from /dev/urandom.
>>
>> IMHO, this would clearly be the correct approach. I don't know of any
>> 21st century Unix or Unix-like system that doesn't have /dev/urandom. I
>> would really like to see shells adopt this idea -- hopefully all with
>> the same variable name.
> 
> OK, this is a reasonable approach. Since /dev/urandom just generates
> random bytes, there's a lot of flexibility and we're not subject to
> any kind of backwards compatibility constraints, especially not the
> 16-bit limit. What do you think would be the best way to present that
> to a user? As a 32-bit random number? A character string you can use to
> create filenames? Some other form?

I'd say numbers would be the most useful, as these are the easiest to
convert into anything else using shell arithmetic and parameter
expansions. E.g. to create a random character string for a temporary
file name, you could do

filename_suffix() {
chars=ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789
length=${#chars}
for ((i=0; i<10; i++)) do
printf '%s' "${chars:$(( SECURE_RANDOM % length + 1 )):1}"
done
}
tmpfile=/tmp/myfile.$(filename_suffix)

(which would of course already work with RANDOM but that would be
totally insecure, as in not effectively eliminating the risk of collisions).

- Martijn

bug: illegal function name?

[Andrey Butirsky]

Hello,
seems like bash allows illegal function names, but refuses to unset them
then:

|$ function 1a () { :; }||
|

|$ declare -f 1a    ||
||1a () ||
||{ ||
||    :||
||}||
|

|$ unset 1a ||
||bash: unset: `1a': not a valid identifier
$
GNU bash, version 4.3.48(1)-release (x86_64-slackware-linux-gnu)|

The first character of a name should not be a digit:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_235

Re: bug: illegal function name?

[Andreas Schwab]

On Jan 20 2019, Andrey Butirsky  wrote:

> |$ unset 1a ||
> ||bash: unset: `1a': not a valid identifier

Use `unset -f'.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: bug: illegal function name?

[Andrey Butirsky]

`1' is not a valid identifier, still:
$ function 1 () { :; }
bash: `1': not a valid identifier

So that is pretty messy..

Bug report: https://savannah.gnu.org/support/?109632


On 20.01.2019 18:26, Andrey Butirsky wrote:
> Andreas, I know it will work with the '-f' flag.
> But for others function names, the '-f' unset flag is not required.
> Moreover, it seem confronts with Open Group Base Specification.
> So I consider it as a bug still.
>
> On 20.01.2019 18:18, Andreas Schwab wrote:
>> On Jan 20 2019, Andrey Butirsky  wrote:
>>
>>> |$ unset 1a ||
>>> ||bash: unset: `1a': not a valid identifier
>> Use `unset -f'.
>>
>> Andreas.
>>

Re: bug: illegal function name?

[Andrey Butirsky]

Andreas, I know it will work with the '-f' flag.
But for others function names, the '-f' unset flag is not required.
Moreover, it seem confronts with Open Group Base Specification.
So I consider it as a bug still.

On 20.01.2019 18:18, Andreas Schwab wrote:
> On Jan 20 2019, Andrey Butirsky  wrote:
>
>> |$ unset 1a ||
>> ||bash: unset: `1a': not a valid identifier
> Use `unset -f'.
>
> Andreas.
>

Installing bash with rpath

[Mohammad Akhlaghi]

Dear Bash developers,

I am trying to build Bash for a relatively closed environment where I 
don't want the Bash executable to link with the system's libreadline and 
libncursesw, but my own installation of these libraries.


However, even when I configure bash with `--enable-rpath', and add 
`-Wl,-rpath-link=$my_install/lib' to LDFLAGS, the bash executable 
doesn't have the absolute address of these libraries in it.


As a result, the only way I can run my installation of Bash is to 
explicitly add `$my_install/lib' to LD_LIBRARY_PATH.


I use the same recipe (to build Bash) for many other low-level GNU 
programs, but I don't have this problem for any other executable (they 
all contain the absolute address of the libraries they link with).


I wanted to see if you have any recommendations to fix this problem.

Thank you very much,
Mohammad

Re: bug: illegal function name?

[Ilkka Virta]

In POSIX mode, Bash disallows names like '1a':

 13. Function names must be valid shell names. That is, they may not
 contain characters other than letters, digits, and underscores, and may
 not start with a digit. Declaring a function with an invalid name
 causes a fatal syntax error in non-interactive shells.

 14. Function names may not be the same as one of the POSIX special
 builtins.

https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html#Bash-POSIX-Mode

The rules are more lax if POSIX mode is not set, but there's nothing 
that requires using nonstandard function names even in that case.



The manual could of course mention something about the accepted function 
names, e.g.  "Function names can contain the characters [...], except in 
POSIX mode, where they must be valid shell /names/."  I'm not exactly 
sure what the accepted characters are, though, so I can't really suggest 
anything concrete.



On 20.1. 17:26, Andrey Butirsky wrote:

Andreas, I know it will work with the '-f' flag.
But for others function names, the '-f' unset flag is not required.
Moreover, it seem confronts with Open Group Base Specification.
So I consider it as a bug still.

On 20.01.2019 18:18, Andreas Schwab wrote:

On Jan 20 2019, Andrey Butirsky  wrote:


|$ unset 1a ||
||bash: unset: `1a': not a valid identifier

Use `unset -f'.

Andreas.







--
Ilkka Virta / itvi...@iki.fi

Re: bug: illegal function name?

[Eduardo A . Bustamante López]

On Sun, Jan 20, 2019 at 06:26:43PM +0300, Andrey Butirsky wrote:
> Andreas, I know it will work with the '-f' flag.
> But for others function names, the '-f' unset flag is not required.
> Moreover, it seem confronts with Open Group Base Specification.
> So I consider it as a bug still.
> 

It is intentional. If you want bash to run in POSIX mode, you should use the 
`POSIXLY_CORRECT' variable or the `--posix'
flag.

| dualbus@system76-pc:~$ POSIXLY_CORRECT=y bash -c 'function 1a() { :; }'
| bash: `1a': not a valid identifier

And as Andreas mentioned, in general you should use `unset -f' if you want to 
operate on functions. Otherwise you risk
conflict with a variable name.

Re: bug: illegal function name?

[Andrey Butirsky]

I'm not quite understand what exactly is "intentional".
The problem is inconsistent behavior of unset '-f' flag for "normal" and
"not-normal" function names (I'm not considering conflicting with
variable names case).
This is just confusing and makes the scripts error-prone, IMO.

On 20.01.2019 21:26, Eduardo A. Bustamante López wrote:
> On Sun, Jan 20, 2019 at 06:26:43PM +0300, Andrey Butirsky wrote:
>> Andreas, I know it will work with the '-f' flag.
>> But for others function names, the '-f' unset flag is not required.
>> Moreover, it seem confronts with Open Group Base Specification.
>> So I consider it as a bug still.
>>
> It is intentional. If you want bash to run in POSIX mode, you should use the 
> `POSIXLY_CORRECT' variable or the `--posix'
> flag.
>
> | dualbus@system76-pc:~$ POSIXLY_CORRECT=y bash -c 'function 1a() { :; }'
> | bash: `1a': not a valid identifier
>
> And as Andreas mentioned, in general you should use `unset -f' if you want to 
> operate on functions. Otherwise you risk
> conflict with a variable name.

Re: bug: illegal function name?

[Eduardo Bustamante]

On Sun, Jan 20, 2019 at 10:46 AM Andrey Butirsky  wrote:
>
> I'm not quite understand what exactly is "intentional".

Accepting different sets of function names (when POSIX mode is either on/off).

> The problem is inconsistent behavior of unset '-f' flag for "normal" and
> "not-normal" function names (I'm not considering conflicting with
> variable names case).
> This is just confusing and makes the scripts error-prone, IMO.

How about this: "/Always/ use unset -f to unset functions". Also,
don't ignore the "variable names case", because that's an important
piece here.

Re: bug: illegal function name?

[Andrey Butirsky]

Eduardo, the question is about inconsistent, error-prone implementation.
Not about the workarounds.

On 20.01.2019 21:59, Eduardo Bustamante wrote:
>> The problem is inconsistent behavior of unset '-f' flag for "normal" and
>> "not-normal" function names (I'm not considering conflicting with
>> variable names case).
>> This is just confusing and makes the scripts error-prone, IMO.
> How about this: "/Always/ use unset -f to unset functions". Also,
> don't ignore the "variable names case", because that's an important
> piece here.

Re: bug: illegal function name?

[Eduardo Bustamante]

On Sun, Jan 20, 2019 at 11:05 AM Andrey Butirsky  wrote:
>
> Eduardo, the question is about inconsistent, error-prone implementation.
> Not about the workarounds.

What's the error prone part?

- Declare a function name
- Unset it with 'unset -f'

Re: 5.0: documentation for 'wait -f'

[Chet Ramey]

On 1/20/19 7:27 AM, Boruch Baum wrote:
> Subject: 5.0: missing documentation for 'wait -f'
> 
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc
> 
> Compilation CFLAGS: -g -O2
> -fdebug-prefix-map=/build/bash-RVqIBY/bash-5.0=.
> -fstack-protector-strong -Wformat -Werror=format-security -Wall
> -Wno-parentheses -Wno-format-security
> 
> uname output: Linux E15-2016 4.14.0-3-amd64 #1 SMP Debian 4.14.17-1
> (2018-02-14) x86_64 GNU/Linux
> 
> Machine Type: x86_64-pc-linux-gnu
> 
> Bash Version: 5.0
> Patch Level: 0
> Release Status: release
> 
> Description:
>   The CHANGES.gz file documents the addition of an '-f' option to the
>   'wait' builtin, and the feature is documented in the bash man page;
>   however, it is absent from the documentation returned when using the
>   'help' command, ie. 'help wait'.
It's there:

$ ./bash --version
GNU bash, version 5.0.0(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ ./bash -c 'help wait'
wait: wait [-fn] [id ...]
Wait for job completion and return exit status.

Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status.  If ID is not
given, waits for all currently active child processes, and the return
status is zero.  If ID is a job specification, waits for all processes
in that job's pipeline.

If the -n option is supplied, waits for the next job to terminate and
returns its exit status.

If the -f option is supplied, and job control is enabled, waits for the
specified ID to terminate, instead of waiting for it to change status.

Exit Status:
Returns the status of the last ID; fails if ID is invalid or an invalid
option is given.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Installing bash with rpath

[Chet Ramey]

On 1/20/19 12:03 PM, Mohammad Akhlaghi wrote:
> Dear Bash developers,
> 
> I am trying to build Bash for a relatively closed environment where I don't
> want the Bash executable to link with the system's libreadline and
> libncursesw, but my own installation of these libraries.
> 
> However, even when I configure bash with `--enable-rpath', and add
> `-Wl,-rpath-link=$my_install/lib' to LDFLAGS, the bash executable doesn't
> have the absolute address of these libraries in it.

Bash doesn't use --enable-rpath, and it's not in the list of options
configure supports. Have you tried --with-installed-readline=path?
And told the linker to use a specific directory for libraries using
LDFLAGS when running the make?

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bug: illegal function name?

[Chet Ramey]

On 1/20/19 10:46 AM, Andrey Butirsky wrote:
> `1' is not a valid identifier, still:
> $ function 1 () { :; }
> bash: `1': not a valid identifier
> 
> So that is pretty messy..

In posix mode, bash restricts function names to be valid shell identifiers,
as the standard specifies.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bug: illegal function name?

[Chet Ramey]

On 1/20/19 10:26 AM, Andrey Butirsky wrote:
> Andreas, I know it will work with the '-f' flag.
> But for others function names, the '-f' unset flag is not required.
> Moreover, it seem confronts with Open Group Base Specification.

When posix mode is enabled, bash restricts function names to valid shell
identifiers. In default mode, it allows virtually any character except
NUL. (So you can create a function whose name contains a slash, but you'll
never be able to call it.)

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bug: illegal function name?

[Chet Ramey]

On 1/20/19 1:22 PM, Ilkka Virta wrote:

> The manual could of course mention something about the accepted function
> names, e.g.  "Function names can contain the characters [...], except in
> POSIX mode, where they must be valid shell /names/."  I'm not exactly sure
> what the accepted characters are, though, so I can't really suggest
> anything concrete.

The default mode accepts just about anything except NUL, so it's better to
mention the restrictions that accompany posix mode.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bug: illegal function name?

[Andrey Butirsky]

Chet, the sample was run in normal, not POSIX, mode.
So it allows '1a' but disallows '1'. Pretty odd.

On 20.01.2019 23:55, Chet Ramey wrote:
> On 1/20/19 10:46 AM, Andrey Butirsky wrote:
>> `1' is not a valid identifier, still:
>> $ function 1 () { :; }
>> bash: `1': not a valid identifier
>>
>> So that is pretty messy..
> In posix mode, bash restricts function names to be valid shell identifiers,
> as the standard specifies.
>

Re: bug: illegal function name?

[Chet Ramey]

On 1/20/19 1:46 PM, Andrey Butirsky wrote:
> I'm not quite understand what exactly is "intentional".
> The problem is inconsistent behavior of unset '-f' flag for "normal" and
> "not-normal" function names (I'm not considering conflicting with
> variable names case).

You should not ignore the variable names case. The behavior of unset
without arguments is to check for a variable first, then optionally
check for a function name. The variable name check enforces the
restrictions on valid characters that may appear in variable names. The
only guaranteed way to make a POSIX shell unset a function name is to
use `unset -f'.

"If neither -f nor -v is specified, name refers to a variable; if a
variable by that name does not exist, it is unspecified whether a function
by that name, if any, shall be unset."

Where is the inconsistent behavior for `unset -f'? It works whatever
the name of the function is when in default mode, and obeys the POSIX
restrictions when in posix mode.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bug: illegal function name?

[Chet Ramey]

On 1/20/19 4:03 PM, Andrey Butirsky wrote:
> Chet, the sample was run in normal, not POSIX, mode.
> So it allows '1a' but disallows '1'. Pretty odd.

Yeah, it should not reject all digits when not in posix mode. I'll take
a look there.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bug: illegal function name?

[Andrey Butirsky]

On 21.01.2019 00:02, Chet Ramey wrote:
> On 1/20/19 1:46 PM, Andrey Butirsky wrote:
>> I'm not quite understand what exactly is "intentional".
>> The problem is inconsistent behavior of unset '-f' flag for "normal" and
>> "not-normal" function names (I'm not considering conflicting with
>> variable names case).
> You should not ignore the variable names case. The behavior of unset
> without arguments is to check for a variable first, then optionally
> check for a function name. The variable name check enforces the
> restrictions on valid characters that may appear in variable names. The
> only guaranteed way to make a POSIX shell unset a function name is to
> use `unset -f'.
>
> "If neither -f nor -v is specified, name refers to a variable; if a
> variable by that name does not exist, it is unspecified whether a function
> by that name, if any, shall be unset."
>
> Where is the inconsistent behavior for `unset -f'? It works whatever
> the name of the function is when in default mode, and obeys the POSIX
> restrictions when in posix mode.
>
I meant rather `unset` without '-f'.
But if what you quoted is from the standard, seems it allows
implementation-specific behavior in that case. Thanks.

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Chet Ramey]

On 1/20/19 7:52 AM, Rawiri Blundell wrote:

> So it might be a case of restricting the usability of this change to
> newer kernels that have dedicated calls like getrandom() or
> getentropy(), and having to handle detecting/selecting those?
> 
> So if this is an exercise that you're happy to entertain, and without
> wanting to feature-creep too much, why not something like this?

I'd probably start with URANDOM as a 32-bit random integer read as
four bytes from /dev/urandom. It's trivial to create a filename from
that with whatever restrictions (and whatever characters) you want.

> As an aside, I can confirm the findings of a performance difference
> between 4.4 and 5.0 when running the script provided earlier in the
> discussion. At first glance it seems to be due to the switch from the
> old LCG to the current MINSTD RNG, 

There's no switch: the bash-4.4 generator and bash-5.0 generators are
identical. I'll have to do some profiling when I get a chance.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Chet Ramey]

On 1/20/19 4:54 PM, Chet Ramey wrote:

>> As an aside, I can confirm the findings of a performance difference
>> between 4.4 and 5.0 when running the script provided earlier in the
>> discussion. At first glance it seems to be due to the switch from the
>> old LCG to the current MINSTD RNG, 
> 
> There's no switch: the bash-4.4 generator and bash-5.0 generators are
> identical. I'll have to do some profiling when I get a chance.

So I ran a quick test.

$ ./bash ./x3
iterations: 100
BASH_VERSION: 5.0.2(4)-maint
time: 9.684
$ ../bash-5.0/bash ./x3
iterations: 100
BASH_VERSION: 5.0.0(1)-release
time: 9.749
$ ../bash-5.0-patched/bash ./x3
iterations: 100
BASH_VERSION: 5.0.2(3)-release
time: 9.840
$ ../bash-4.4-patched/bash ./x3
iterations: 100
BASH_VERSION: 4.4.23(7)-release
time: 11.365
$ ../bash-4.4-patched/bash ./x3
iterations: 100
BASH_VERSION: 4.4.23(7)-release
time: 11.235
jenna.local(1)

Where the script is Eduardo's iterator that just expands $RANDOM
N times.

The random number generator has been the same since bash-4.0.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Installing bash with rpath

[Mohammad Akhlaghi]

Thank you very much for the prompt reply,

On 1/20/19 7:54 PM, Chet Ramey wrote:

On 1/20/19 12:03 PM, Mohammad Akhlaghi wrote:

Dear Bash developers,

I am trying to build Bash for a relatively closed environment where I don't
want the Bash executable to link with the system's libreadline and
libncursesw, but my own installation of these libraries.

However, even when I configure bash with `--enable-rpath', and add
`-Wl,-rpath-link=$my_install/lib' to LDFLAGS, the bash executable doesn't
have the absolute address of these libraries in it.


Bash doesn't use --enable-rpath, and it's not in the list of options
configure supports. Have you tried --with-installed-readline=path?
And told the linker to use a specific directory for libraries using
LDFLAGS when running the make?


You are right about `--enable-rpath'. Since `--disable-rpath' existed, I 
assumed an `--enable-rpath' must also be present. Most programs only 
show one of the `--enable-FEATURE' or `--disable-FEATURE'. So, because 
my final bash executable didn't have any RPATH, I thought that 
`--enable-rpath' might also be present, but as you mentioned, it had no 
effect.


Yes, I configure Bash using the `--with-installed-readline' and set 
LDFLAGS. It builds and installs successfully and there is no problem in 
that part.


The problem is in running bash. I have installed Bash 5.0 and the 
respective libreadline and libncursesw in a non-standard directory (I 
don't have root access on the system). So, it will crash (because of not 
finding libreadline) unless I set LD_LIBRARY_PATH.


I am trying to find a way to embed RPATH in the bash executable so I 
don't have to set LD_LIBRARY_PATH before calling bash (and thus 
inheriting this directory to everything that I call under bash).


I would be grateful you could guide me on how I can configure Bash to 
have RPATH in its executable.


Thank you very much,
Mohammad

Re: Installing bash with rpath

[Chet Ramey]

On 1/20/19 6:28 PM, Mohammad Akhlaghi wrote:

> Yes, I configure Bash using the `--with-installed-readline' and set
> LDFLAGS. It builds and installs successfully and there is no problem in
> that part.
> 
> The problem is in running bash. I have installed Bash 5.0 and the
> respective libreadline and libncursesw in a non-standard directory (I don't
> have root access on the system). So, it will crash (because of not finding
> libreadline) unless I set LD_LIBRARY_PATH.
> 
> I am trying to find a way to embed RPATH in the bash executable so I don't
> have to set LD_LIBRARY_PATH before calling bash (and thus inheriting this
> directory to everything that I call under bash).

Isn't there a linker option you can supply, possibly as part of LDFLAGS,
to embed that into the bash binary?

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Eduardo A . Bustamante López]

On Sun, Jan 20, 2019 at 05:22:12PM -0500, Chet Ramey wrote:
> On 1/20/19 4:54 PM, Chet Ramey wrote:
> 
> >> As an aside, I can confirm the findings of a performance difference
> >> between 4.4 and 5.0 when running the script provided earlier in the
> >> discussion. At first glance it seems to be due to the switch from the
> >> old LCG to the current MINSTD RNG, 
(...)
> So I ran a quick test.
> 
> $ ./bash ./x3
> iterations: 100
> BASH_VERSION: 5.0.2(4)-maint
> time: 9.684
> $ ../bash-5.0/bash ./x3
> iterations: 100
> BASH_VERSION: 5.0.0(1)-release
> time: 9.749
> $ ../bash-5.0-patched/bash ./x3
> iterations: 100
> BASH_VERSION: 5.0.2(3)-release
> time: 9.840
> $ ../bash-4.4-patched/bash ./x3
> iterations: 100
> BASH_VERSION: 4.4.23(7)-release
> time: 11.365
> $ ../bash-4.4-patched/bash ./x3
> iterations: 100
> BASH_VERSION: 4.4.23(7)-release
> time: 11.235
> jenna.local(1)
> 
> Where the script is Eduardo's iterator that just expands $RANDOM
> N times.
> 
> The random number generator has been the same since bash-4.0.

I'm sorry, my tests were wrong. I built bash using the default `./configure'
behavior for the `devel' branch, which I always forget, uses the internal
allocator with debugging enabled, and thus, all of my times were off due to the
additional malloc overhead.

I rebuilt it with `../bash/configure --without-bash-malloc', which causes it to
use the system's allocator, and surely enough, the timings make more sense now:

(`build-bash-devel-malloc' is `configure --with-bash-malloc',
 `build-bash-devel' is `configure --without-bash-malloc')

| dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ ./bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 5.0.0(1)-maint
| time: 8.765
| 
| dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ 
../build-bash-devel/bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 5.0.0(1)-maint
| time: 3.431
| 
| dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 5.0.0(1)-release
| time: 3.435
| 
| dualbus@system76-pc:~/src/gnu/build-bash-4.4$ ./bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 4.4.0(1)-release
| time: 3.443

Re: Installing bash with rpath

[Mohammad Akhlaghi]

On 1/21/19 12:25 AM, Chet Ramey wrote:

Isn't there a linker option you can supply, possibly as part of LDFLAGS,
to embed that into the bash binary?


I pass `-Wl,-rpath-link=$instdir/lib' to LDFLAGS. It sets RPATH properly 
on all the programs I install (including libreadline, and many other 
basic programs), except for Bash and AWK.


In my built programs, Bash and AWK are the only programs that depend on 
libreadline, but RPATH does get written into libreadline, so I don't 
know if the linking with libreadline has any affect on this problem or not.


Is there any step in the build or install of Bash that somehow disables 
using this method of setting RPATH?


So far I found this work-around: to manually write RPATH into the Bash 
executable after installation using patchelf: 
https://nixos.org/patchelf.html . But it would be much better/elegant if 
this manual intervention wouldn't be necessary.


Thanks a lot,
Mohammad

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Rawiri Blundell]

On Mon, Jan 21, 2019 at 1:36 PM Eduardo A. Bustamante López
 wrote:
>
> On Sun, Jan 20, 2019 at 05:22:12PM -0500, Chet Ramey wrote:
> > On 1/20/19 4:54 PM, Chet Ramey wrote:
> >
> > >> As an aside, I can confirm the findings of a performance difference
> > >> between 4.4 and 5.0 when running the script provided earlier in the
> > >> discussion. At first glance it seems to be due to the switch from the
> > >> old LCG to the current MINSTD RNG,
> (...)
> > So I ran a quick test.
> >
> > $ ./bash ./x3
> > iterations: 100
> > BASH_VERSION: 5.0.2(4)-maint
> > time: 9.684
> > $ ../bash-5.0/bash ./x3
> > iterations: 100
> > BASH_VERSION: 5.0.0(1)-release
> > time: 9.749
> > $ ../bash-5.0-patched/bash ./x3
> > iterations: 100
> > BASH_VERSION: 5.0.2(3)-release
> > time: 9.840
> > $ ../bash-4.4-patched/bash ./x3
> > iterations: 100
> > BASH_VERSION: 4.4.23(7)-release
> > time: 11.365
> > $ ../bash-4.4-patched/bash ./x3
> > iterations: 100
> > BASH_VERSION: 4.4.23(7)-release
> > time: 11.235
> > jenna.local(1)
> >
> > Where the script is Eduardo's iterator that just expands $RANDOM
> > N times.
> >
> > The random number generator has been the same since bash-4.0.
>
> I'm sorry, my tests were wrong. I built bash using the default `./configure'
> behavior for the `devel' branch, which I always forget, uses the internal
> allocator with debugging enabled, and thus, all of my times were off due to 
> the
> additional malloc overhead.
>
> I rebuilt it with `../bash/configure --without-bash-malloc', which causes it 
> to
> use the system's allocator, and surely enough, the timings make more sense 
> now:
>
> (`build-bash-devel-malloc' is `configure --with-bash-malloc',
>  `build-bash-devel' is `configure --without-bash-malloc')
>
> | dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ ./bash 
> ~/test-speed.sh
> | iterations: 100
> | BASH_VERSION: 5.0.0(1)-maint
> | time: 8.765
> |
> | dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ 
> ../build-bash-devel/bash ~/test-speed.sh
> | iterations: 100
> | BASH_VERSION: 5.0.0(1)-maint
> | time: 3.431
> |
> | dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ bash ~/test-speed.sh
> | iterations: 100
> | BASH_VERSION: 5.0.0(1)-release
> | time: 3.435
> |
> | dualbus@system76-pc:~/src/gnu/build-bash-4.4$ ./bash ~/test-speed.sh
> | iterations: 100
> | BASH_VERSION: 4.4.0(1)-release
> | time: 3.443

Hi,
Perfect timing - I was just about to reply to Chet's earlier message
with the following finding:

*snip*

On Mon, Jan 21, 2019 at 11:22 AM Chet Ramey  wrote:
>
> On 1/20/19 4:54 PM, Chet Ramey wrote:
>
> So I ran a quick test.
>
> [results removed for brevity]

That's interesting.  For comparison here's what I'm seeing:

▓▒░$ ~/bin/bash5/bash /tmp/randtest   # Downloaded and compiled as-is
iterations: 100
BASH_VERSION: 5.0.0(1)-release
time: 7.210
▓▒░$ ~/bin/bash5lcg/bash /tmp/randtest # With all the RANDOM related
code reverted to the 4.4 code
iterations: 100
BASH_VERSION: 5.0.0(2)-release
time: 7.271
▓▒░$ ~/bin/bash5chacha/bash /tmp/randtest # With Ole's chacha patch
iterations: 100
BASH_VERSION: 5.0.0(1)-release
time: 7.443
▓▒░$ /bin/bash /tmp/randtest # Distro provided package, wait a minute...
iterations: 100
BASH_VERSION: 4.4.19(1)-release
time: 5.610

I hadn't thought of the pre-compiled package vs source compiled
difference until just now.  So I figured that while I'd downloaded
4.4's source to double check variables.c, I may as well compile and
test it:

▓▒░$ ~/bin/bash44/bash /tmp/randtest
iterations: 100
BASH_VERSION: 4.4.0(1)-release
time: 7.432

So in the best tradition of n=1 science, it looks like the difference
may be down to compilation choices?  Can anyone else replicate this?

*/snip*

So it looks like problem solved?

Rawiri

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Chet Ramey]

On 1/20/19 8:07 PM, Rawiri Blundell wrote:

> */snip*
> 
> So it looks like problem solved?

There never was a problem.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: "return" should not continue script execution, even if used inappropriately

[don fong]

>
> Then use the "  sh -c '. script'  " version instead.
>

i don't see how this helps.  the point is to have one file of code that
behaves differently depending on whether it's dotted in or executed at the
top level.  the script should do nothing but define stuff when dotted in;
but call the main function (or some other function) when run at the top
level. "sh -c '. script'" doesn't accomplish that.

On Thu, Jan 10, 2019 at 2:53 PM Robert Elz  wrote:

> Date:Tue, 8 Jan 2019 23:40:40 -0800
> From:don fong 
> Message-ID:  <
> cahqakpu03jfz3smu0hcfie9rh8+kyqxv5gy7h6t-7zwf-x5...@mail.gmail.com>
>
>
>   | to me, your suggested wrapper script pattern seems unnatural.  i don't
>   | always want users to have to carry around 2 files (the dottable
> library and
>   | the wrapper to dot it in).
>
> Then use the "  sh -c '. script'  " version instead.
>
>   | speaking of breakage, i'd also note that your suggested pattern has a
> bug,
>   | it assumes that run-dotscr will only be run from the directory where
> the
>   | file lives.  yes, the bug can be easily fixed.  but fixing it will make
>   | your script a bit less "simple and natural".
>
> Not really a bug, simply a simpliciation for the purposes of this e-mail
> exchange ... obviously you're expected to be smart enough to use the
> actual form that is needed to work in the environment in which the
> script is to be tested - whether that means using a full path name, or
> a PATH search, or whatever is appropriate for the situation (that is, the
> "./dotscr" part of the example should be the same thing you'd expect the
> user to type if they were to run it as an independent script.
>
> The purpose of the example was to illustrate the echnique, not to provide
> a ready made (complete) solution.
>
> kre
>
>

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Rawiri Blundell]

On Mon, Jan 21, 2019 at 10:54 AM Chet Ramey  wrote:
>
> On 1/20/19 7:52 AM, Rawiri Blundell wrote:
>
> > So it might be a case of restricting the usability of this change to
> > newer kernels that have dedicated calls like getrandom() or
> > getentropy(), and having to handle detecting/selecting those?
> >
> > So if this is an exercise that you're happy to entertain, and without
> > wanting to feature-creep too much, why not something like this?
>
> I'd probably start with URANDOM as a 32-bit random integer read as
> four bytes from /dev/urandom. It's trivial to create a filename from
> that with whatever restrictions (and whatever characters) you want.
>

For what it's worth I did consider suggesting URANDOM, however I
figured some users may confuse it like this:

RANDOM -> /dev/random
URANDOM -> /dev/urandom

Couple that with an established base of myths about /dev/urandom, I
thought it might be best to suggest something else to eliminate that
potential confusion.

(SRANDOM was another one I considered, has a bit of awk familiarity to it...)

> > As an aside, I can confirm the findings of a performance difference
> > between 4.4 and 5.0 when running the script provided earlier in the
> > discussion. At first glance it seems to be due to the switch from the
> > old LCG to the current MINSTD RNG,
>
> There's no switch: the bash-4.4 generator and bash-5.0 generators are
> identical. I'll have to do some profiling when I get a chance.
>

I suspect that we're talking at cross purposes, but it's now neither
here nor there.

You've expressed that RANDOM's period and seeding are issues for you.
I think the ChaCha20 patch is a bit overkill for RANDOM's
requirements, but would you be interested in some investigation into
middle-ground alternatives like PCG or JSF32?

Rawiri

Re: bug: illegal function name?

[Robert Elz]

Date:Sun, 20 Jan 2019 16:02:51 -0500
From:Chet Ramey 
Message-ID:  

  | You should not ignore the variable names case. The behavior of unset
  | without arguments is to check for a variable first, then optionally
  | check for a function name.

That's fine.

  | The variable name check enforces the
  | restrictions on valid characters that may appear in variable names.

That's fine too, but there is no need to generate an error if the
name cannot be a variable name.   Nothing requires that, posix
just says that *if* the name refers to a variable, it gets unset, there
is no specific syntax (note that in the standard for unset, the "name"
there is a placeholder for a word, not the defined "name" word, which
would be written in a different font).  If the name cannot be a variable,
it also cannot be a variable that is set, and if you are going to allow
unset to unset functions in that case, probably you should always do so.

  | The only guaranteed way to make a POSIX shell unset a function name is to
  | use `unset -f'.

That's true.

  | "If neither -f nor -v is specified, name refers to a variable; if a
  | variable by that name does not exist, it is unspecified whether a function
  | by that name, if any, shall be unset."

And that's fine.   Nothing is questioning what the application really
should do, just what happens when the application (or more likely,
some user at the terminal in this case) does something different.

It is certainly not a standards issue.

  | Where is the inconsistent behavior for `unset -f'? It works whatever
  | the name of the function is when in default mode, and obeys the POSIX
  | restrictions when in posix mode.

I think his point is that if unset "unset f" (no flags) works to unset 
function f, if f is not a (set) variable, then it should work every time
"f" is not a set variable, not only the times when the word "f" happens
to be of the correct syntax to be a variable name, but happens not
to be.

I think that is a good point.

In the NetBSD sh we avoid that issue completely, the only way to
unset a function is with "unset -f", a simple "unset f" only ever
unsets variables.

kre

Re: bug: illegal function name?

[Eduardo A . Bustamante López]

On Mon, Jan 21, 2019 at 09:26:59AM +0700, Robert Elz wrote:
(...)
> I think his point is that if unset "unset f" (no flags) works to unset 
> function f, if f is not a (set) variable, then it should work every time
> "f" is not a set variable, not only the times when the word "f" happens
> to be of the correct syntax to be a variable name, but happens not
> to be.
> 
> I think that is a good point.
> 
> In the NetBSD sh we avoid that issue completely, the only way to
> unset a function is with "unset -f", a simple "unset f" only ever
> unsets variables.

I think that's a good point too. But how would you fix this particular issue?

Changing the behavior of `unset f' to only ever unset variables means
potentially breaking existing scripts. Is the inconsistency reported severe
enough to make this change?

Re: bug: illegal function name?

[Robert Elz]

Date:Sun, 20 Jan 2019 19:50:35 -0800
From:Eduardo =?iso-8859-1?Q?A=2E_Bustamante_L=F3pez?= 

Message-ID:  <20190121035035.gc31...@system76-pc.vc.shawcable.net>

  | Changing the behavior of `unset f' to only ever unset variables means
  | potentially breaking existing scripts. Is the inconsistency reported severe
  | enough to make this change?

No, of course not, and no-one is suggesting that.   Rather make the
algorithm for "unset" when no flags are given be something like the
following (written in sh rather than C, as that's what most people on
this list should be able to follow easily) ...

for name
do
if  case "${name}" in
( [!A-Za-z_]* | *[!A-Za-z0-9_]*) false;;
(*) true;;
esac &&
eval test "''\${${name}+set}" = set
then
unset -v "${name}"
else
unset -f "${name}"
fi
done

The actual implementation would most likely be nothing like that of course.

kre

Re: "return" should not continue script execution, even if used inappropriately

[Robert Elz]

Date:Sun, 20 Jan 2019 17:43:04 -0800
From:don fong 
Message-ID:  


  | i don't see how this helps.  the point is to have one file of code that
  | behaves differently depending on whether it's dotted in or executed at the
  | top level.

That's fine, if you are writing something to work like this, you
just make sure that it will work when run either way.  That means
rthat you cannot do a "return" outside of a function in the script.

This is easy to accomplish, the code just needs to be written
to meet both sets of requirements (if you want to finish in a reliable
way, you execute to EOF, and don't use either exit or return).

  | the script should do nothing but define stuff when dotted in;
  | but call the main function (or some other function) when run at the top
  | level. "sh -c '. script'" doesn't accomplish that.

No, but it allows you to test a script that is not written to be able
to be run either way, someone's script that is only intended to be
executed as ". script" which you then want to test.In many cases
you might need to add more than just the '.' command - depending
on what is in the script (as I said way back in my first message on
this subject).

kre

Re: bug: illegal function name?

[pepa65]

On 20/1/2019 19:50, Eduardo A. Bustamante López wrote:
> Changing the behavior of `unset f' to only ever unset variables means
> potentially breaking existing scripts. Is the inconsistency reported severe
> enough to make this change?

The alternative would be to allow anything (that is not a proper
variable name) after unset, and if it can't be a variable name, only the
functions need to be checked and unset if they exist.

Peter