Re: Bug in [ -d ... ] ?

[Greg Wooledge]

On Fri, Nov 03, 2017 at 10:29:13AM +0700, Robert Elz wrote:
>   if [ -d "${FOOFOOFOO:=""}" ]; ...
> 
> is not really portable.   In bash, and several other shells, it does what
> it looks like it should do, but in posix (as it stands today) and in some
> other shells, including the original Bourne shell, that is parsed as
> 
>   if [ -d  "${FOOFOOFOO:=""}" ]; ...
>  ^=v^=v

I am not seeing any evidence of this.

wooledg:~$ bash
wooledg:~$ /home/wooledg/bin/args "${FOO:=""}"
1 args: <>

wooledg:~$ sh
$ /home/wooledg/bin/args "${FOO:=""}"
1 args: <>

$ ssh root@imadev
... (c)Copyright 1983-1996 Hewlett-Packard Co.,  All Rights Reserved. ...
# exec ksh
# /home/wooledg/bin/args "${FOO:=""}"
1 args: <>

# /usr/old/bin/sh
# /home/wooledg/bin/args "${FOO:=""}"
1 args: <>

That's bash 4.4, dash 0.5.8, ksh88 on HP-UX 10.20, and a Bourne shell on
HP-UX 10.20, respectively.

Re: New MILLISECONDS “special” variable?

[Alan Dipert]

Thanks all, EPOCHREALTIME sounds perfect. I’m also glad to have learned
about loadable modules. Really cool.

Alan

On Thu, Nov 2, 2017 at 5:43 PM, Chet Ramey  wrote:

> On 11/2/17 4:09 PM, DJ Mills wrote:
> >
> >
> > On Thu, Nov 2, 2017 at 3:35 PM, Chet Ramey  > > wrote:
> >
> > There is a special builtin in the devel branch version
> (EPOCHREALTIME) that
> > gives you the epoch time with microsecond resolution. It will be in
> the
> > next release.
> >
> >
> > Out of curiosity, is that functionality going to extend to printf
> '%(fmt)T'
> > ? A %N or
> > so would be lovely
>
> It depends on what strftime(3) provides; the T format just passes what's
> between the parens to strftime().
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~
> chet/
>

Re: New MILLISECONDS “special” variable?

[Chet Ramey]

On 11/3/17 11:13 AM, Alan Dipert wrote:
> Thanks all, EPOCHREALTIME sounds perfect. I’m also glad to have learned
> about loadable modules. Really cool.

Just to be clear: it's a special variable, not a builtin.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

UID|EUID variables lose their ro properties

[mike b]

Hi!

Below is some odd stuff I recently noticed, looks to me as a bug, though
it's better if smarter people than me look at it as well to confirm that.
:)

Configuration Information:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu'
-DCONF_VENDOR='unknown' -DLOCALEDIR='/root/git_repos/bash/out/share/locale'
-DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib
-g -O2 -Wno-parentheses -Wno-format-security
uname output: Linux test 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2
(2016-10-19) x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.4
Patch Level: 12
Release Status: release

Description:
Whenever UID|EUID are exported to the env of newly exec Bash
instance said variables lose their ro properties. Here's example:

#!/bin/bash

if (( ! UID )) \
&& (( ! EUID )) ; then
printf 'I am root (%u)\n' "$UID"
# at this point, you can change UID|EUID to whatever
# you like
UID=BANANA
EUID=PEANUTS
printf 'UID=%s\nEUID=%s\n' "$UID" "$EUID"
exit
else
printf 'I am %u, not root\n' "$UID"
fi
export UID EUID
exec bash -c "UID=0 EUID=0 $0"

Here's example of an output:
$ ./test
I am 1000, not root
I am root (0)
UID=BANANA
EUID=PEANUTS
To my understanding, those variables should not be changed at any
point, just like e.g. PPID (which is not affected by the above).
Regards,
Michal

Re: Bug in [ -d ... ] ?

[Robert Elz]

Date:Fri, 3 Nov 2017 08:21:41 -0400
From:Greg Wooledge 
Message-ID:  <20171103122141.tvbmv5sfkcs7h...@eeg.ccf.org>

  | I am not seeing any evidence of this.

As I said, for that particular usage, the effect is the same, either way.
The way the shell gets there differs, but the result happens not to.

To see some difference, run

for i in ${x-"a ${x-"b c"}" d}; do echo "z${i}z"; done

If you get (when x is unset of course)

za bz
zcz
zdz

then the shell is doing it the (current) posix way.   Bash (and several
other shells) produce (because the 'b c' is being treated as quoted, when
it should not be with current posix rules - until the change is published
and it becomes unspecified.)

za b cz
zdz

and it is all because of the "what gets quoted, and where" issue (in many
examples, the "assign and then expand" behaviour often masks the differences,
so the '=' operator isn't the best way to see it.)

kre

Re: UID|EUID variables lose their ro properties

[Chet Ramey]

On 11/3/17 12:09 PM, mike b wrote:

> Bash Version: 4.4
> Patch Level: 12
> Release Status: release
> 
> Description:
> Whenever UID|EUID are exported to the env of newly exec Bash
> instance said variables lose their ro properties. Here's example:

Two reasons: you can't pass variable attributes through the environment;
and bash doesn't overwrite EUID or UID if a shell instance inherits them
via the environment. This is the general case; PPID is a rare exception.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Uncomplete multiline PS1 after 'display-shell-version'

[Arnaud Gaillard]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: cygwin
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash.exe' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='cygwin' -DCONF_MACHTYPE='x86_64-unknown-cygwin'
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/locale'
-DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DRECYCLES_PIDS   -I.
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4/include
-I/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4/lib  -DWORDEXP_OPTION
-ggdb -O2 -pipe -Wimplicit-function-declaration
-fdebug-prefix-map=/usr/src/bash-4.4.12-3.x86_64/build=/usr/src/debug/bash-4.4.12-3
-fdebug-prefix-map=/usr/src/bash-4.4.12-3.x86_64/src/bash-4.4=/usr/src/debug/bash-4.4.12-3
-Wno-parentheses -Wno-format-security
uname output: CYGWIN_NT-6.1 MD1MB4SC 2.8.0(0.309/5/3) 2017-04-01 20:47
x86_64 Cygwin
Machine Type: x86_64-unknown-cygwin

Bash Version: 4.4
Patch Level: 12
Release Status: release

Description:
When using a multi-line PS1, pressing C-x C-v (invoking
*display-shell-version*) only redraws the last ("most bottom") line of the
PROMPT.


Repeat-By:
~$ PS1='a\nb\nc\nd$ '
a
b
c
d$  
GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)
d$


This was also tested on 4.3 and the bug occurred as well.

Best regards,

-- 
Arnaud Gaillard
El Magnifico

Memory leak in subshell + read before bash 4.4

[Adrien Mahieux]

Hello,

I think I've found a bug in the loop management. Maybe it's an
expected behavior (didn't find any related topic on the manpages) but
it's annoying in long-running scripts.

The leak is triggered by this idiom (wether or not jobcontrol + lastpipe) :
cmd | read var

but not by this one :
read var < <(cmd)

Tested with :
- bash-4.2.46-19.el7.x86_64
- bash-4.3.42-5.fc23.x86_64
- bash-4.3-48.x86_64

>From the version tested, it was fixed in version 4.4 (tested 4.4-beta)
Do you have an idea of the commit that fixed this behavior, so the fix
can be backported to older versions ?



Here's some code to check this behavior:
#!/bin/bash

# use strict
set -u
# Also fails with no monitor + lastpipe
#set +m
#shopt -s lastpipe

typeset -i i=0
typeset -i max=1

function getrss { grep VmRSS /proc/$$/status; }
function getdata { echo 1 2 3 4  }

typeset rss_start="$(getrss)"
echo "Before loop: RSS = $rss_start"

while [[ $i -lt $max ]]; do

typeset rss_curr="$(getrss)"
[[ "$rss_start" != "$rss_curr" ]] && {
echo "New mem usage at loop $i: $rss_start => $rss_curr"
rss_start=$rss_curr
}

# First half, let's try the working method
if [[ $i -lt $(( $max/2 )) ]]; then
# That bashism is fine
read some data < <(getdata)
else
# That one leaks something
getdata | read some data
fi

i=i+1
done



and here's the output :

$ bash-4.3.42-5.fc23.x86_64 ~/bin/leak.sh
Before loop: RSS = VmRSS:2704 kB
New mem usage at loop 1: VmRSS:2704 kB => VmRSS:2984 kB
New mem usage at loop 2: VmRSS:2984 kB => VmRSS:2988 kB
New mem usage at loop 5069: VmRSS:2988 kB => VmRSS:2992 kB
New mem usage at loop 5198: VmRSS:2992 kB => VmRSS:2996 kB
New mem usage at loop 5318: VmRSS:2996 kB => VmRSS:3000 kB
New mem usage at loop 5447: VmRSS:3000 kB => VmRSS:3004 kB
New mem usage at loop 5574: VmRSS:3004 kB => VmRSS:3008 kB
New mem usage at loop 5703: VmRSS:3008 kB => VmRSS:3012 kB
New mem usage at loop 5831: VmRSS:3012 kB => VmRSS:3016 kB
New mem usage at loop 5959: VmRSS:3016 kB => VmRSS:3020 kB
New mem usage at loop 6087: VmRSS:3020 kB => VmRSS:3024 kB
New mem usage at loop 6215: VmRSS:3024 kB => VmRSS:3028 kB
New mem usage at loop 6343: VmRSS:3028 kB => VmRSS:3032 kB
New mem usage at loop 6471: VmRSS:3032 kB => VmRSS:3036 kB
New mem usage at loop 6599: VmRSS:3036 kB => VmRSS:3040 kB
New mem usage at loop 6727: VmRSS:3040 kB => VmRSS:3044 kB
New mem usage at loop 6855: VmRSS:3044 kB => VmRSS:3048 kB
New mem usage at loop 6982: VmRSS:3048 kB => VmRSS:3052 kB
New mem usage at loop 7111: VmRSS:3052 kB => VmRSS:3056 kB
New mem usage at loop 7246: VmRSS:3056 kB => VmRSS:3060 kB
New mem usage at loop 7361: VmRSS:3060 kB => VmRSS:3064 kB
New mem usage at loop 7497: VmRSS:3064 kB => VmRSS:3068 kB
New mem usage at loop 7627: VmRSS:3068 kB => VmRSS:3072 kB
New mem usage at loop 7757: VmRSS:3072 kB => VmRSS:3076 kB
New mem usage at loop 7883: VmRSS:3076 kB => VmRSS:3080 kB
New mem usage at loop 8010: VmRSS:3080 kB => VmRSS:3084 kB
New mem usage at loop 8139: VmRSS:3084 kB => VmRSS:3088 kB
New mem usage at loop 8268: VmRSS:3088 kB => VmRSS:3092 kB
New mem usage at loop 8393: VmRSS:3092 kB => VmRSS:3096 kB
New mem usage at loop 8520: VmRSS:3096 kB => VmRSS:3100 kB
New mem usage at loop 8648: VmRSS:3100 kB => VmRSS:3104 kB
New mem usage at loop 8776: VmRSS:3104 kB => VmRSS:3108 kB
New mem usage at loop 8904: VmRSS:3108 kB => VmRSS:3112 kB
New mem usage at loop 9033: VmRSS:3112 kB => VmRSS:3116 kB
New mem usage at loop 9161: VmRSS:3116 kB => VmRSS:3120 kB
New mem usage at loop 9289: VmRSS:3120 kB => VmRSS:3124 kB
New mem usage at loop 9411: VmRSS:3124 kB => VmRSS:3128 kB
New mem usage at loop 9539: VmRSS:3128 kB => VmRSS:3132 kB
New mem usage at loop 9666: VmRSS:3132 kB => VmRSS:3136 kB
New mem usage at loop 9795: VmRSS:3136 kB => VmRSS:3140 kB
New mem usage at loop 9923: VmRSS:3140 kB => VmRSS:3144 kB


$./bash-4.3.48 ~/bin/leak.sh
Before loop: RSS = VmRSS:2708 kB
New mem usage at loop 0: VmRSS:2708 kB => VmRSS:3004 kB
New mem usage at loop 5007: VmRSS:3004 kB => VmRSS:3008 kB
New mem usage at loop 5135: VmRSS:3008 kB => VmRSS:3012 kB
New mem usage at loop 5263: VmRSS:3012 kB => VmRSS:3016 kB
New mem usage at loop 5391: VmRSS:3016 kB => VmRSS:3020 kB
New mem usage at loop 5519: VmRSS:3020 kB => VmRSS:3024 kB
New mem usage at loop 5647: VmRSS:3024 kB => VmRSS:3028 kB
New mem usage at loop 5775: VmRSS:3028 kB => VmRSS:3032 kB
New mem usage at loop 5903: VmRSS:3032 kB => VmRSS:3036 kB
New mem usage at loop 6031: VmRSS:3036 kB => VmRSS:3040 kB
New mem usage at loop 6159: VmRSS:3040 kB => VmRSS:3044 kB
New mem usage at loop 6287: VmRSS:3044 kB => 

building: recutils bash builtins/readrec.so; ./configure --with-bash-headers=... # fails

[bug-bash]

No response today from bug-recutils:

http://lists.gnu.org/archive/html/bug-recutils/2017-11/msg4.html

For the ./configure, I untarred the latest bash source into a subdir, and set 
--with-bash-headers=
to that pathname. 

Perhaps someone with an understanding of GNU configure/autoconf etc could take
a look and give me some hints.  Or tell me where to ask.

--
thanks!

Re: Uncomplete multiline PS1 after 'display-shell-version'

[Chet Ramey]

On 11/3/17 2:30 PM, Arnaud Gaillard wrote:

> Bash Version: 4.4
> Patch Level: 12
> Release Status: release
> 
> Description:
> When using a multi-line PS1, pressing C-x C-v (invoking
> *display-shell-version*) only redraws the last ("most bottom") line of the
> PROMPT.
> 
> 
> Repeat-By:
> ~$ PS1='a\nb\nc\nd$ '
> a
> b
> c
> d$  
> GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)
> d$

This is intentional and the desired behavior.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Memory leak in subshell + read before bash 4.4

[Chet Ramey]

On 11/3/17 6:01 PM, Adrien Mahieux wrote:
> Hello,
> 
> I think I've found a bug in the loop management. Maybe it's an
> expected behavior (didn't find any related topic on the manpages) but
> it's annoying in long-running scripts.
> 
> The leak is triggered by this idiom (wether or not jobcontrol + lastpipe) :
> cmd | read var
> 
> but not by this one :
> read var < <(cmd)
> 
> Tested with :
> - bash-4.2.46-19.el7.x86_64
> - bash-4.3.42-5.fc23.x86_64
> - bash-4.3-48.x86_64
> 
> From the version tested, it was fixed in version 4.4 (tested 4.4-beta)
> Do you have an idea of the commit that fixed this behavior, so the fix
> can be backported to older versions ?

Have you tried bisecting the devel git branch?

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: building: recutils bash builtins/readrec.so; ./configure --with-bash-headers=... # fails

[Chet Ramey]

On 11/3/17 6:28 PM, bug-b...@trodman.com wrote:
> No response today from bug-recutils:
> 
> http://lists.gnu.org/archive/html/bug-recutils/2017-11/msg4.html
> 
> For the ./configure, I untarred the latest bash source into a subdir, and set 
> --with-bash-headers=
> to that pathname. 

I don't know which part of the configuration process constructs the set of
bash include directories, but you need to add 'bash/include' to that list.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Memory leak in subshell + read before bash 4.4

[Adrien Mahieux]

On Sat, Nov 4, 2017 at 1:18 AM, Chet Ramey  wrote:
> On 11/3/17 6:01 PM, Adrien Mahieux wrote:
>> Hello,
>>
>> I think I've found a bug in the loop management. Maybe it's an
>> expected behavior (didn't find any related topic on the manpages) but
>> it's annoying in long-running scripts.
>>
>> The leak is triggered by this idiom (wether or not jobcontrol + lastpipe) :
>> cmd | read var
>>
>> but not by this one :
>> read var < <(cmd)
>>
>> Tested with :
>> - bash-4.2.46-19.el7.x86_64
>> - bash-4.3.42-5.fc23.x86_64
>> - bash-4.3-48.x86_64
>>
>> From the version tested, it was fixed in version 4.4 (tested 4.4-beta)
>> Do you have an idea of the commit that fixed this behavior, so the fix
>> can be backported to older versions ?
>
> Have you tried bisecting the devel git branch?

Didn't have time, nor tried. That's my first bisect \o/

85ec0778f9d778e1820fb8c0e3e996f2d1103b45 : no leak
1a088fee7368a8620e0bf64c481ac0bc5dcf : leak


This modification from 85ec0778 did fix the leak in 1a088fee73 :

-  if ((dflags & DEL_NOBGPID) == 0)
+  if ((dflags & DEL_NOBGPID) == 0 && (temp->flags &
(J_ASYNC|J_FOREGROUND)) == J_ASYNC)


Can you check and backport it ?

Cheers.

Re: Memory leak in subshell + read before bash 4.4

[Chet Ramey]

On 11/3/17 9:18 PM, Adrien Mahieux wrote:
> On Sat, Nov 4, 2017 at 1:18 AM, Chet Ramey  wrote:
>> On 11/3/17 6:01 PM, Adrien Mahieux wrote:
>>> Hello,
>>>
>>> I think I've found a bug in the loop management. Maybe it's an
>>> expected behavior (didn't find any related topic on the manpages) but
>>> it's annoying in long-running scripts.
>>>
>>> The leak is triggered by this idiom (wether or not jobcontrol + lastpipe) :
>>> cmd | read var
>>>
>>> but not by this one :
>>> read var < <(cmd)
>>>
>>> Tested with :
>>> - bash-4.2.46-19.el7.x86_64
>>> - bash-4.3.42-5.fc23.x86_64
>>> - bash-4.3-48.x86_64
>>>
>>> From the version tested, it was fixed in version 4.4 (tested 4.4-beta)
>>> Do you have an idea of the commit that fixed this behavior, so the fix
>>> can be backported to older versions ?
>>
>> Have you tried bisecting the devel git branch?
> 
> Didn't have time, nor tried. That's my first bisect \o/
> 
> 85ec0778f9d778e1820fb8c0e3e996f2d1103b45 : no leak
> 1a088fee7368a8620e0bf64c481ac0bc5dcf : leak
> 
> 
> This modification from 85ec0778 did fix the leak in 1a088fee73 :
> 
> -  if ((dflags & DEL_NOBGPID) == 0)
> +  if ((dflags & DEL_NOBGPID) == 0 && (temp->flags &
> (J_ASYNC|J_FOREGROUND)) == J_ASYNC)
> 
> 
> Can you check and backport it ?

If it's that, it's not a leak. It's bash-4.3 keeping exit statuses for the
last CHILD_MAX processes.  Bash-4.4 restricts that to the last CHILD_MAX
asynchronous background processes, with accompanying loss of functionality.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Memory leak in subshell + read before bash 4.4

[Adrien Mahieux]

On Sat, Nov 4, 2017 at 2:21 AM, Chet Ramey  wrote:
> If it's that, it's not a leak. It's bash-4.3 keeping exit statuses for the
> last CHILD_MAX processes.  Bash-4.4 restricts that to the last CHILD_MAX
> asynchronous background processes, with accompanying loss of functionality.

Ok, seems indeed to stall around the 50K loops (with "getconf
CHILD_MAX = 63460").

So, what's the difference in handling between "< <(cmd)" and "cmd | " ?

Is there any way to disable this feature ?
or should we consider these 500KB low enough to live with it,
regarding the current memory available on servers ?