Re: Leak in BASH "named" file descriptors?
On Wed, Jan 27, 2016 at 01:18:11PM -0500, Mathieu Patenaude wrote:
> When using "named" file descriptors inside a function, the file descriptors
> are not automatically un-linked when the function returns, but when using
> regular "numbered" file descriptors they are automatically "destroyed".
Could not reproduce in an interactive shell, on bash 4.3.30 (Debian).
$ f() { local fd; exec {fd}
Re: Leak in BASH "named" file descriptors?
Greg Wooledge writes:
> On Wed, Jan 27, 2016 at 01:18:11PM -0500, Mathieu Patenaude wrote:
>> When using "named" file descriptors inside a function, the file descriptors
>> are not automatically un-linked when the function returns, but when using
>> regular "numbered" file descriptors they are automatically "destroyed".
>
> Could not reproduce in an interactive shell, on bash 4.3.30 (Debian).
>
> $ f() { local fd; exec {fd} $ f
> $ g() { exec 9 $ g
> $ lsof -p $$
> ...
> bash931 wooledg0u CHR 136,0 0t0 3 /dev/pts/0
> bash931 wooledg1u CHR 136,0 0t0 3 /dev/pts/0
> bash931 wooledg2u CHR 136,0 0t0 3 /dev/pts/0
> bash931 wooledg9r CHR1,3 0t01028 /dev/null
> bash931 wooledg 10r CHR1,3 0t01028 /dev/null
> bash931 wooledg 255u CHR 136,0 0t0 3 /dev/pts/0
>
> I actually ran the lsof multiple times, in between the calls to f and g.
> FD 10 was opened by f (and kept open), and FD 9 was of course opened by g
> (and kept open).
Try replacing exec with :. Redirections established with exec are
always permanent.
Andreas.
--
Andreas Schwab, SUSE Labs, sch...@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."
Re: Leak in BASH "named" file descriptors?
Yes, using ":" also illustrate the same (or similar) behavior that I'm
experiencing with my script. Using the "here" string creates the
additional weirdness of showing that the temporary file content is actually
"deleted", but the bash process keep the FD open. Which is quite strange
since it appears to have done half the job...
Below is another difference with the "here" string construct - of course
this is a very inefficient way to assign a string to a variable!!! ;-)
but just used as a simple example here.
# Start a new shell to make sure there is no old FD left from previous tests
bash
unset -v TESTVAR
9<<<"test data" read -r -u 9 TESTVAR
printf "OUTPUT: $TESTVAR\n"
OUTPUT: test data
FD #9 is not left open or "deleted" here:
lsof -a -d '0-20' -p $$
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash12084 mathp0u CHR 136,4 0t07 /dev/pts/4
bash12084 mathp1u CHR 136,4 0t07 /dev/pts/4
bash12084 mathp2u CHR 136,4 0t07 /dev/pts/4
Now testing with the "named" (if that's a good name to call this) FD:
unset -v TEST_FD TESTVAR
{TEST_FD}<<<"test data" read -r -u $TEST_FD TESTVAR
bash: read: TESTVAR: invalid file descriptor specification
But strangely, the TEST_FD exists and is left in the "deleted" mode:
lsof -a -d '0-20' -p $$
COMMAND PID USER FD TYPE DEVICE SIZE/OFFNODE NAME
bash12084 mathp0u CHR 136,4 0t0 7 /dev/pts/4
bash12084 mathp1u CHR 136,4 0t0 7 /dev/pts/4
bash12084 mathp2u CHR 136,4 0t0 7 /dev/pts/4
bash12084 mathp 10r REG8,1 10 1049150
/tmp/sh-thd-512950288 (deleted)
Looks like the process of dynamically assigning a FD to the FD "variable"
breaks the redirection. But the FD is created and it's value did get
assigned to TEST_FD:
echo $TEST_FD
10
and temporary file content deleted, but the FD not un-linked
Doing the same thing but using 2 steps works, but you're still left with a
"deleted" FD:
unset -v TEST_FD TESTVAR
: {TEST_FD}<<<"test data"
read -r -u $TEST_FD TESTVAR
printf "OUTPUT: $TESTVAR FD: $TEST_FD\n"
OUTPUT: test data FD: 11
lsof -a -d '0-20' -p $$
COMMAND PID USER FD TYPE DEVICE SIZE/OFFNODE NAME
bash12084 mathp0u CHR 136,4 0t0 7 /dev/pts/4
bash12084 mathp1u CHR 136,4 0t0 7 /dev/pts/4
bash12084 mathp2u CHR 136,4 0t0 7 /dev/pts/4
bash12084 mathp 10r REG8,1 10 1049150
/tmp/sh-thd-512950288 (deleted)
bash12084 mathp 11r REG8,1 10 1049151
/tmp/sh-thd-263257624 (deleted)
I think that having a common behavior for "named" vs "numbered" FD in bash
would be great since obviously using dynamically assigned FD (numbers)
offers some advantages when coding more complex scripts.
Also related, assigning the output of the process substitution construct to
a "named" FD has the same behavior, i.e. you're left with an un-linked pipe:
unset -v TESTVAR
9< <(ls ~/t) read -r -u 9 TESTVAR
printf "OUTPUT: $TESTVAR\n"
OUTPUT: 0095.out
lsof -a -d '0-20' -p $$
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash16098 mathp0u CHR 136,4 0t07 /dev/pts/4
bash16098 mathp1u CHR 136,4 0t07 /dev/pts/4
bash16098 mathp2u CHR 136,4 0t07 /dev/pts/4
unset -v TEST_FD TESTVAR
: {TEST_FD}< <(ls ~/t)
read -r -u $TEST_FD TESTVAR
printf "OUTPUT: $TESTVAR FD: $TEST_FD\n"
OUTPUT: 0095.out FD: 10
lsof -a -d '0-20' -p $$
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash16098 mathp0u CHR 136,4 0t07 /dev/pts/4
bash16098 mathp1u CHR 136,4 0t07 /dev/pts/4
bash16098 mathp2u CHR 136,4 0t07 /dev/pts/4
bash16098 mathp 10r FIFO0,8 0t0 96683179 pipe
Thanks!
Math.
On Thu, Jan 28, 2016 at 8:58 AM, Andreas Schwab wrote:
> Greg Wooledge writes:
>
> > On Wed, Jan 27, 2016 at 01:18:11PM -0500, Mathieu Patenaude wrote:
> >> When using "named" file descriptors inside a function, the file
> descriptors
> >> are not automatically un-linked when the function returns, but when
> using
> >> regular "numbered" file descriptors they are automatically "destroyed".
> >
> > Could not reproduce in an interactive shell, on bash 4.3.30 (Debian).
> >
> > $ f() { local fd; exec {fd} > $ f
> > $ g() { exec 9 > $ g
> > $ lsof -p $$
> > ...
> > bash931 wooledg0u CHR 136,0 0t0 3 /dev/pts/0
> > bash931 wooledg1u CHR 136,0 0t0 3 /dev/pts/0
> > bash931 wooledg2u CHR 136,0 0t0 3 /dev/pts/0
> > bash931 wooledg9r CHR1,3 0t01028 /dev/null
> > bash931 wooledg 10r CHR1,3 0t01028 /dev/null
> > bash931 wooledg 255u CHR 136,0 0t0 3 /dev/pts/0
> >
> > I actually ran the lsof multiple times, in between the calls to f and g.
> > FD 10 was opened by f (and kept open), and FD 9 w
Re: Leak in BASH "named" file descriptors?
On Thu, Jan 28, 2016 at 12:40:57PM -0500, Mathieu Patenaude wrote: > Yes, using ":" also illustrate the same (or similar) behavior that I'm > experiencing with my script. Using the "here" string creates the > additional weirdness of showing that the temporary file content is actually > "deleted", but the bash process keep the FD open. Which is quite strange > since it appears to have done half the job... That's perfectly normal. The here-document or here-string payload is written to a temporary file, which is kept open, but unlinked. That way, when bash closes it (or is killed) the contents are simply deleted by the file system, and bash doesn't have to do the clean-up. I still don't know whether your original issue is a bug or not, though.
Re: Leak in BASH "named" file descriptors?
On 1/27/16 1:18 PM, Mathieu Patenaude wrote: > Hi, > > When using "named" file descriptors inside a function, the file descriptors > are not automatically un-linked when the function returns, but when using > regular "numbered" file descriptors they are automatically "destroyed". Yes. That's the intent. The idea is that if you assign a variable, you have a `handle' on the file descriptor and can manage it yourself. It's closer to how open(2) works: the system selects a descriptor and tells you what it is, and it's up to you to track it from there. In that sense, variables assigned by redirections are similar to how `exec' makes redirections persist beyond a single command. (I thought this had come up before, but I couldn't find that particular discussion.) Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Leak in BASH "named" file descriptors?
I guess the question remains, why does a "here string" assigned to a named
FD rely on the system to the the clean-up but when assigned to a regular,
numbered FD, it does not?
The issue I see with relying on the bash EXIT to actually have the system
do the cleanup is when you have a script that does things in a forever
loop, you end up with FD exhaustion when using "named" FD and here strings.
(kind of what my original script was showing) Say you have a "ulimit -f
1024, and you do more than 1024 iterations of the function, you clearly end
up with a "Too many open files". Again, not the case if I do the same
thing with a regular FD number.
Similar scenario if you replace the "here string" with the process
substitution output...
The issue does not seams to be with the "here string" at all, but the use
of the "named" FD.
Again, simple example of this idea, i.e. let's watch for a file and do
something if found. Assuming a f ulimit of 1024, this will stop printing
at about 1011 on my system
ittr=0
while :; do
while read -r -u $fh fname; do
if [[ $fname == file3 ]]; then
printf "$((ittr++)), "
# we could delete the file here and wait for the
# next time it appears, etc... Just an example.
fi
done {fh}< <(ls -1)
# we would normally wait some time here.
#sleep 1
[[ $? -ne 0 ]] && exit $?
done
But this will go on forever:
ittr=0
while :; do
while read -r -u 9 fname; do
if [[ $fname == file3 ]]; then
printf "$((ittr++)), "
# we could delete the file here and wait for the
# next time it appears, etc... Just an example.
fi
done 9< <(ls -1)
# we would normally wait some time here.
#sleep 1
[[ $? -ne 0 ]] && exit $?
done
Same thing with the "here" string, the following will print forever, but
not if I replace the 9 with $fh...
ittr=0
while :; do
while read -r -u 9 fname; do
if [[ $fname == file3 ]]; then
printf "$((ittr++)), "
# we could delete the file here and wait for the
# next time it appears, etc... Just an example.
fi
done 9<<<"file3"
# we would normally wait some time here.
#sleep 1
[[ $? -ne 0 ]] && exit $?
done
Again, thanks for looking into this weirdness.
On Thu, Jan 28, 2016 at 12:54 PM, Greg Wooledge wrote:
> On Thu, Jan 28, 2016 at 12:40:57PM -0500, Mathieu Patenaude wrote:
> > Yes, using ":" also illustrate the same (or similar) behavior that I'm
> > experiencing with my script. Using the "here" string creates the
> > additional weirdness of showing that the temporary file content is
> actually
> > "deleted", but the bash process keep the FD open. Which is quite strange
> > since it appears to have done half the job...
>
> That's perfectly normal. The here-document or here-string payload is
> written to a temporary file, which is kept open, but unlinked. That
> way, when bash closes it (or is killed) the contents are simply deleted
> by the file system, and bash doesn't have to do the clean-up.
>
> I still don't know whether your original issue is a bug or not, though.
>
Re: Leak in BASH "named" file descriptors?
Chet, thanks for the clarification So the answer is - this is by design, i.e. as soon as you use a variable for the FD, you are responsible of managing it for the duration of its life. Knowing this makes it much more "reliable" to use ;-) I also could not find the relevant info on the list / web, so I was really starting to think this was a bug. Maybe the "REDIRECTION" section of the BASH man page would be a good spot to mention this? Thanks to all for answering my questions, Math. On Thu, Jan 28, 2016 at 1:33 PM, Chet Ramey wrote: > On 1/27/16 1:18 PM, Mathieu Patenaude wrote: > > Hi, > > > > When using "named" file descriptors inside a function, the file > descriptors > > are not automatically un-linked when the function returns, but when using > > regular "numbered" file descriptors they are automatically "destroyed". > > Yes. That's the intent. The idea is that if you assign a variable, you > have a `handle' on the file descriptor and can manage it yourself. It's > closer to how open(2) works: the system selects a descriptor and tells you > what it is, and it's up to you to track it from there. > > In that sense, variables assigned by redirections are similar to how > `exec' makes redirections persist beyond a single command. > > (I thought this had come up before, but I couldn't find that particular > discussion.) > > Chet > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.edu > http://cnswww.cns.cwru.edu/~chet/ >
Re: Leak in BASH "named" file descriptors?
Mathieu Patenaude writes: > The issue I see with relying on the bash EXIT to actually have the system > do the cleanup is when you have a script that does things in a forever > loop, you end up with FD exhaustion when using "named" FD and here strings. Of course, if you don't close (or reuse) your descriptors you will run out of them eventually. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
[PATCH] admit 'typeset' is here to stay
In 'help typeset', the 'typeset' builtin is called obsolete and has been so since at least bash 2.05b (2002) or possibly earlier. Perhaps it's time to just call it a synonym, as indeed the texinfo documentation does. diff -ur bash-4.4-beta.orig/builtins/declare.def bash-4.4-beta/builtins/declare.def --- bash-4.4-beta.orig/builtins/declare.def 2015-08-24 19:02:28.0 +0100 +++ bash-4.4-beta/builtins/declare.def 2016-01-28 22:32:16.0 + @@ -65,7 +65,7 @@ $SHORT_DOC typeset [-aAfFgilnrtux] [-p] name[=value] ... Set variable values and attributes. -Obsolete. See `help declare'. +Synonym of `declare'. See `help declare'. $END #include
Re: Leak in BASH "named" file descriptors?
Hi, Am 28.01.2016 um 23:25 schrieb Andreas Schwab: > Mathieu Patenaude writes: > >> The issue I see with relying on the bash EXIT to actually have the system >> do the cleanup is when you have a script that does things in a forever >> loop, you end up with FD exhaustion when using "named" FD and here strings. > > Of course, if you don't close (or reuse) your descriptors you will run > out of them eventually. The man page of bash 4.2 says about REDIRECTION: In this case [...] the shell will allocate a file descriptor greater than 10 and assign it to varname. While looking into this discussion, I get 10 as first varname assigned in this way. Either a bug in bash or the man page? -- Reuti > > Andreas. > > -- > Andreas Schwab, sch...@linux-m68k.org > GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 > "And now for something completely different." >
Re: Leak in BASH "named" file descriptors?
On 1/28/16 6:56 PM, Reuti wrote: > The man page of bash 4.2 says about REDIRECTION: In this case [...] the shell > will allocate a file descriptor greater than 10 and assign it to varname. > > While looking into this discussion, I get 10 as first varname assigned in > this way. Either a bug in bash or the man page? That was changed years ago. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Leak in BASH "named" file descriptors?
Yes, thanks, that part I figured out many lines of code ago ;-) Chet answer earlier today kind of closed the loop on my "observations". By design, as soon as you work with FD by varname you have to do all the housekeeping (regardless on how you create them). But, depending on how you use regular "numbered" FD, it's not always the case (easy to see in all the examples I sent). On Thu, Jan 28, 2016 at 5:25 PM, Andreas Schwab wrote: > Mathieu Patenaude writes: > > > The issue I see with relying on the bash EXIT to actually have the system > > do the cleanup is when you have a script that does things in a forever > > loop, you end up with FD exhaustion when using "named" FD and here > strings. > > Of course, if you don't close (or reuse) your descriptors you will run > out of them eventually. > > Andreas. > > -- > Andreas Schwab, sch...@linux-m68k.org > GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 > "And now for something completely different." >
