Re: [Parameter Expansion] bug in ${variable% *}
On Mon, Feb 11, 2013 at 06:50:49PM +0100, Dashing wrote: > $ ./pe 'mplayer foo1\ foo2\ foo3\ 4\ 5\ foo6\ 7' > :--Mistake--: > :--Mistake--: Whatever you're doing, it's wrong. ./pe mplayer foo1\ foo2\ foo3\ 4\ 5\ foo6\ 7 #!/bin/bash prog=$1 shift exec "$prog" extra args go here "$@" THAT is how a wrapper should be done. Do not combine a program and its arguments all together into a single argument and then try to parse it apart.
Re: [Parameter Expansion] bug in ${variable% *}
On Tue, 12 Feb 2013 14:16:24 +0100 "Greg Wooledge" wrote: >On Mon, Feb 11, 2013 at 06:50:49PM +0100, Dashing wrote: >> $ ./pe 'mplayer foo1\ foo2\ foo3\ 4\ 5\ foo6\ 7' >> :--Mistake--: >> :--Mistake--: > >Whatever you're doing, it's wrong. > >./pe mplayer foo1\ foo2\ foo3\ 4\ 5\ foo6\ 7 > >#!/bin/bash >prog=$1 >shift >exec "$prog" extra args go here "$@" > >THAT is how a wrapper should be done. Do not combine a program >and its >arguments all together into a single argument and then try to >parse it >apart. For my purposes this is irrelevant, because the nature of the script from which my example code derived is tab completion. READLINE_LINE will contain mplayer foo1\ foo2\ etc. The last argument needs to be found and if it contains spaces this is what needs to happen.
Re: [Parameter Expansion] bug in ${variable% *}
On Tue, Feb 12, 2013 at 02:37:03PM +0100, Dashing wrote:
> For my purposes this is irrelevant, because the nature of the
> script from which my example code derived is tab completion.
> READLINE_LINE will contain mplayer foo1\ foo2\ etc.
Then you are still doing something wrong.
imadev:~$ args Tori\ Amos\ -\ Mr.\ Zebra.ogg
1 args:
There are no literal backslashes in an argument that is produced by
tab completion. The backslashes are a form of quoting, and they are
removed by the calling shell during the quote removal phase.
If you believe you have found a bug in parameter expansion, then
simplify the example so that it directly demonstrates the bug. For
example,
foo='some thing with \ \ \ \ backslashes'
echo "${foo##* }"
echo "I expected to see 'blah'"
If you are playing around with read -e, then your script should CONTAIN
that. And if you are reading shell commands in a shell script using
read -e and then attempting to parse them the way a shell would, then you
should reevaluate your goals, or the methods you are attempting to use to
attain them, because writing a shell in a shell is kinda wonky, isn't it?
Re: [Parameter Expansion] bug in ${variable% *}
On Tue, 12 Feb 2013 14:45:26 +0100 "Greg Wooledge" wrote: >There are no literal backslashes in an argument that is >produced by tab completion. The backslashes are a form >of quoting, and they are removed by the calling shell >during the quote removal phase. When bind -x sets READLINE_LINE backslashes are kept intact. What I meant was that I wrote my own tab completion. >If you believe you have found a bug in parameter expansion, >then simplify the example so that it directly demonstrates >the bug. As I showed in the output, it does not happen all the time. Even the same value of $rest one time works and another time does not. Therefore, looping over a bunch of items is required.
Re: eval doesn't close file descriptor?
Ok, but I see the same behaviour when eval runs in a subshell:
$ bash -c 'llfd () { echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/ >&2;
}; x=3; eval "exec $x>/dev/null"; llfd; echo | eval "llfd $x>&-"'
[same output, fd 10 open, pointing to /dev/null, even though it's a
subshell]
$ bash -c 'llfd () { echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/ >&2;
}; x=3; eval "exec $x>/dev/null"; llfd; echo | llfd 3>&-'
[not the same output; no fds pointing to /dev/null, as expected]
When eval is run by the main shell and I want $x closed, I can just do
'eval "exec $x>&-"'. However, I cannot do that with eval runs in a
subshell. In my script I needed $x open for other processes in that
pipeline.
On a different but related note, I hate having to do eval to manipulate an
fd stored in a variable. Why doesn't 'llfd $x>&-' work, especially since
'llfd >&$x' works just fine... so by the time >& is handled, the variable
substitutions seem to be done already.
On Tue, Feb 12, 2013 at 2:13 AM, Pierre Gaston wrote:
>
>
> On Tue, Feb 12, 2013 at 1:54 AM, wrote:
>
>> With the script below, I'd expect any fd pointing to /dev/null to be
>> closed when the second llfd() is executed. Surprisingly, fd 3 is closed,
>> but fd 10 is now open, pointing to /dev/null, as if eval copied it instead
>> of closing it. Is this a bug?
>>
>> Thanks,
>> M
>>
>>
>> $ bash -c 'llfd () { ls -l /proc/$BASHPID/fd/; }; x=3; eval "exec
>> $x>/dev/null"; llfd; eval "llfd $x>&-"'
>> total 0
>> lrwx-- 1 matei matei 64 Feb 11 18:36 0 -> /dev/pts/2
>> lrwx-- 1 matei matei 64 Feb 11 18:36 1 -> /dev/pts/2
>> lrwx-- 1 matei matei 64 Feb 11 18:36 2 -> /dev/pts/2
>> l-wx-- 1 matei matei 64 Feb 11 18:36 3 -> /dev/null
>> lr-x-- 1 matei matei 64 Feb 11 18:36 8 -> /proc/4520/auxv
>> total 0
>> lrwx-- 1 matei matei 64 Feb 11 18:36 0 -> /dev/pts/2
>> lrwx-- 1 matei matei 64 Feb 11 18:36 1 -> /dev/pts/2
>> l-wx-- 1 matei matei 64 Feb 11 18:36 10 -> /dev/null
>> lrwx-- 1 matei matei 64 Feb 11 18:36 2 -> /dev/pts/2
>> lr-x-- 1 matei matei 64 Feb 11 18:36 8 -> /proc/4520/auxv
>> $ bash --version
>> GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu)
>> Copyright (C) 2011 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>>
>> This is free software; you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.
>> $
>>
>
> Note that the same happens without using eval:
> $ llfd 3>&-
> total 0
> lrwx-- 1 pgas pgas 64 Feb 12 08:00 0 -> /dev/pts/0
> lrwx-- 1 pgas pgas 64 Feb 12 08:00 1 -> /dev/pts/0
> l-wx-- 1 pgas pgas 64 Feb 12 08:00 10 -> /dev/null
> lrwx-- 1 pgas pgas 64 Feb 12 08:00 2 -> /dev/pts/0
> lrwx-- 1 pgas pgas 64 Feb 12 08:00 255 -> /dev/pts/0
>
> But you need to consider what process you are examining, you use a
> function and you examine the file descriptors of the process where this
> function runs.
>
> A function runs in the same process as the parent shell, if it simply
> closes 3 then there will be no more fd opened on >/dev/null in the parent
> shell when the function returns
> So what bash does is a little juggling with the file descriptors, moving 3
> temporarily to be able to restore it.
>
>
>
>
>
>
>
>
Re: eval doesn't close file descriptor?
On Tue, Feb 12, 2013 at 11:07:06AM -0500, Matei David wrote:
> On a different but related note, I hate having to do eval to manipulate an
> fd stored in a variable. Why doesn't 'llfd $x>&-' work, especially since
> 'llfd >&$x' works just fine... so by the time >& is handled, the variable
> substitutions seem to be done already.
Each redirection that may be preceded by a file descriptor number may
instead be preceded by a word of the form {varname}. In this case,
for each redirection operator except >&- and <&-, the shell will
allocate a file descriptor greater than 10 and assign it to varname.
If >&- or <&- is preceded by {varname}, the value of varname defines
the file descriptor to close.
This was added in Bash 4.1.
Re: [Parameter Expansion] bug in ${variable% *}
Greg Wooledge writes: > Then you are still doing something wrong. No matter how badly the script is written, bash should execute it properly, which it doesn't. Try running it under valgrind in a multibyte locale and you will see the error immediately: ==18677== Conditional jump or move depends on uninitialised value(s) ==18677==at 0x4504B6: xdupmbstowcs (xmbsrtowcs.c:240) ==18677==by 0x42A43F: remove_pattern (subst.c:3964) ==18677==by 0x42F3B7: parameter_brace_expand (subst.c:4543) ==18677==by 0x466E48: param_expand (subst.c:7651) ==18677==by 0x467C91: expand_word_internal (subst.c:8139) ==18677==by 0x467F24: expand_word_internal (subst.c:8285) ==18677==by 0x42F981: call_expand_word_internal.constprop.15 (subst.c:3197) ==18677==by 0x42FD94: expand_string_assignment (subst.c:3283) ==18677==by 0x466BB3: expand_string_if_necessary (subst.c:3046) ==18677==by 0x47632F: do_assignment_internal (subst.c:2772) ==18677==by 0x469726: expand_word_list_internal (subst.c:2866) ==18677==by 0x45CCB3: execute_simple_command (execute_cmd.c:3776) ==18677== Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Re: eval doesn't close file descriptor?
On Tue, Feb 12, 2013 at 6:07 PM, Matei David wrote:
> Ok, but I see the same behaviour when eval runs in a subshell:
>
> $ bash -c 'llfd () { echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/
> >&2; }; x=3; eval "exec $x>/dev/null"; llfd; echo | eval "llfd $x>&-"'
> [same output, fd 10 open, pointing to /dev/null, even though it's a
> subshell]
>
eval runs in a subshell, but it's the same thing inside this subshell.
eg you could have: echo | { eval "llfd "$x>&-"; echo blah >&3; }
Bash could optimize this once it realizes there's only one command, but
it's probably not that simple to implement.
Try with a function that spawns a subshell eg:
llfd () ( echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/ >&2; )
or llfd () { bash -c 'ls -l /proc/$$/fd' ; }
Re: [Parameter Expansion] bug in ${variable% *}
On 2/12/13 11:37 AM, Andreas Schwab wrote: > Greg Wooledge writes: > >> Then you are still doing something wrong. > > No matter how badly the script is written, bash should execute it > properly, which it doesn't. Try running it under valgrind in a > multibyte locale and you will see the error immediately: There is an error in xdupmbstowcs, but beyond that the output from valgrind is fairly useless, all the more so because you didn't post the version (most likely the 4.2 patch level) you're using. If you're using patch 42, the line number corresponds to a return of a value that's always initialized. In any event, the problem is in a different place, and I've fixed it for the next version. Chet > > ==18677== Conditional jump or move depends on uninitialised value(s) > ==18677==at 0x4504B6: xdupmbstowcs (xmbsrtowcs.c:240) > ==18677==by 0x42A43F: remove_pattern (subst.c:3964) > ==18677==by 0x42F3B7: parameter_brace_expand (subst.c:4543) > ==18677==by 0x466E48: param_expand (subst.c:7651) > ==18677==by 0x467C91: expand_word_internal (subst.c:8139) > ==18677==by 0x467F24: expand_word_internal (subst.c:8285) > ==18677==by 0x42F981: call_expand_word_internal.constprop.15 > (subst.c:3197) > ==18677==by 0x42FD94: expand_string_assignment (subst.c:3283) > ==18677==by 0x466BB3: expand_string_if_necessary (subst.c:3046) > ==18677==by 0x47632F: do_assignment_internal (subst.c:2772) > ==18677==by 0x469726: expand_word_list_internal (subst.c:2866) > ==18677==by 0x45CCB3: execute_simple_command (execute_cmd.c:3776) > ==18677== > > Andreas. > -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Parameter Expansion] bug in ${variable% *}
> Bash version: 4.2.042
>
> I have a script that behaves erratically:
Thanks for the report. The reason it is erratic and not always
reproducible in the same spot is that it is the result of an array
bounds error, and so depends on the contents of memory beyond the
end of a string.
I've fixed the problem, and the fix will be in the next release. If
you're interested in the fix, I've attached a patch.
Chet
*** ../bash-4.2-patched/lib/glob/xmbsrtowcs.c 2012-07-08 21:53:19.0
-0400
--- lib/glob/xmbsrtowcs.c 2013-02-12 12:00:39.0 -0500
***
*** 217,220
--- 217,226
n = mbsnrtowcs(wsbuf+wcnum, &p, nms, wsbuf_size-wcnum, &state);
+ if (n == 0 && p == 0)
+ {
+ wsbuf[wcnum] = L'\0';
+ break;
+ }
+
/* Compensate for taking single byte on wcs conversion failure above. */
if (wcslength == 1 && (n == 0 || n == (size_t)-1))
***
*** 222,226
state = tmp_state;
p = tmp_p;
! wsbuf[wcnum++] = *p++;
}
else
--- 228,238
state = tmp_state;
p = tmp_p;
! wsbuf[wcnum] = *p;
! if (*p == 0)
! break;
! else
! {
! wcnum++; p++;
! }
}
else
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Parameter Expansion] bug in ${variable% *}
On 11.02.2013 18:50, Dashing wrote:
Bash version: 4.2.042
I have a script that behaves erratically:
=
#! /bin/bash
last=${1##* }
rest=${1% *}
while [[ "${rest: -1}" == '\' ]]; do
last="${rest##* } $last"
oldrest=$rest
rest=${rest% *}
if [[ "$oldrest" == "$rest" ]]; then
echo :--Mistake--:
# sleep 0.01
# rest=${rest% *}
# if [[ "$oldrest" == "$rest" ]]; then
# echo 'unable to interpret'
# break
# fi
fi
done
echo REST:$rest:
echo LAST:$last:
=
$ ./pe 'mplayer foo1\ foo2\ foo3\ 4\ 5\ foo6\ 7'
:--Mistake--:
:--Mistake--:
REST:mplayer:
LAST:foo1\ foo1\ foo1\ foo2\ foo3\ 4\ 5\ foo6\ 7:
=
What happens is that rest=${rest% *} doesn't always update $rest,
even when there are spaces left in $rest.
Meanwhile last="${rest##* } $last" works every time.
In the above example it got stuck twice when $rest was mplayer
foo1\. It tends to vary wildly.
The script runs fine if you omit the superfluous blank in the ${rest:
-1} expression:
while [[ "${rest:-1}" == '\' ]]; do
If the script behaves indeterministic, it is probably a bash bug.
Regards,
Bernd
--
www.sudrala.de
printf question
Dear All, I need to run a command that concat a list of files. It work like cat in UNIX. teqc.exe file1 file2 ... > output Then I want to build this list of file with bash, and insert it in a shell script. > liste=`ls *.T02` > echo $liste SHOB087k.T02 SHOB087l.T02 SHOB087m.T02 SHOB087n.T02 SHOB087o.T02 SHOB087p.T02 SHOB087q.T02 SHOB087r.T02 SHOB087s.T02 SHOB087t.T02 SHOB087u.T02 SHOB087v.T02 SHOB087w.T02 SHOB087x.T02 SHOB088a.T02 SHOB088b.T02 SHOB088c.T02 SHOB088d.T02 SHOB088e.T02 SHOB088f.T02 SHOB088g.T02 SHOB088h.T02 SHOB088i.T02 SHOB088j.T02 SHOB088k.T02 SHOB088l.T02 SHOB088m.T02 SHOB088n.T02 SHOB088o.T02 SHOB088p.T02 SHOB088q.T02 Echo gives me the list files I want (although the first 4 letter are in UPPERCASE which is not the case of the real list?), in a row format, but printf doesn't work the same way and I don't understant why. > printf $liste > SHOB087k.T02 return the first element (in uppercase format ???) > printf "$liste" > SHOB087k.T02 SHOB087l.T02 SHOB087m.T02 SHOB087n.T02 . . return the list element in column format ?? What shouls I put i my shell 1> teqc.exe `ls *.T02` > out.T02 2> teqc.exe $liste> out.T02 3> teqc.exe "$liste" > out.T02 What is the proper way of doing this ? Thanks for your help Regards, Laurent
Re: eval doesn't close file descriptor?
Wow, thanks, I didn't know that. So this syntax achieves a bit more than I
asked for- it allows you to get a new unused file descriptor, right? It
seems that the only useful way to use a non-closing form (>&-, <&-) is with
exec, as in 'exec {new_fd}>&2'. (Why would I want the fd in a variable
otherwise.)
Too bad the "natural" syntax 'llfd $x>&-' doesn't work, but I guess this
will do.
On Tue, Feb 12, 2013 at 11:12 AM, Greg Wooledge wrote:
> On Tue, Feb 12, 2013 at 11:07:06AM -0500, Matei David wrote:
> > On a different but related note, I hate having to do eval to manipulate
> an
> > fd stored in a variable. Why doesn't 'llfd $x>&-' work, especially since
> > 'llfd >&$x' works just fine... so by the time >& is handled, the variable
> > substitutions seem to be done already.
>
> Each redirection that may be preceded by a file descriptor number may
> instead be preceded by a word of the form {varname}. In this case,
> for each redirection operator except >&- and <&-, the shell will
> allocate a file descriptor greater than 10 and assign it to varname.
> If >&- or <&- is preceded by {varname}, the value of varname defines
> the file descriptor to close.
>
> This was added in Bash 4.1.
>
Re: [Parameter Expansion] bug in ${variable% *}
On Tue, 12 Feb 2013 18:02:05 +0100 "Chet Ramey"
>I've fixed the problem, and the fix will be in the next release.
Thank you, Chet!
On Tue, 12 Feb 2013 10:13:46 +0100 "Bernd Eggink"
>superfluous blank in the ${rest: -1} expression
Bernd, this blank makes a big difference:
${rest: -1} last character of $rest
${rest:-1} $rest or 1 if $rest is empty
Re: printf question
On Tue, Feb 12, 2013 at 04:20:53AM -0800, laurent.tes...@gmail.com wrote:
> > liste=`ls *.T02`
This is broken because filenames may contain spaces, newlines, etc.
Use an array instead:
liste=(*.T02)
See http://mywiki.wooledge.org/BashPitfalls for an explanation of some
of the issues.
This type of question is more appropriate for the help-bash mailing list,
by the way.
> 1> teqc.exe `ls *.T02` > out.T02
> 2> teqc.exe $liste> out.T02
> 3> teqc.exe "$liste" > out.T02
All of those are wrong, each in its own way.
> What is the proper way of doing this ?
Well, the easiest way would be:
teqc.exe *.T02 > out.T02
If you insist on storing the filenames in advance for some reason, use
an array:
files=(*.T02)
teqc.exe "${files[@]}" > out.T02
If you run into command line length issues, then you may have to process
the array in several chunks. See http://mywiki.wooledge.org/BashFAQ/095
for examples of that.
Re: eval doesn't close file descriptor?
So in other words, you're saying I should use '... | eval "exec $x>&-;
llfd"' instead of '... | eval "llfd $x>&-"'. This way the subshell won't be
assuming I might use $x later. That works, but I still find it
counterintuitive that with the original syntax the subshell doesn't realize
there's nothing left to execute after $x>&-.
Also, I tried your other suggestions. The second one 'llfd () { bash -c
}' works, but the other 'llfd () ( ... )' doesn't! I tried to
understand why...
Looking at this:
$ bash -c 'llfd () { echo "pid:$BASHPID" >&2; ls -gG /proc/$BASHPID/fd >&2;
}; f () { llfd; (llfd); bash -c "echo pid:\$\$ >&2; ls -gG /proc/\$\$/fd
>&2"; }; x=3; exec 4>/tmp/fd_4; coproc cat; eval "exec $x>/tmp/fd_3"; llfd;
echo | eval "f $x>&-"'
pid:14920
total 0
lrwx-- 1 64 Feb 12 14:03 0 -> /dev/pts/2
lrwx-- 1 64 Feb 12 14:03 1 -> /dev/pts/2
lrwx-- 1 64 Feb 12 14:03 2 -> /dev/pts/2
l-wx-- 1 64 Feb 12 14:03 3 -> /tmp/fd_3
l-wx-- 1 64 Feb 12 14:03 4 -> /tmp/fd_4
l-wx-- 1 64 Feb 12 14:03 60 -> pipe:[5010928]
lr-x-- 1 64 Feb 12 14:03 63 -> pipe:[5010927]
lr-x-- 1 64 Feb 12 14:03 8 -> /proc/4520/auxv
pid:14924
total 0
lr-x-- 1 64 Feb 12 14:03 0 -> pipe:[5007145]
lrwx-- 1 64 Feb 12 14:03 1 -> /dev/pts/2
l-wx-- 1 64 Feb 12 14:03 10 -> /tmp/fd_3
lrwx-- 1 64 Feb 12 14:03 2 -> /dev/pts/2
l-wx-- 1 64 Feb 12 14:03 4 -> /tmp/fd_4
lr-x-- 1 64 Feb 12 14:03 8 -> /proc/4520/auxv
pid:14926
total 0
lr-x-- 1 64 Feb 12 14:03 0 -> pipe:[5007145]
lrwx-- 1 64 Feb 12 14:03 1 -> /dev/pts/2
l-wx-- 1 64 Feb 12 14:03 10 -> /tmp/fd_3
lrwx-- 1 64 Feb 12 14:03 2 -> /dev/pts/2
l-wx-- 1 64 Feb 12 14:03 4 -> /tmp/fd_4
lr-x-- 1 64 Feb 12 14:03 8 -> /proc/4520/auxv
pid:14928
total 0
lr-x-- 1 64 Feb 12 14:03 0 -> pipe:[5007145]
lrwx-- 1 64 Feb 12 14:03 1 -> /dev/pts/2
lrwx-- 1 64 Feb 12 14:03 2 -> /dev/pts/2
l-wx-- 1 64 Feb 12 14:03 4 -> /tmp/fd_4
lr-x-- 1 64 Feb 12 14:03 8 -> /proc/4520/auxv
... there seem to be not 2 but 3(!) types of file descriptors:
1. fds which are copied across both subshells and exec; like 4
2. fds which are not copied across subshells; like 60&63
3. fds which are copied across subshells, but not exec; like 10
I knew about types 1&2, but not about type 3. Apparently with your first
suggestion, fd 10 is created and survives a subshell creation. Is this
correct??
On Tue, Feb 12, 2013 at 11:40 AM, Pierre Gaston wrote:
>
>
> On Tue, Feb 12, 2013 at 6:07 PM, Matei David wrote:
>
>> Ok, but I see the same behaviour when eval runs in a subshell:
>>
>> $ bash -c 'llfd () { echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/
>> >&2; }; x=3; eval "exec $x>/dev/null"; llfd; echo | eval "llfd $x>&-"'
>> [same output, fd 10 open, pointing to /dev/null, even though it's a
>> subshell]
>>
>
> eval runs in a subshell, but it's the same thing inside this subshell.
> eg you could have: echo | { eval "llfd "$x>&-"; echo blah >&3; }
>
> Bash could optimize this once it realizes there's only one command, but
> it's probably not that simple to implement.
>
> Try with a function that spawns a subshell eg:
> llfd () ( echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/ >&2; )
>
> or llfd () { bash -c 'ls -l /proc/$$/fd' ; }
>
>
>
Re: eval doesn't close file descriptor?
On 2/12/13 2:07 PM, Matei David wrote: > ... there seem to be not 2 but 3(!) types of file descriptors: > 1. fds which are copied across both subshells and exec; like 4 > 2. fds which are not copied across subshells; like 60&63 > 3. fds which are copied across subshells, but not exec; like 10 > > I knew about types 1&2, but not about type 3. Apparently with your first > suggestion, fd 10 is created and survives a subshell creation. Is this > correct?? Yes, file descriptors used to save the state of other file descriptors are set close-on-exec. `man fcntl' for a description. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: eval doesn't close file descriptor?
Hi Chet, Conceptually, the availability of a single flag "close-on-exec" (BTW, is there a way to check those flags in bash or using /proc?) should create only 2, not 3 types of file descriptors- that's what I find confusing. Does subprocess creation as in '(llfd)' entail an execve() call? My guess is that no, because the "extended environment" like arrays and function definitions would not survive it, so you'd have to copy them by hand. Either way: If subprocess creation DOES NOT perform execve(), then I don't understand fd's of type 2, like 60&63: they should continue to exist in a subshell. If subprocess creation DOES perform execve(), then I don't understand fd's of type 3: they should not exist in a subshell. My best guess here is that type 2 is non-standard? Meaning that they are closed by bash during subprocess creation, even though there is no execve()? I only came across those when trying to use coproc... I wonder why this decision was made. Generally speaking, it took me quite some time to figure out how to properly create a "double pipe" without using any intermediate files or fifos. The concept is very easy: in -> tee -> two parallel, independent pipes -> join -> out. A first complication is the limited pipe capacity, and the possibility of one side getting stuck if the other stops pulling in data. I then wrote a cat-like program which doesn't block on stdout, but keeps reading stdin, buffering in as much data as needed. I used it at the very end of either pipe. But then I had to create the actual processes, and then I stumbled upon all these issues with coproc and file descriptors. You leave the wrong one open and the thing gets stuck... I wish there was a howto on this subject. Thanks, M On Tue, Feb 12, 2013 at 2:50 PM, Chet Ramey wrote: > On 2/12/13 2:07 PM, Matei David wrote: > > > ... there seem to be not 2 but 3(!) types of file descriptors: > > 1. fds which are copied across both subshells and exec; like 4 > > 2. fds which are not copied across subshells; like 60&63 > > 3. fds which are copied across subshells, but not exec; like 10 > > > > I knew about types 1&2, but not about type 3. Apparently with your first > > suggestion, fd 10 is created and survives a subshell creation. Is this > > correct?? > > Yes, file descriptors used to save the state of other file descriptors > are set close-on-exec. `man fcntl' for a description. > > Chet > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.edu > http://cnswww.cns.cwru.edu/~chet/ >
Re: eval doesn't close file descriptor?
On 2/12/13 2:07 PM, Matei David wrote: OK. Let me try this again. One key to the explanations is that child processes inherit all of the parent's open file descriptors. > ... there seem to be not 2 but 3(!) types of file descriptors: > 1. fds which are copied across both subshells and exec; like 4 File descriptors that are explicitly created by user redirections are inherited and not set close-on-exec. > 2. fds which are not copied across subshells; like 60&63 You created a coproc; bash opened pipes to the coproc and is careful to close those when child processes are created. Having, say, the read end of a pipe open in multiple processes really messes with SIGPIPE generation. Multiple fds open for write means that read will never see EOF, even if the process you intend to write to it exits. This is true for pipes in general, so bash is very careful to keep track of file descriptors open on pipes and close them in child processes. > 3. fds which are copied across subshells, but not exec; like 10 File descriptors which are used internally to save state of file descriptors used in redirections are set close-on-exec. > I knew about types 1&2, but not about type 3. Apparently with your first > suggestion, fd 10 is created and survives a subshell creation. Is this > correct?? Yes. A subshell gets copies of its parent's file descriptors. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: eval doesn't close file descriptor?
On Tue, Feb 12, 2013 at 04:22:08PM -0500, Matei David wrote: > Generally speaking, it took me quite some time to figure out how to > properly create a "double pipe" without using any intermediate files or > fifos. The concept is very easy: in -> tee -> two parallel, independent > pipes -> join -> out. A first complication is the limited pipe capacity, > and the possibility of one side getting stuck if the other stops pulling in > data. I then wrote a cat-like program which doesn't block on stdout, but > keeps reading stdin, buffering in as much data as needed. I used it at the > very end of either pipe. But then I had to create the actual processes, and > then I stumbled upon all these issues with coproc and file descriptors. You > leave the wrong one open and the thing gets stuck... I wish there was a > howto on this subject. I really like to see these threads that poke at the edges of things in bash, providing a great learning opportunity for lurkers like me. But that said, perhaps what you're trying to so might be handled easily by socat; from http://www.dest-unreach.org/socat/doc/README: socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device..., a socket..., a file descriptor (stdin etc.) ... Just a thought. Ken
Re: [Parameter Expansion] bug in ${variable% *}
On 12.02.2013 18:50, Dashing wrote:
On Tue, 12 Feb 2013 18:02:05 +0100 "Chet Ramey"
I've fixed the problem, and the fix will be in the next release.
Thank you, Chet!
On Tue, 12 Feb 2013 10:13:46 +0100 "Bernd Eggink"
superfluous blank in the ${rest: -1} expression
Bernd, this blank makes a big difference:
${rest: -1} last character of $rest
${rest:-1} $rest or 1 if $rest is empty
You're right, of course. Sigh. Fortunately I've not yet fallen into that
pithole...
Bernd
--
www.sudrala.de
Re: eval doesn't close file descriptor?
On 2/12/13 11:40 AM, Pierre Gaston wrote:
> On Tue, Feb 12, 2013 at 6:07 PM, Matei David wrote:
>
>> Ok, but I see the same behaviour when eval runs in a subshell:
>>
>> $ bash -c 'llfd () { echo "pid:$BASHPID" >&2; ls -l /proc/$BASHPID/fd/
>>> &2; }; x=3; eval "exec $x>/dev/null"; llfd; echo | eval "llfd $x>&-"'
>> [same output, fd 10 open, pointing to /dev/null, even though it's a
>> subshell]
>>
>
> eval runs in a subshell, but it's the same thing inside this subshell.
> eg you could have: echo | { eval "llfd "$x>&-"; echo blah >&3; }
>
> Bash could optimize this once it realizes there's only one command, but
> it's probably not that simple to implement.
The basic flow is like this for any builtin command or shell function that
has a redirection (let's choose 'llfd 3>&-').
1. The redirection is performed in the current shell, noting that it
should be `undoable'. That takes three steps:
1a. In this case, since fd 3 is in use, we dup it (to fd 10) and mark fd
10 as close-on-exec. We add a separate redirection to an internal
list that basically says "close fd 10". Then we add another
redirection to the front of the same internal list that says "dup fd
10 back to fd 3". Let's call this list "redirection_undo_list". We
will use it to restore the original state after the builtin or
function completes.
1b. Take the first redirection from step 1a and add it to a separate
internal list that will clean up internal redirections in the case
that exec causes the redirections to be preserved, and not undone.
Let's call this list "exec_redirection_undo_list".
1c. Perform the redirection. Here, that means close fd 3.
[perform step 1 for each redirection associated with the command]
2. If we're running the exec builtin, throw away the list from 1a. If
we're not running the exec builtin, throw away the list from 1b. Save
a handle to the list we didn't discard.
3. Run the function or builtin.
4. Take the list saved in step 2 and perform the redirections to
restore the previous state. Here, that means we dup fd 10 back to fd
3, then close fd 10.
If you look at the steps, it should be clear why fd 10 is still open when
llfd executes.
Bash `cheats' when running builtins or shell functions in pipelines or
other subshells. It knows it's already going to be in a child process
when it performs the redirections, so it doesn't bother setting up the
structures to undo them.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
