Re: why are \d and \D not implemented but don't throw errors in regex?
On 12/10/2013 12:44 PM, Eric Blake wrote: On 12/10/2013 01:28 PM, Linda Walsh wrote: But before bash takes on another library, it would be nicer to FIRST get libc patched to provide pcre regex by default. In other words, I think you are better off lobbying with the POSIX folks to standardize pcre --- I was under the impression that they looked for current practices to indicate such a change would be good. If people don't implement ahead of the curve, the curve won't change.
Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used
On 12/11/2013 01:04, Piotr Grzybowski wrote: hullo, maybe post exact scripts that generate those files, together with some description of the environment, arent some of those scripts run from su? Simple command run under unprivileged user creates such files: tee >(grep "XXX" | do-filter >> my.log) Yuri
Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used
Also, as a correction to your original message, a FIFO _is_ a file, just not a regular file. Sent from my iPad > On Dec 11, 2013, at 1:13 AM, Yuri wrote: > >> On 12/11/2013 01:04, Piotr Grzybowski wrote: >> hullo, >> >> maybe post exact scripts that generate those files, together with >> some description of the environment, arent some of those scripts run >> from su? > > > Simple command run under unprivileged user creates such files: > tee >(grep "XXX" | do-filter >> my.log) > > Yuri >
Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used
First off, remind Piotr that he forgot to carbon-copy his e-mail message to this list (bug-bash@gnu.org); and tell him to do so. Second, you should post the definition of 'do-filter', if 'do-filter' is not a binary (also post the definitions of other custom functions 'do-filter' calls; but do not post the complete script). Sincerely, Ryan Cunningham Sent from my iPad > On Dec 11, 2013, at 1:13 AM, Yuri wrote: > >> On 12/11/2013 01:04, Piotr Grzybowski wrote: >> hullo, >> >> maybe post exact scripts that generate those files, together with >> some description of the environment, arent some of those scripts run >> from su? > > > Simple command run under unprivileged user creates such files: > tee >(grep "XXX" | do-filter >> my.log) > > Yuri >
Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used
On 12/11/2013 07:20, Ryan Cunningham wrote:
Second, you should post the definition of 'do-filter', if 'do-filter' is not a
binary (also post the definitions of other custom functions 'do-filter' calls;
but do not post the complete script).
do-filter is some shell script running some other commands. I didn't
think it mattered for this purpose. But in any case, it is one line
script placed in a file and made executable:
gawk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'
Also, as a correction to your original message, a FIFO _is_ a file, just not a
regular file.
Yes, it is a special kind of file, but it has a special type S_IFIFO and
ls(1) shows it with 'p' type and it also has size=0. What was my point
is that this FIFO is sometimes created as a proper FIFO:
prw--- 1 yuri wheel 0 Dec 11 01:51 sh-np-1386834327
and sometimes as a regular (non-FIFO) file:
-rw-r--r-- 1 yuri wheel3721815 Dec 11 02:00 sh-np-1386805482
And the code shows that only mkfifo(2) is used. (FIFO doesn't leave any
persistent data when abandoned, but those files do leave data.) I am not
sure why is that that it creates them as files? Doesn't it look strange?
Yuri
Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used
On 12/10/2013 23:29, Yuri wrote:
Some of my scripts use command substitution, and now I see that there
are lots of files like these in /tmp:
prw--- 1 yuri wheel 0 Dec 10 13:32 sh-np-1386738492
-rw-r--r-- 1 yuri wheel3278909 Dec 10 14:54 sh-np-1386721176
Besides the obvious question why they aren't deleted in the end, two
questions:
Shouldn't bash delete the pipe files once the child process opened
them? On most platforms the deleted pipe will function the same with
the benefit that it will get deleted automatically.
Why some of he files /tmp/sh-np-NNN are regular files, not pipes? When
I look through code I see mkfifo call creates them. Shouldn't it
always create fifos and not files?
Ok, I figured the details out.
The actual script that I run is this radio playing command:
stdbuf -i 0 -o 0 -e 0 mplayer
http://kqed-ice.streamguys.org:80/kqedradio-ch-e1 | \
stdbuf -i 0 -o 0 -e 0 tee >(grep "^XXX XXX" | prepend-time >> test.log)
prepend-time is a script:
gawk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'
The problem only occurs with stdbuf(1) commands inserted there, and not
without them.
1: bash creates a FIFO /tmp/sh-np-N
2: bash unlinks this FIFO
3: tee is passed /tmp/sh-np-N as an argument, and it creates a
regular file with the same name
This is the race condition.
Obviously, bash doesn't ensure that unlink is issued after the child
command opons it, and not before that. Order of operations 2 and 3
should be reversed.
And just waiting for a while isn't enough! bash should only unlink the
fifo after it has been open by the child command.
Please also note, that this is on FreeBSD, which behavior might be
different. Linux or other OSes might be getting lucky here with timing,
not sure. FreeBSD also has its own version of stdbuf(1).
This bug needs to be fixed.
The original intention of stdbuf there was to make it all unbufferer.
Whether this works or not isn't clear, but it doesn't matter for the
purpose of this bug report.
Suggested solution:
On FreeBSD, for example, there is the kqueue(2) facility, which is able
to wait on the event of "link-change": EVFILT_VNODE operation (takes
file descriptor) allows sub-operation NOTE_LINK (The link count on the
file changed)
So the sequence will be like this:
1. bash creates FIFO
2. bash opens FIFO
3. bash waits with kqueue(2) with EVFILT_VNODE/NOTE_LINK
4. bash launches the child command
5. child command opens the FIFO as stdio
6. bash catches the link-change event
7. bash unlinks FIFO
Linux and other OSes must have an equivalent facility.
Sounds like a fun project to do. -)
Absolute must-have if bash needs to support command substitution.
Yuri
CCing to the maintainer of bash on FreeBSD
Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used
helo!
before Ryan kills me :) I am replying to all, no matter what
meaningless rubbish I post. tell Ryan I got his message :)
Yuri: I have verified that under linux 3.2.0, bash-4.2.25, nothing of
the sort takes place. tee gets /dev/fd/${somefd}, why it is not
supported in bsd kernel i have no idea, but I thought it was. I've
even built mplayer and run your script (btw. probably there is an
easier way of doing what you want), I also performed some tests with
command that actually outputs what grep looks for. Also please note
that `command substitution`, or $(command substitution) is slightly
different from >(process substitution).
sincerely,
pg
On Wed, Dec 11, 2013 at 10:36 PM, Yuri wrote:
> On 12/10/2013 23:29, Yuri wrote:
>>
>> Some of my scripts use command substitution, and now I see that there are
>> lots of files like these in /tmp:
>> prw--- 1 yuri wheel 0 Dec 10 13:32 sh-np-1386738492
>> -rw-r--r-- 1 yuri wheel3278909 Dec 10 14:54 sh-np-1386721176
>>
>> Besides the obvious question why they aren't deleted in the end, two
>> questions:
>> Shouldn't bash delete the pipe files once the child process opened them?
>> On most platforms the deleted pipe will function the same with the benefit
>> that it will get deleted automatically.
>> Why some of he files /tmp/sh-np-NNN are regular files, not pipes? When I
>> look through code I see mkfifo call creates them. Shouldn't it always create
>> fifos and not files?
>
>
> Ok, I figured the details out.
> The actual script that I run is this radio playing command:
> stdbuf -i 0 -o 0 -e 0 mplayer
> http://kqed-ice.streamguys.org:80/kqedradio-ch-e1 | \
> stdbuf -i 0 -o 0 -e 0 tee >(grep "^XXX XXX" | prepend-time >> test.log)
>
> prepend-time is a script:
>
> gawk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'
>
> The problem only occurs with stdbuf(1) commands inserted there, and not
> without them.
> 1: bash creates a FIFO /tmp/sh-np-N
> 2: bash unlinks this FIFO
> 3: tee is passed /tmp/sh-np-N as an argument, and it creates a regular
> file with the same name
>
> This is the race condition.
> Obviously, bash doesn't ensure that unlink is issued after the child command
> opons it, and not before that. Order of operations 2 and 3 should be
> reversed.
> And just waiting for a while isn't enough! bash should only unlink the fifo
> after it has been open by the child command.
>
> Please also note, that this is on FreeBSD, which behavior might be
> different. Linux or other OSes might be getting lucky here with timing, not
> sure. FreeBSD also has its own version of stdbuf(1).
> This bug needs to be fixed.
>
> The original intention of stdbuf there was to make it all unbufferer.
> Whether this works or not isn't clear, but it doesn't matter for the purpose
> of this bug report.
>
> Suggested solution:
> On FreeBSD, for example, there is the kqueue(2) facility, which is able to
> wait on the event of "link-change": EVFILT_VNODE operation (takes file
> descriptor) allows sub-operation NOTE_LINK (The link count on the file
> changed)
> So the sequence will be like this:
> 1. bash creates FIFO
> 2. bash opens FIFO
> 3. bash waits with kqueue(2) with EVFILT_VNODE/NOTE_LINK
> 4. bash launches the child command
> 5. child command opens the FIFO as stdio
> 6. bash catches the link-change event
> 7. bash unlinks FIFO
>
> Linux and other OSes must have an equivalent facility.
> Sounds like a fun project to do. -)
> Absolute must-have if bash needs to support command substitution.
>
> Yuri
>
> CCing to the maintainer of bash on FreeBSD
