Re: weird bash pipe behavior
On Wed, Jan 20, 2010 at 07:34:35AM -0800, CptDpondo wrote: > mencoder 2>&1 | tr '\r' '\n' | grep -v -B 1 '^Pos' > > this eventually creates the correct output, but it's buffered until > mencoder completes. I have no idea why; the tr command streams > without buffering and I've used grep for years and never run into this > behavior. grep definitely has this buffering behavior. If you're using GNU grep, you can give it the --line-buffered option to change this, at least according to whichever contributor added that one to http://mywiki.wooledge.org/BashFAQ/009
Re: weird bash pipe behavior
Greg Wooledge wrote: On Wed, Jan 20, 2010 at 07:34:35AM -0800, CptDpondo wrote: mencoder 2>&1 | tr '\r' '\n' | grep -v -B 1 '^Pos' this eventually creates the correct output, but it's buffered until mencoder completes. I have no idea why; the tr command streams without buffering and I've used grep for years and never run into this behavior. grep definitely has this buffering behavior. If you're using GNU grep, you can give it the --line-buffered option to change this, at least according to whichever contributor added that one to http://mywiki.wooledge.org/BashFAQ/009 That is the correct solution. Apparently the behavior is not limited to grep; any process in a pipe that's longer than 2 can exhibit this. Something about libc behavior... I replaced grep with awk, ran in to the same problem, and ended up using fflush() after scanning each line. Interesting. I learned something today. -- o__ ,>/'_ o__ (_)\(_) ,>/'_ o__ A day may come (_)\(_) ,>/'_ o__ when the courage of men fails, (_)\(_) ,>/'_ when we forsake our friends and break all (_)\(_) bonds of fellowship, but it is not this day. It is not this day! This day we ride!
Re: weird bash pipe behavior
On Thu, Jan 21, 2010 at 06:11:10AM -0800, Yan Seiner wrote:
> Greg Wooledge wrote:
> >grep definitely has this buffering behavior. If you're using GNU grep,
> >you can give it the --line-buffered option to change this, at least
> >according to whichever contributor added that one to
> >http://mywiki.wooledge.org/BashFAQ/009
> >
> That is the correct solution. Apparently the behavior is not limited to
> grep; any process in a pipe that's longer than 2 can exhibit this.
> Something about libc behavior...
It's not the pipeline; it's the command that's doing the writing which
causes the undesired behavior.
Most standard Unix commands will use large buffers when writing if
their stdout is not a terminal. Compare:
$ for i in {1..10}; do sleep 1; echo $i; done | grep .
$ for i in {1..10}; do sleep 1; echo $i; done | grep . | cat
$ for i in {1..10}; do sleep 1; echo $i; done | while read x; do echo "$x";
done | cat
The first one writes a number every second. The second one writes 10
numbers after 10 seconds. The grep command buffered its output because
stdout wasn't a terminal in the second case -- hence the --line-buffered
workaround for GNU grep mentioned previously.
The third one writes a number every second despite being a 3-command
pipeline. The while/read/echo loop doesn't buffer its output the way
grep does.
