Coprocess terminated, pipe closed before I read the data out
Configuration Information
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib -
D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -Wall
uname output: Linux adamant 4.0.0-2-amd64 #1 SMP Debian 4.0.8-2 (2015-
07-22) x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu
Bash Version: 4.3
Patch Level: 42
Release Status: release
Description:
Coprocess's output pipe is destroyed when coprocess terminates
(possibly before coprocess output has been read)
I realize this is by design, essentially, as Bash's coprocs are meant
to be "self-cleaning" - but in practice I think this is a bad policy,
as variables and file descriptors used by the script are abruptly
removed from the environment.
Repeat-By:
$ coproc head { head -n 10; }
[1] 1864
$ ls >&${head[1]} &
# "head" gets its 10 lines, writes them to output pipe, and terminates.
$ read -r first_line <&${head[0]}
[1]- Done coproc head { head -n 10; }
$ read -r second_line <&${head[0]}
# ${head[@]} is already unset, and pipes closed, so...
-bash: ${head[0]}: ambiguous redirect
Fix:
Perhaps the simplest fix is just don't close the pipes or unset the
variable. Leave that up to the caller.
Another option might be to use poll() to check if the pipe is empty
prior to closing it:
poll_fd.events = (POLLHUP | POLLIN);
int poll_result = poll(&poll_fd, 1, 0);
// Then don't close the file descriptor until
// (!(poll_fd.revents & POLLIN) && (poll_fd.revents & POLLHUP)).
But this is still problematic: Suppose the shell script is running a
loop, processing lines of text from the coproc:
$ while read line <&${coproc[0]}; do cmd $line; done
Even if the pipe isn't closed until all the data has been read out of
it, the loop may (depending on timing) wind up terminating with an
"ambiguous redirect" error (or "unbound variable" if "nounset" is in
effect), when it should have ended happily with an EOF condition.
Users can work around this by duplicating the file descriptor:
$ exec {fd_that_wont_vanish_on_me}<&${coproc[0]}-
But it kind of negates the benefit of having coproc accept a name for
the fd array if you just wind up having to re-bind it anyway. And the
command could still fail if it's not run immediately after launching
the coproc.
Thus, I think coproc shouldn't close its file descriptors or erase its
environment variables.
---GEC
Memory leak in hc_erasedups
Hi all,
I found memory leak in case of using "HISTCONTOL=erasedups" in bash-4.2.
- bash version
CentOS7: bash-4.2.45-5.el7_0.4.x86_64
$ echo $BASH_VERSION
4.2.45(1)-release
- How to reproduce memory leak
1. add ~/.bashrc
export HISTCONTROL=erasedups
export HISTSIZE=10
export PROMPT_COMMAND="history -a; history -r"
2. login from other terminal(e.g. ssh or telnet)
3. command execute repeatedly (e.g. echo "a"; echo "b")
=> When terminal macro is used, it's easy to reproduce.
Following is a python script for localhost.
Please change 'pass' to right password in your environment.
$ cat /tmp/bash-test.py
#!/usr/bin/python
import pexpect
p = pexpect.spawn ('ssh localhost')
p.expect("password:")
p.sendline('pass')
p.expect('\$ ')
while 1:
p.sendline('echo a')
p.expect('\$ ')
p.sendline('echo b')
p.expect('\$ ')
p.close()
- Result
By executing 'bash-test.py', you can see a following anon page is increased.
$ pmap -p 26941 | head -n 5; sleep 60; pmap -p 26941 | head -n 5
26941: -bash
0040884K r-x-- /usr/bin/bash
006dc000 4K r /usr/bin/bash
006dd000 36K rw--- /usr/bin/bash
006e6000 2468K rw--- [ anon ]
26941: -bash
0040884K r-x-- /usr/bin/bash
006dc000 4K r /usr/bin/bash
006dd000 36K rw--- /usr/bin/bash
006e6000 2700K rw--- [ anon ]
- Proposal patch
Attached is the testing proposal patch.
---
bashhist.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/bashhist.c b/bashhist.c
index 7240a5b..9bf3b5c 100644
--- a/bashhist.c
+++ b/bashhist.c
@@ -625,6 +625,7 @@ hc_erasedups (line)
char *line;
{
HIST_ENTRY *temp;
+ HIST_ENTRY *discard;
int r;
using_history ();
@@ -633,7 +634,9 @@ hc_erasedups (line)
if (STREQ (temp->line, line))
{
r = where_history ();
- remove_history (r);
+ discard = remove_history (r);
+ if (discard)
+ free_history_entry (discard);
}
}
using_history ();
--
1.8.3.1
I have tested at bash-4.2 branch.
url = http://git.savannah.gnu.org/r/bash.git
Best regards,
Seiichi Ishitsuka
bash "while do echo" can't function correctly
Hi, I use a very simple statement "cat test.txt | while read line; do echo $line; done", test.txt is very small, no more than 100 lines. but the execution of the statement paused during process. test.txt is pasted on http://paste.bradleygill.com/index.php?paste_id=1647399 desmond.he@xgimi-dev:/studio/desmond.he$ help | head -n 1 GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu) desmond.he@xgimi-dev:/studio/desmond.he$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS" Thanks for help~
Re: bash "while do echo" can't function correctly
On Wed, Apr 13, 2016 at 10:01:23AM +0800, 何建军 wrote: > Hi, I use a very simple statement "cat test.txt | while read line; do echo > $line; done", test.txt is very small, no more than 100 lines. but the > execution of the statement paused during process. test.txt is pasted on > http://paste.bradleygill.com/index.php?paste_id=1647399 > desmond.he@xgimi-dev:/studio/desmond.he$ help | head -n 1 GNU bash, version > 4.3.11(1)-release (x86_64-pc-linux-gnu) > desmond.he@xgimi-dev:/studio/desmond.he$ cat /etc/lsb-release > DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty > DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS" Thanks for help~ You didn't quote the expansion of the line variable, so the result of the expansion is first split into words based on the characters in the special IFS variable, then each of those words that contain glob characters (like *, ? , [...]) will be replaced by matching filenames, if any. It's the last part that bites you here, since several of your lines contain * characters. Worse if globstar is enabled, since there's also ** in there, so the "pause" you get is probably bash trying to recurse through the entire filesystem, which may take a while. The fix is easy. Just surround the expansion in double quotes, which prevent word-splitting and pathname expansion. while read -r line; do echo "$line"; done < test.txt though printf should be preferred over echo: while read -r line; do printf '%s\n' "$line"; done < test.txt -- Geir Hauge
