Re: Memory leak in wait
Hi Chet, Thanks for the fast reply and the explanations, very appreciated. On Thu, 06 Nov 2014 20:57:11 -0500, Chet Ramey wrote: > On 11/6/14 8:09 AM, Jean Delvare wrote: > > A memory leak has been reported in a bash script I maintain [1]. After > > investigation, I was able to shrink the test case down to: > > > > while true > > do > > sleep 1 & > > wait $! > > done > > This isn't a memory leak, and the memory use is bounded. The shell, as OK, this is in line with valgrind's claim that all allocated memory was still reachable. But how bounded is it? The shell script for which the issue was originally reported starts with about 3 MB of memory but reaches 32 MB over time (I don't know which version of bash did that though.) That seems to be a lot of memory just to record process exit statuses. > per Posix, keeps the statues of the last child_max processes. It gets > child_max from the process's resource limit (ulimit -v, 709 (?) on my > system). The list is FIFO, so when the number of background statuses > equals child_max, the oldest statuses are discarded. "help ulimit" says: -vthe size of virtual memory the scope of which seems to exceed just the number of child process statuses to remember? > Posix allows the shell to discard saved status values if $! isn't > referenced before another async job is started, but bash doesn't try > to do that. > > > The above loop has an always-rising memory consumption (RSS value as > > reported by ps -o rss.) This one OTOH: > > > > while true > > do > > sleep 1 & > > wait > > done > > When you wait for all background processes, the shell, as per Posix, > discards all saved statuses. OK, thanks for the information. So I think in the case of a shell script daemon, I have two options: 1* Use "wait" without argument. As we are always waiting for a single process (sleep) to finish, I suppose this is equivalent to "wait $?", right? 2* Limit the value child_max. But using "ulimit -v" for that seems to be a rather violent and risky way to achieve this? Also three questions remain, if you would be kind enough to answer them. 1* Why is the memory consumption steady when using bash 3.0.16? Was this version of bash not recording process exit statuses yet? 2* If bash remembers the process statuses, how does one access the information? I couldn't find anything related to that in the manual page, but it's huge so maybe I missed it. 3* If I never call "wait" and thus never free the recorded exit statuses, does it mean that the system can't recycle the process IDs in question until the shell script exits? That would be a major flaw in my script :-( Thanks again, -- Jean Delvare SUSE L3 Support
Re: PROMPT_COMMAND='history -a; history -n' causes shell hang in OX 10.10 Yosemite / bash 3.2.53
On Thu, Nov 6, 2014 at 3:15 PM, Chet Ramey wrote: > > There is no "standard ipc" except pipes across the range of systems bash > runs on. thanks. good to know, before I open the flood gate of git commits ;-) can I assume that this includes named pipes? cheers, pg
Re: PROMPT_COMMAND='history -a; history -n' causes shell hang in OX 10.10 Yosemite / bash 3.2.53
On 11/07/2014 12:23 PM, Piotr Grzybowski wrote: > On Thu, Nov 6, 2014 at 3:15 PM, Chet Ramey wrote: >> >> There is no "standard ipc" except pipes across the range of systems bash >> runs on. > > thanks. good to know, before I open the flood gate of git commits ;-) > can I assume that this includes named pipes? Named pipes (aka fifos) are notorious for being buggy on Cygwin and non-available on mingw. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: [PATCH] Support configurable mode strings in prompt
On Thursday, September 4, 2014 5:22:58 AM UTC-6, Dylan Cali wrote: > Hi Chet, > > Just curious if it will be possible to merge this enhancement? > > Thanks, > Dylan This is what it looks like right now. It's still really helpful as is, but it doesn't really fit in with the theme I was going for :) http://i.imgur.com/xXpOJqe.png
Re: [PATCH] Support configurable mode strings in prompt
On Thursday, September 4, 2014 7:00:16 AM UTC-6, Chet Ramey wrote: > On 9/4/14, 7:22 AM, Dylan Cali wrote: > > Hi Chet, > > > > Just curious if it will be possible to merge this enhancement? > > I haven't looked at it yet. > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer >``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/ I hope this gets merged in soon. I love the bash vi mode and I love the realtime mode indicator that we now have with show-mode-in-prompt. If I could customize that portion of my command line I could add that portion of the prompt in a much more stylized way. If I could make the color change and make it say "insert", "visual", and "normal" as the mode changes, that would be pretty cool.
Re: PROMPT_COMMAND='history -a; history -n' causes shell hang in OX 10.10 Yosemite / bash 3.2.53
On Fri, Nov 7, 2014 at 1:26 PM, Eric Blake wrote: > > Named pipes (aka fifos) are notorious for being buggy on Cygwin and > non-available on mingw. thanks Eric. good news. so this means, that shared history can only be implemented using regular files. is that right? cheers, pg
Re: PROMPT_COMMAND='history -a; history -n' causes shell hang in OX 10.10 Yosemite / bash 3.2.53
Guys thanks so much for taking this discussion further than just the reported bug that prevented the existing workaround. I’ve often considered putting this work in myself but wasn’t sure what the reason was for this not being in bash already and if there was an architectural difficulty with implementing it. It sounds more like that it just wasn’t considered as an option rather than there is a problem implementing it. Given that ksh only ever worked this way (but its shared history performed flawlessly)l, I would think that the mechanism they used and particularly how they handled the locking is a good place to start with. GJ > On 7 Nov 2014, at 11:29 pm, Piotr Grzybowski wrote: > > On Fri, Nov 7, 2014 at 1:26 PM, Eric Blake wrote: >> >> Named pipes (aka fifos) are notorious for being buggy on Cygwin and >> non-available on mingw. > > thanks Eric. good news. > so this means, that shared history can only be implemented using > regular files. is that right? > > cheers, > pg
Re: [PATCH] Support configurable mode strings in prompt
On Thu, Nov 6, 2014 at 11:53 PM, wrote: > I hope this gets merged in soon. I love the bash vi mode and I love the > realtime mode indicator that we now have with show-mode-in-prompt. If I could > customize that portion of my command line I could add that portion of the > prompt in a much more stylized way. If I could make the color change and make > it say "insert", "visual", and "normal" as the mode changes, that would be > pretty cool. As an FYI I've been using this patch on both my work and home boxes for several months now without issue.. which isn't saying there aren't any, but that includes Debian, Arch, and Centos. Thanks, Dylan
Re: Memory leak in wait
On 11/7/14, 3:49 AM, Jean Delvare wrote: > Hi Chet, > > Thanks for the fast reply and the explanations, very appreciated. > > On Thu, 06 Nov 2014 20:57:11 -0500, Chet Ramey wrote: >> On 11/6/14 8:09 AM, Jean Delvare wrote: >>> A memory leak has been reported in a bash script I maintain [1]. After >>> investigation, I was able to shrink the test case down to: >>> >>> while true >>> do >>> sleep 1 & >>> wait $! >>> done >> >> This isn't a memory leak, and the memory use is bounded. The shell, as > > OK, this is in line with valgrind's claim that all allocated memory was > still reachable. But how bounded is it? The shell script for which the > issue was originally reported starts with about 3 MB of memory but > reaches 32 MB over time (I don't know which version of bash did that > though.) That seems to be a lot of memory just to record process exit > statuses. Maybe. That depends on the number of children your maxproc resource says you can create. On my system, the struct that holds a pid's status is 16 bytes. >> per Posix, keeps the statues of the last child_max processes. It gets >> child_max from the process's resource limit (ulimit -v, 709 (?) on my >> system). The list is FIFO, so when the number of background statuses >> equals child_max, the oldest statuses are discarded. > > "help ulimit" says: > > -v the size of virtual memory > > the scope of which seems to exceed just the number of child process > statuses to remember? Sorry, it's `ulimit -u'. > So I think in the case of a shell script daemon, I have two options: > 1* Use "wait" without argument. As we are always waiting for a single >process (sleep) to finish, I suppose this is equivalent to "wait >$?", right? Yes, as long as you mean $!. > 2* Limit the value child_max. But using "ulimit -v" for that seems to >be a rather violent and risky way to achieve this? -u the maximum number of user processes > Also three questions remain, if you would be kind enough to answer them. > > 1* Why is the memory consumption steady when using bash 3.0.16? Was >this version of bash not recording process exit statuses yet? That feature came in in bash-3.1. > 2* If bash remembers the process statuses, how does one access the >information? I couldn't find anything related to that in the manual >page, but it's huge so maybe I missed it. You can use wait with a pid argument, maybe one that you saved earlier in a script or obtained from `ps'. > 3* If I never call "wait" and thus never free the recorded exit >statuses, does it mean that the system can't recycle the process IDs >in question until the shell script exits? That would be a major flaw >in my script :-( No, the system will recycle the process ID as soon as bash reaps it with waitpid(2). It's bash that is responsible for making sure to discard a saved status if fork(2) returns a PID that's been saved. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: PROMPT_COMMAND='history -a; history -n' causes shell hang in OX 10.10 Yosemite / bash 3.2.53
Good point Graham, definitely good place to start. If you are interested we can discuss further, privately. Mainly because I want to say some nasty things about implementing sharing and locking resources withouth ipc. :)) cheers, pg On Fri, Nov 7, 2014 at 1:38 PM, Graham Jones wrote: > Guys thanks so much for taking this discussion further than just the reported > bug that prevented the existing workaround. > I’ve often considered putting this work in myself but wasn’t sure what the > reason was for this not being in bash already and if there was an > architectural difficulty with implementing it. It sounds more like that it > just wasn’t considered as an option rather than there is a problem > implementing it. > > Given that ksh only ever worked this way (but its shared history performed > flawlessly)l, I would think that the mechanism they used and particularly how > they handled the locking is a good place to start with. > > GJ > >> On 7 Nov 2014, at 11:29 pm, Piotr Grzybowski wrote: >> >> On Fri, Nov 7, 2014 at 1:26 PM, Eric Blake wrote: >>> >>> Named pipes (aka fifos) are notorious for being buggy on Cygwin and >>> non-available on mingw. >> >> thanks Eric. good news. >> so this means, that shared history can only be implemented using >> regular files. is that right? >> >> cheers, >> pg >
bash version: 4.3.27(1) and .bashrc / .bash_profile
Hi all! I patched 4 servers some weeks ago ( bash version: 4.3.27(1) ) following the https://shellshocker.net/ guide (which it's a simple easy way for downloading https://ftp.gnu.org/gnu/bash/bash-4.3.tar.gz plus patches and compile it) Today I realized some processes were not working as expected. After some investigation I've realized that when executing commands as: su - [user] -c [command] neither .bashrc o .bash_profile on user home directory are executed. If I just execute su - [user] everything it's fine. I've changed the user shell on /etc/passwd to the previous ( original vulnerable) bash and my scripts are back to work. Servers are Fedora 6 (I know, a bit old... bus stable :-), CentOS and Oracle Linux. All of them RedHat based, just in case it matters. Should I take this as bug?. Or it's a side effect?. A fault in my side for some compilation option? Thank you in advanced,
