Re: Bash 5 increase in RAM from loop with many interations that append to file
> I tried to bisect but I'm not sure the result is useful. I got the following: > > d233b485e83c3a784b803fb894280773f16f2deb is the first bad commit > commit d233b485e83c3a784b803fb894280773f16f2deb > Author: Chet Ramey > Date: Mon Jan 7 09:27:52 2019 -0500 > > bash-5.0 distribution sources and documentation FWIW, on which branch did you performed bisect? The above commit is from the branch of distribution. You can perform bisect against `devel' branch, for example, between 118fb670..5cc55f2f. The Bash source is not really managed by the version control system in a usual sense, but the repository is just a sequence of snapshots. So, you can only narrow down to a snapshot of some day, but it will become easier to identify the corresponding changesets. -- Koichi
Re: Behavior change
On 10/26/20 5:58 PM, Gregory Heytings via Bug reports for the GNU Bourne Again SHell wrote: > > Hi, > > I just installed Bash 5.1 rc1. I have IGNOREEOF set, and in 5.0 (and > earlier version) when pressing C-d one sees: > > $ Use "logout" to leave the shell. > > With 5.1 rc1 there is a newline between the prompt and the message: > > $ > Use "logout" to leave the shell. > > Is this change intentional? It's bracketed paste, which is enabled by default in bash-5.1-rc1. The escape sequence that readline sends to the terminal to turn off bracketed paste ends with a carriage return, to avoid confusing the Linux terminal driver about the physical location of the cursor, so the cursor ends up at column 0. If readline has seen EOF, the shell is going to print something one way or another, and that text will overwrite what is on the current screen line. Readline outputs a newline to compensate. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903936 for an example. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Behavior change
Hi Chet, Thanks for your reply! I just installed Bash 5.1 rc1. I have IGNOREEOF set, and in 5.0 (and earlier version) when pressing C-d one sees: $ Use "logout" to leave the shell. With 5.1 rc1 there is a newline between the prompt and the message: $ Use "logout" to leave the shell. Is this change intentional? It's bracketed paste, which is enabled by default in bash-5.1-rc1. The escape sequence that readline sends to the terminal to turn off bracketed paste ends with a carriage return, to avoid confusing the Linux terminal driver about the physical location of the cursor, so the cursor ends up at column 0. If readline has seen EOF, the shell is going to print something one way or another, and that text will overwrite what is on the current screen line. Readline outputs a newline to compensate. Okay. I just set "set enable-bracketed-paste off" in my /etc/inputrc. Perhaps this change should be documented in CHANGES, and in the man page (where enable-bracketed-paste is said to have the default value Off). Gregory
Re: Bash 5 increase in RAM from loop with many interations that append to file
On 10/27/20 12:11 AM, Scott Kostyshak wrote:
> The following example uses more peak RAM on new bash versions than old
> versions:
>
> for i in {1..100}; do
> echo "${i}" >> example.txt
> done
>
> By measuring peak memory usage with time (/usr/bin/time -f "%E %P %M"),
> I get that newer versions of Bash use about 284M, where older versions
> use about 191M.
>
> Is this perceived increase in memory usage worth looking more into or is
> it intended?
My guess is that the huge list that results from the brace expansion caused
the bash malloc to cross over into a larger memory bin because of unrelated
memory allocation patterns that changed between bash versions. Once you
allocate that much memory, a binary bin allocation method is going to waste
some. In bash-5.0, that large allocation is going to use mmap, which adds
some overhead of its own.
If you can use valgrind or a similar tool to identify a memory leak, that
would be useful.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Behavior change
On 10/27/20 9:55 AM, Gregory Heytings wrote: > > Okay. I just set "set enable-bracketed-paste off" in my /etc/inputrc. > Perhaps this change should be documented in CHANGES, and in the man page > (where enable-bracketed-paste is said to have the default value Off). The manual page in the released version will reflect the default setting of bracketed paste. While it will probably remain on by default, there have already been several reports of the bracketed paste enable/disable escape sequences interfering with program-driven readline use cases, so it's not certain it will. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Behavior change
Okay. I just set "set enable-bracketed-paste off" in my /etc/inputrc. Perhaps this change should be documented in CHANGES, and in the man page (where enable-bracketed-paste is said to have the default value Off). The manual page in the released version will reflect the default setting of bracketed paste. While it will probably remain on by default, there have already been several reports of the bracketed paste enable/disable escape sequences interfering with program-driven readline use cases, so it's not certain it will. FWIW, I've been using bash for twenty years (starting with version 2.05 IIRC), and it's the first time I see changes that I find disturbing and want to turn off. The paste behavior with enable-bracketed-paste on also disturbed me, but I wasn't sure how to explain it. ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Nice mottoes :-)
Re: Behavior change
On 10/27/20 10:10 AM, Gregory Heytings wrote: > >>> Okay. I just set "set enable-bracketed-paste off" in my /etc/inputrc. >>> Perhaps this change should be documented in CHANGES, and in the man page >>> (where enable-bracketed-paste is said to have the default value Off). >> >> The manual page in the released version will reflect the default setting >> of bracketed paste. While it will probably remain on by default, there >> have already been several reports of the bracketed paste enable/disable >> escape sequences interfering with program-driven readline use cases, so >> it's not certain it will. >> > > FWIW, I've been using bash for twenty years (starting with version 2.05 > IIRC), and it's the first time I see changes that I find disturbing and > want to turn off. The paste behavior with enable-bracketed-paste on also > disturbed me, but I wasn't sure how to explain it. I understand. There are folks on the other side of that discussion who are just as passionate in their belief that the default bash behavior is broken and leads to security issues. Bracketed paste is currently the only way to determine whether or not input comes from a paste or some other source. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Bash 5 increase in RAM from loop with many interations that append to file
On Tue, Oct 27, 2020 at 08:54:37PM +0900, Koichi Murase wrote: > > I tried to bisect but I'm not sure the result is useful. I got the > > following: > > > > d233b485e83c3a784b803fb894280773f16f2deb is the first bad commit > > commit d233b485e83c3a784b803fb894280773f16f2deb > > Author: Chet Ramey > > Date: Mon Jan 7 09:27:52 2019 -0500 > > > > bash-5.0 distribution sources and documentation > > FWIW, on which branch did you performed bisect? The above commit is > from the branch of distribution. You can perform bisect against > `devel' branch, for example, between 118fb670..5cc55f2f. > > The Bash source is not really managed by the version control system in > a usual sense, but the repository is just a sequence of snapshots. So, > you can only narrow down to a snapshot of some day, but it will become > easier to identify the corresponding changesets. Thank you for the explanation, Koichi. I thought I had bisected on devel but perhaps I did something wrong. I can attempt another bisect at some point in the future, but these days it is very difficult for me to find time. Thanks for your quick reply! Scott
Re: Bash 5 increase in RAM from loop with many interations that append to file
On Tue, Oct 27, 2020 at 09:40:53AM -0400, Chet Ramey wrote:
> On 10/27/20 12:11 AM, Scott Kostyshak wrote:
> > The following example uses more peak RAM on new bash versions than old
> > versions:
> >
> > for i in {1..100}; do
> > echo "${i}" >> example.txt
> > done
> >
> > By measuring peak memory usage with time (/usr/bin/time -f "%E %P %M"),
> > I get that newer versions of Bash use about 284M, where older versions
> > use about 191M.
> >
> > Is this perceived increase in memory usage worth looking more into or is
> > it intended?
>
> My guess is that the huge list that results from the brace expansion caused
> the bash malloc to cross over into a larger memory bin because of unrelated
> memory allocation patterns that changed between bash versions. Once you
> allocate that much memory, a binary bin allocation method is going to waste
> some. In bash-5.0, that large allocation is going to use mmap, which adds
> some overhead of its own.
>
> If you can use valgrind or a similar tool to identify a memory leak, that
> would be useful.
Thanks for the explanation, Chet. I would normally jump at the chance to help
out because I enjoy digging into things like this, but these days I just can't
find the time. I will come back to this at some point, but possibly not for a
while. I'm sorry about that.
Thanks again for all of your work on Bash!
Scott
