Re: "HISTSIZE=999999999" cause bash failure after lastest upgrade

[Klaas van Schelven]

Hello Chet,

I'm replying to a 2-year old bug; I think it's still relevant.

subject: "HISTSIZE=9" cause bash failure after lastest upgrade
link: https://lists.gnu.org/archive/html/bug-bash/2016-09/msg00113.html
text bellow:

> > Bash Version: 4.4
>
> > Patch Level: 0
> > Release Status: release
> >
> > Description:
> >   I use a huge value for HISTSIZE (=9) to enable infinite
> >   history items. The actural size of ~/.bash_history is only 4MB
now.
> >   Everything worked fine before the lastest upgrade. Now bash refuse
> >   me to login due to memory allocation failure. After choosing
> >   a smaller HISTSIZE, bash still eatup too much unnesssary memory.
> >
> > Repeat-By:
> >   open a workable terminal with small HISTSIZE setting,
> >   $ echo HISTSIZE=9 > ~/test.rc
> >   $ bash --rcfile ~/test.rc
> > bash: xmalloc: cannot allocate 88 bytes (114688 bytes
> > allocated)
>
> Since you've specified the desired history size, bash tries to allocate
> enough entries to hold all of the entries.  The assumption is that this
> will reduce the number of allocations/reallocations and the number of
> times the history list has to be copied as it's reallocated.  I suppose
> I should cap the maximum value for which that occurs instead of trusting
> the supplied max number of entries.
>
> Chet
>

I'm on GNU bash, version 4.4.5(1) myself.

I'm not sure the solution mentioned above is sufficient to solve all
relevant variant problems caused by the pre-allocation of the memory.

In particular, I ran into this myself with a value of HISTSIZE=1 (8
zeros), which gobbled up approximately 1GB of memory per bash instance.

Experimenting a bit led to  HISTSIZE=10 (9 zeros), which was enough
to crash and is apparently still under this cap (if that is implemented in
4.4.5(1) at all).

The fall out of this problem is potentially quite large, e.g. lockout from
remote systems if bash is the default installed shell and the only user
with access has some arbitrarily large HISTSIZE configured.

Aside from this particular scenario I would question the trade-off of
pre-allocation more generally. Besides the waste of memory, even for
non-crashing cases, there's also a performance penalty on startup (in the
1GB case mentioned above, this was visible with the naked eye).

Until not too long ago the option of setting -1 as a value for HISTSIZE was
not available (bash 4.3 introduced it) , so I imagine instances of very
large values are not entirely uncommon.

Klaas

Re: "HISTSIZE=999999999" cause bash failure after lastest upgrade

[Chet Ramey]

On 8/16/17 9:34 AM, Klaas van Schelven wrote:

> 
> I'm on GNU bash, version 4.4.5(1) myself.

And you tested with this version?
> 
> I'm not sure the solution mentioned above is sufficient to solve all
> relevant variant problems caused by the pre-allocation of the memory.

It appears to work with the current version, and should work with 4.4.5,
since it was addressed in patch 1.

$ ./bash --version
GNU bash, version 4.4.12(2)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ HISTSIZE=10 ./bash
$ echo $HISTSIZE
10


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Performance issue of find function in Gluster File System

[Zhao Li]

Hi,

I found there is a big difference of time performance between "ls" function
and "find" function in Gluster File System
.
Here is the minimal working example.

mkdir tmp
touch tmp/{000..300}.txt

time find ./ -path '*tmp*' -name '*.txt'> /dev/null
real 0m42.629s
user 0m0.675s
sys 0m1.438s

time ls tmp/*.txt > /dev/null
real 0m0.042s
user 0m0.003s
sys 0m0.003s

So I am wondering what C code you use for "ls" and "find" and how you
explain "*" in "ls" and "find" to lead to this big difference in Gluster
File System.

Thanks a lot.
Zhao

Re: Performance issue of find function in Gluster File System

[L A Walsh]

Zhao Li wrote:

So I am wondering what C code you use for "ls" and "find" and how you
explain "*" in "ls" and "find" to lead to this big difference in Gluster
File System.
  

---
   Nothing to do with Gluster, bash, ls or find, but w/the example.
I did both w/output to files in ./tmp -- you need to start
at the same place in both.

your 'find' command should be:

time find tmp/*.txt -path '*tmp*' -name '*.txt'> /dev/null

I.e.: use the same starting point: 'tmp/*.txt'

Re: Performance issue of find function in Gluster File System

[Pierre Gaston]

On Wed, Aug 16, 2017 at 11:02 PM, Zhao Li 
wrote:

> Hi,
>
> I found there is a big difference of time performance between "ls" function
> and "find" function in Gluster File System
>  ide/GlusterFS%20Introduction/>.
> Here is the minimal working example.
>
> mkdir tmp
> touch tmp/{000..300}.txt
>
> time find ./ -path '*tmp*' -name '*.txt'> /dev/null
> real 0m42.629s
> user 0m0.675s
> sys 0m1.438s
>
> time ls tmp/*.txt > /dev/null
> real 0m0.042s
> user 0m0.003s
> sys 0m0.003s
>
> So I am wondering what C code you use for "ls" and "find" and how you
> explain "*" in "ls" and "find" to lead to this big difference in Gluster
> File System.
>
> Thanks a lot.
> Zhao
>

There are several differences, first note  that "ls" is not the one finding
the files. The shell is expanding *.txt then ls is passed all the files as
arguments.
*.txt is not recursive so only the files directly under /tmp will be search

In your find command, -path matches the whole path (/ included) and your
find command will descend in all the directories, whether they match tmp or
not, so depending on where you started to search from, it may search your
whole / partition.

A more comparable command would be:

find /tmp -name tmp -o -prune -name '*.txt' -print

or if your find command supports it:

find /tmp -maxdepth 1 -name '*.txt'

Note also that ls and find are separate tools that are not developed along
with bash.

For gnu find: https://www.gnu.org/software/findutils/
For gnu ls: https://www.gnu.org/software/coreutils/coreutils.html

But there are also other implementation for various systems.