It is btw. a Linux system and autosoftcommit is set to -1. However, indeed openSearcher is set to false. A commit is set to true after doing all the updates, but the index is not shrinking. The files are not disappearing during shutdown, but they disappear after starting up again.
On Tue, Jan 21, 2020 at 4:04 PM Jörn Franke <jornfra...@gmail.com> wrote: > thanks for the answer I will look into it - it is a possible explanation. > > > Am 20.01.2020 um 14:30 schrieb Erick Erickson <erickerick...@gmail.com>: > > > > Jörn: > > > > The only thing I can think of that _might_ cause this (I’m not all that > familiar with the code) is if your solrconfig settings never open a > searcher. Either you need to be sure openSearcher is set to true in the > autocommit section in solrconfig.xml or your autoSoftCommit is set to > something other than -1. Real Time Get requires access to all segments and > it takes a new searcher being opened to release them. Actually, a very > quick test would be to submit > “http://host:port/solr/collection/update?commit=true” > and see if the index shrinks as a result. You don’t need to change > solrconfig.xml for that test. > > > > If you are opening a new searcher, this is very concerning. There > shouldn’t be anything else you have to set to prevent the index from > growing. Could you check one thing? Compare the directory listing of the > data/index directory just before you shut down Solr and then just after. > What I’m interested in is whether some subset of files disappears when you > shut down Solr. This assumes you’re running on a *nix system, if Windows > you may have to start Solr again to see the difference. > > > > So if you open a searcher and still see the problem, I can try to > reproduce it. Can you share your solrconfig file or at least the autocommit > and cache portions? > > > > Best, > > Erick > > > >> On Jan 20, 2020, at 5:40 AM, Jörn Franke <jornfra...@gmail.com> wrote: > >> > >> From what is see it basically duplicates the index files, but does not > delete the old ones. > >> It uses caffeine cache. > >> > >> What I observe is that there is an exception when shutting down for the > collection that is updated - timeout waiting for all directory ref counts > to be released - gave up waiting on CacheDir. > >> > >>>> Am 20.01.2020 um 11:26 schrieb Jörn Franke <jornfra...@gmail.com>: > >>> > >>> Sorry I missed a line - not tlog is growing but the /data/index > folder is growing - until restart when it seems to be purged. > >>> > >>>> Am 20.01.2020 um 10:47 schrieb Jörn Franke <jornfra...@gmail.com>: > >>>> > >>>> Hi, > >>>> > >>>> I have a test system here with Solr 8.4 (but this is also > reproducible in older Solr versions), which has an index which is growing > and growing - until the SolrCloud instance is restarted - then it is > reduced tot the expected normal size. > >>>> The collection is configured to do auto commit after 15000 ms. I > expect the index grows comes due to the usage of atomic updates, but I > would expect that due to the auto commit this does not grow all the time. > >>>> After the atomic updates a commit is done in any case. > >>>> > >>>> I don’t see any error message in the log files, but the growth is > quiet significant and frequent restarts are not a solution of course. > >>>> > >>>> Maybe I am overlooking here a tiny configuration issue? > >>>> > >>>> Thank you. > >>>> > >>>> > >>>> Best regards > > >
