I think this is pretty bad. I created https://issues.apache.org/jira/browse/SOLR-12743. Feel free to add any more details you have there.
On Mon, Sep 3, 2018 at 1:50 PM Markus Jelsma <markus.jel...@openindex.io> wrote: > Hello Björn, > > Take great care, 7.2.1 cannot read an index written by 7.4.0, so you > cannot roll back but need to reindex! > > Andrey Kudryavtsev made a good suggestion in the thread on how to find the > culprit, but it will be a tedious task. I have not yet had the time or > courage to venture there. > > Hope it helps, > Markus > > > > -----Original message----- > > From:Björn Häuser <bjoernhaeu...@gmail.com> > > Sent: Monday 3rd September 2018 22:28 > > To: solr-user@lucene.apache.org > > Subject: Re: Heap Memory Problem after Upgrading to 7.4.0 > > > > Hi Markus, > > > > this reads exactly like what we have. Where you able to figure out > anything? Currently thinking about rollbacking to 7.2.1. > > > > > > > > > On 3. Sep 2018, at 21:54, Markus Jelsma <markus.jel...@openindex.io> > wrote: > > > > > > Hello, > > > > > > Getting an OOM plus the fact you are having a lot of IndexSearcher > instances rings a familiar bell. One of our collections has the same issue > [1] when we attempted an upgrade 7.2.1 > 7.3.0. I managed to rule out all > our custom Solr code but had to keep our Lucene filters in the schema, the > problem persisted. > > > > > > The odd thing, however, is that you appear to have the same problem, > but not with 7.3.0? Since you shortly after 7.3.0 upgraded to 7.4.0, can > you confirm the problem is not also in 7.3.0? > > > > > > > We had very similar problems with 7.3.0 but never analyzed them and just > updated to 7.4.0 because I thought thats the bug we hit: > https://issues.apache.org/jira/browse/SOLR-11882 < > https://issues.apache.org/jira/browse/SOLR-11882> > > > > > > > You should see the instance count for IndexSearcher increase by one > for each replica on each commit. > > > > > > Sorry, where can I find this? ;) Sorry, did not find anything. > > > > Thanks > > Björn > > > > > > > > Regards, > > > Markus > > > > > > [1] > http://lucene.472066.n3.nabble.com/RE-7-3-appears-to-leak-td4396232.html > > > > > > > > > > > > -----Original message----- > > >> From:Erick Erickson <erickerick...@gmail.com> > > >> Sent: Monday 3rd September 2018 20:49 > > >> To: solr-user <solr-user@lucene.apache.org> > > >> Subject: Re: Heap Memory Problem after Upgrading to 7.4.0 > > >> > > >> I would expect at least 1 IndexSearcher per replica, how many total > > >> replicas hosted in your JVM? > > >> > > >> Plus, if you're actively indexing, there may temporarily be 2 > > >> IndexSearchers open while the new searcher warms. > > >> > > >> And there may be quite a few caches, at least queryResultCache and > > >> filterCache and documentCache, one of each per replica and maybe two > > >> (for queryResultCache and filterCache) if you have a background > > >> searcher autowarming. > > >> > > >> At a glance, your autowarm counts are very high, so it may take some > > >> time to autowarm leading to multiple IndexSearchers and caches open > > >> per replica when you happen to hit a commit point. I usually start > > >> with 16-20 as an autowarm count, the benefit decreases rapidly as you > > >> increase the count. > > >> > > >> I'm not quite sure why it would be different in 7x .vs. 6x. How much > > >> heap do you allocate to the JVM? And do you see similar heap dumps in > > >> 6.6? > > >> > > >> Best, > > >> Erick > > >> On Mon, Sep 3, 2018 at 10:33 AM Björn Häuser <bjoernhaeu...@gmail.com> > wrote: > > >>> > > >>> Hello, > > >>> > > >>> we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard > each, 4 replicas each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We > are running Zookeeper 4.1.13. > > >>> > > >>> Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space > exhaustion. After obtaining a heap dump it looks like that we have a lot of > IndexSearchers open for our largest collection. > > >>> > > >>> The dump contains around ~60 IndexSearchers, and each containing > around ~40mb heap. Another 500MB of heap is the fieldcache, which is > expected in my opinion. > > >>> > > >>> The current config can be found here: > https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844 < > https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844> > > >>> > > >>> Analyzing the heap dump eclipse MAT says this: > > >>> > > >>> Problem Suspect 1 > > >>> > > >>> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded > by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy > 1.981.148.336 (38,26%) bytes. > > >>> > > >>> Biggest instances: > > >>> > > >>> • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - > 70.087.272 (1,35%) bytes. > > >>> • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - > 65.678.264 (1,27%) bytes. > > >>> • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - > 63.050.600 (1,22%) bytes. > > >>> > > >>> > > >>> Problem Suspect 2 > > >>> > > >>> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded > by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy > 1.373.110.208 (26,52%) bytes. > > >>> > > >>> > > >>> Any help is appreciated. Thank you very much! > > >>> Björn > > >> > > > > >
