All: Let's move the rest of the conversation over to the JIRA Tomás raised, so we have a record of what's been attempted to track this.
It would be great if Markus and Björn could add some environment info on the JIRA, in particular the version of Java you're both using and the op system etc... Thanks, Erick On Thu, Sep 6, 2018 at 1:29 AM Markus Jelsma <markus.jel...@openindex.io> wrote: > > Thanks Tomás! > > Björn, can you reproduce the problem in a local and controlled environment? > > Markus > > > > -----Original message----- > > From:Tomás Fernández Löbbe <tomasflo...@gmail.com> > > Sent: Wednesday 5th September 2018 18:32 > > To: solr-user@lucene.apache.org > > Subject: Re: Heap Memory Problem after Upgrading to 7.4.0 > > > > I think this is pretty bad. I created > > https://issues.apache.org/jira/browse/SOLR-12743. Feel free to add any more > > details you have there. > > > > On Mon, Sep 3, 2018 at 1:50 PM Markus Jelsma <markus.jel...@openindex.io> > > wrote: > > > > > Hello Björn, > > > > > > Take great care, 7.2.1 cannot read an index written by 7.4.0, so you > > > cannot roll back but need to reindex! > > > > > > Andrey Kudryavtsev made a good suggestion in the thread on how to find the > > > culprit, but it will be a tedious task. I have not yet had the time or > > > courage to venture there. > > > > > > Hope it helps, > > > Markus > > > > > > > > > > > > -----Original message----- > > > > From:Björn Häuser <bjoernhaeu...@gmail.com> > > > > Sent: Monday 3rd September 2018 22:28 > > > > To: solr-user@lucene.apache.org > > > > Subject: Re: Heap Memory Problem after Upgrading to 7.4.0 > > > > > > > > Hi Markus, > > > > > > > > this reads exactly like what we have. Where you able to figure out > > > anything? Currently thinking about rollbacking to 7.2.1. > > > > > > > > > > > > > > > > > On 3. Sep 2018, at 21:54, Markus Jelsma <markus.jel...@openindex.io> > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > > Getting an OOM plus the fact you are having a lot of IndexSearcher > > > instances rings a familiar bell. One of our collections has the same issue > > > [1] when we attempted an upgrade 7.2.1 > 7.3.0. I managed to rule out all > > > our custom Solr code but had to keep our Lucene filters in the schema, the > > > problem persisted. > > > > > > > > > > The odd thing, however, is that you appear to have the same problem, > > > but not with 7.3.0? Since you shortly after 7.3.0 upgraded to 7.4.0, can > > > you confirm the problem is not also in 7.3.0? > > > > > > > > > > > > > We had very similar problems with 7.3.0 but never analyzed them and just > > > updated to 7.4.0 because I thought thats the bug we hit: > > > https://issues.apache.org/jira/browse/SOLR-11882 < > > > https://issues.apache.org/jira/browse/SOLR-11882> > > > > > > > > > > > > > You should see the instance count for IndexSearcher increase by one > > > for each replica on each commit. > > > > > > > > > > > > Sorry, where can I find this? ;) Sorry, did not find anything. > > > > > > > > Thanks > > > > Björn > > > > > > > > > > > > > > Regards, > > > > > Markus > > > > > > > > > > [1] > > > http://lucene.472066.n3.nabble.com/RE-7-3-appears-to-leak-td4396232.html > > > > > > > > > > > > > > > > > > > > -----Original message----- > > > > >> From:Erick Erickson <erickerick...@gmail.com> > > > > >> Sent: Monday 3rd September 2018 20:49 > > > > >> To: solr-user <solr-user@lucene.apache.org> > > > > >> Subject: Re: Heap Memory Problem after Upgrading to 7.4.0 > > > > >> > > > > >> I would expect at least 1 IndexSearcher per replica, how many total > > > > >> replicas hosted in your JVM? > > > > >> > > > > >> Plus, if you're actively indexing, there may temporarily be 2 > > > > >> IndexSearchers open while the new searcher warms. > > > > >> > > > > >> And there may be quite a few caches, at least queryResultCache and > > > > >> filterCache and documentCache, one of each per replica and maybe two > > > > >> (for queryResultCache and filterCache) if you have a background > > > > >> searcher autowarming. > > > > >> > > > > >> At a glance, your autowarm counts are very high, so it may take some > > > > >> time to autowarm leading to multiple IndexSearchers and caches open > > > > >> per replica when you happen to hit a commit point. I usually start > > > > >> with 16-20 as an autowarm count, the benefit decreases rapidly as you > > > > >> increase the count. > > > > >> > > > > >> I'm not quite sure why it would be different in 7x .vs. 6x. How much > > > > >> heap do you allocate to the JVM? And do you see similar heap dumps in > > > > >> 6.6? > > > > >> > > > > >> Best, > > > > >> Erick > > > > >> On Mon, Sep 3, 2018 at 10:33 AM Björn Häuser > > > > >> <bjoernhaeu...@gmail.com> > > > wrote: > > > > >>> > > > > >>> Hello, > > > > >>> > > > > >>> we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard > > > each, 4 replicas each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We > > > are running Zookeeper 4.1.13. > > > > >>> > > > > >>> Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space > > > exhaustion. After obtaining a heap dump it looks like that we have a lot > > > of > > > IndexSearchers open for our largest collection. > > > > >>> > > > > >>> The dump contains around ~60 IndexSearchers, and each containing > > > around ~40mb heap. Another 500MB of heap is the fieldcache, which is > > > expected in my opinion. > > > > >>> > > > > >>> The current config can be found here: > > > https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844 < > > > https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844> > > > > >>> > > > > >>> Analyzing the heap dump eclipse MAT says this: > > > > >>> > > > > >>> Problem Suspect 1 > > > > >>> > > > > >>> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded > > > by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy > > > 1.981.148.336 (38,26%) bytes. > > > > >>> > > > > >>> Biggest instances: > > > > >>> > > > > >>> • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - > > > 70.087.272 (1,35%) bytes. > > > > >>> • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - > > > 65.678.264 (1,27%) bytes. > > > > >>> • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - > > > 63.050.600 (1,22%) bytes. > > > > >>> > > > > >>> > > > > >>> Problem Suspect 2 > > > > >>> > > > > >>> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded > > > by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy > > > 1.373.110.208 (26,52%) bytes. > > > > >>> > > > > >>> > > > > >>> Any help is appreciated. Thank you very much! > > > > >>> Björn > > > > >> > > > > > > > > > > > > >
