Hi Erick, thank you for your answer.
Unfortunately I do not have a heap dump from 6.6. > On 3. Sep 2018, at 20:48, Erick Erickson <erickerick...@gmail.com> wrote: > > I would expect at least 1 IndexSearcher per replica, how many total > replicas hosted in your JVM? 27 replicas per JVM. > > Plus, if you're actively indexing, there may temporarily be 2 > IndexSearchers open while the new searcher warms. > > And there may be quite a few caches, at least queryResultCache and > filterCache and documentCache, one of each per replica and maybe two > (for queryResultCache and filterCache) if you have a background > searcher autowarming. > > At a glance, your autowarm counts are very high, so it may take some > time to autowarm leading to multiple IndexSearchers and caches open > per replica when you happen to hit a commit point. I usually start > with 16-20 as an autowarm count, the benefit decreases rapidly as you > increase the count. As a counter measure I reduced the autowarm counts now per API calls to 10. Let me see if the system is now more stable. Tomorrow morning I will create a new heap dump, to see if the there are searchers. Is there any metrics which could tell me that without a heap dump? > > I'm not quite sure why it would be different in 7x .vs. 6x. How much > heap do you allocate to the JVM? And do you see similar heap dumps in > 6.6? > > Best, > Erick Thanks Erick! Björn > On Mon, Sep 3, 2018 at 10:33 AM Björn Häuser <bjoernhaeu...@gmail.com> wrote: >> >> Hello, >> >> we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard each, 4 >> replicas each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We are >> running Zookeeper 4.1.13. >> >> Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space >> exhaustion. After obtaining a heap dump it looks like that we have a lot of >> IndexSearchers open for our largest collection. >> >> The dump contains around ~60 IndexSearchers, and each containing around >> ~40mb heap. Another 500MB of heap is the fieldcache, which is expected in my >> opinion. >> >> The current config can be found here: >> https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844 >> <https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844> >> >> Analyzing the heap dump eclipse MAT says this: >> >> Problem Suspect 1 >> >> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded by >> "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy >> 1.981.148.336 (38,26%) bytes. >> >> Biggest instances: >> >> • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - 70.087.272 >> (1,35%) bytes. >> • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - 65.678.264 >> (1,27%) bytes. >> • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - 63.050.600 >> (1,22%) bytes. >> >> >> Problem Suspect 2 >> >> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded by >> "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy >> 1.373.110.208 (26,52%) bytes. >> >> >> Any help is appreciated. Thank you very much! >> Björn
