Hello, we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard each, 4 replicas each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We are running Zookeeper 4.1.13.
Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space exhaustion. After obtaining a heap dump it looks like that we have a lot of IndexSearchers open for our largest collection. The dump contains around ~60 IndexSearchers, and each containing around ~40mb heap. Another 500MB of heap is the fieldcache, which is expected in my opinion. The current config can be found here: https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844 <https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844> Analyzing the heap dump eclipse MAT says this: Problem Suspect 1 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy 1.981.148.336 (38,26%) bytes. Biggest instances: • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - 70.087.272 (1,35%) bytes. • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - 65.678.264 (1,27%) bytes. • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - 63.050.600 (1,22%) bytes. Problem Suspect 2 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy 1.373.110.208 (26,52%) bytes. Any help is appreciated. Thank you very much! Björn
