3-node SOLR 7.4.0 24gb max heap memory 13 collections, each with 500mb-2gb index (on disk)
We are investigating high heap memory usage/spikes with our SOLR cluster (details above). After rebooting the cluster, all three instances stay under 2gb for about a day. Then suddenly, one instance (srch01 in the below graph) spikes to about 7.5gb and begins a cycle of 3gb-7.5gb ups-and-downs. On this cluster, srch01 is both the overseer and the leader for all collections. A few days later, the same trend begins occurring for another node (srch02). Are there known usage patterns that would cause this kind of memory usage with SOLR? In particular, it seems odd that it would only affect the overseer/leaders node for days. Also, any tips on investigation? We haven't been able to deduce much from visualvm profiling. Additional context. For years, we set max heap memory to 4gb. But our SOLR instances recently began to OOM. Increasing to 8gb helped, but the OOMs still eventually occurred. This is how we eventually set it to 24gb (following SOLR documentation saying 10-20gb was not uncommon for production instances). But the recent change is what makes us suspicious that some client usage pattern is the root cause. <https://lucene.472066.n3.nabble.com/file/t494250/ue1_10-4_to_10-7.jpg> -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
