Helllo, We are experiencing unexplained OOM crashes. We have already seen it a few times, over our different solr instances. The crash happens only at a single shard of the collection.
Environment details: 1. Solr 4.3, running on tomcat. 2. 24 Shards. 3. Indexing rate of ~800 docs per minute. Solrconfig.xml: 1. Merge factor 4 2. Sofrcommit every 10 min 3. Hardcommit every 30 min Main findings: 1. Solr logs: No query failures prior to the OOM, but DOUBLE the amount of soft and hard commits in comparison to other shards. 2. Analyzing the dump (VisualVM): Class byte[] takes 4gb out of 5gb resourced to the JVM, mainly referenced by CompressingStoredFieldsReader GC root (which by looking at the code, we suspect they were created due to CompressingSortedFieldsWriter.merge). Sub findings: 1. GC logs: Showed 108 GC fails prior to the crash. 2. CPI: Overall usage seems fine, but the % of CPU time for the GC stays high 6 min before the OOM. 3. Memory: Half an hour before OOM the usage slowly rises, until it gets to 5.4gb. Has anyone encountered higher than normal commit rate that seem to increase merge rate and cause what I described?
