Hi Sandra, Not a direct answer, but if you are seeing this around merges, have you tried relaxing the merge factor to, say, 10?
Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Dec 12, 2013 at 12:10 PM, Sandra Scott <scottsandr...@gmail.com>wrote: > Helllo, > > We are experiencing unexplained OOM crashes. We have already seen it a few > times, over our different solr instances. The crash happens only at a > single shard of the collection. > > Environment details: > 1. Solr 4.3, running on tomcat. > 2. 24 Shards. > 3. Indexing rate of ~800 docs per minute. > > Solrconfig.xml: > 1. Merge factor 4 > 2. Sofrcommit every 10 min > 3. Hardcommit every 30 min > > Main findings: > 1. Solr logs: No query failures prior to the OOM, but DOUBLE the amount of > soft and hard commits in comparison to other shards. > 2. Analyzing the dump (VisualVM): Class byte[] takes 4gb out of 5gb > resourced to the JVM, mainly referenced by CompressingStoredFieldsReader GC > root (which by looking at the code, we suspect they were created due to > CompressingSortedFieldsWriter.merge). > > Sub findings: > 1. GC logs: Showed 108 GC fails prior to the crash. > 2. CPI: Overall usage seems fine, but the % of CPU time for the GC stays > high 6 min before the OOM. > 3. Memory: Half an hour before OOM the usage slowly rises, until it gets to > 5.4gb. > > Has anyone encountered higher than normal commit rate that seem to increase > merge rate and cause what I described? >
