Hi All I am using Solr 4 from trunk and using it with Tomcat 6. I am noticing that when we are indexing lots of data with 16 concurrent threads, Heap grows continuously. It remains high and ultimately most of the stuff ends up being moved to Old Gen. Eventually, Old Gen also fills up and we start getting into excessive GC problem.
I took a heap dump and found that most of the memory is consumed by CloseableThreadLocal which is holding a WeakHashMap of Threads and its state. Most of the old gen is full with ThreadLocal eating up 3GB of heap and heap dump shows that all such entries are using Snowball Filter. I looked into LUCENE-3841 and verified that my version of SOLR 4 has that code. So, I am wondering the reason for this memory leak - is it due to some other bug with Solr/Lucene? Here is a brief snapshot of HeapDump showing the problem Class Name | Shallow Heap | Retained Heap ------------------------------------------------------------------------------------------------------------------------------------------------------------- *org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer @ 0x300c3eb28 | 24 | 3,885,213,072* |- <class> class org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer @ 0x2f9753340 | 0 | 0 |- this$0 org.apache.solr.schema.IndexSchema @ 0x300bf4048 | 96 | 276,704 *|- reuseStrategy org.apache.lucene.analysis.Analyzer$PerFieldReuseStrategy @ 0x300c3eb40 | 16 | 3,885,208,728* | |- <class> class org.apache.lucene.analysis.Analyzer$PerFieldReuseStrategy @ 0x2f98368c0 | 0 | 0 | |- storedValue org.apache.lucene.util.CloseableThreadLocal @ 0x300c3eb50 | 24 | 3,885,208,712 | | |- <class> class org.apache.lucene.util.CloseableThreadLocal @ 0x2f9788918 | 8 | 8 | | |- t java.lang.ThreadLocal @ 0x300c3eb68 | 16 | 16 | | | '- <class> class java.lang.ThreadLocal @ 0x2f80f0868 System Class | 8 | 24 *| | |- hardRefs java.util.WeakHashMap @ 0x300c3eb78 | 48 | 3,885,208,656* | | | |- <class> class java.util.WeakHashMap @ 0x2f8476c00 System Class | 16 | 16 | | | |- table java.util.WeakHashMap$Entry[16] @ 0x300c3eba8 | 80 | 2,200,016,960 | | | | |- <class> class java.util.WeakHashMap$Entry[] @ 0x2f84789e8 | 0 | 0 | | | | |-* [7] java.util.WeakHashMap$Entry @ 0x306a24950 | 40 | 318,502,920* | | | | | |- <class> class java.util.WeakHashMap$Entry @ 0x2f84786f8 System Class | 0 | 0 | | | | | |- queue java.lang.ref.ReferenceQueue @ 0x300c3ebf8 | 32 | 48 | | | | | |- referent java.lang.Thread @ 0x30678c2c0 web-23 | 112 | 160 | | | | | |- value java.util.HashMap @ 0x30678cbb0 | 48 | 318,502,880 | | | | | | |- <class> class java.util.HashMap @ 0x2f80b9428 System Class | 24 | 24 *| | | | | | |- table java.util.HashMap$Entry[32768] @ 0x3c07c6f58 | 131,088 | 318,502,832* | | | | | | | |- <class> class java.util.HashMap$Entry[] @ 0x2f80bd9c8 | 0 | 0 | | | | | | | |- [10457] java.util.HashMap$Entry @ 0x30678cbe0 | 32 | 40,864 | | | | | | | | |- <class> class java.util.HashMap$Entry @ 0x2f80bd400 System Class | 0 | 0 | | | | | | | | |- key java.lang.String @ 0x30678cc00 prod_desc_keywd_en_CA | 32 | 96 | | | | | | | | |- value org.apache.solr.analysis.TokenizerChain$SolrTokenStreamComponents @ 0x30678cc60 | 24 | 20,344 | | | | | | | | |- next java.util.HashMap$Entry @ 0x39a2c9100 | 32 | 20,392 | | | | | | | | | |- <class> class java.util.HashMap$Entry @ 0x2f80bd400 System Class | 0 | 0 | | | | | | | | | |- key java.lang.String @ 0x39a2c9120 3637994_fr_CA_cat_name_keywd | 32 | 104 | | | | | | | | | |- value org.apache.solr.analysis.TokenizerChain$SolrTokenStreamComponents @ 0x39a2c9188 | 24 | 20,256 | | | | | | | | | | |- <class> class org.apache.solr.analysis.TokenizerChain$SolrTokenStreamComponents @ 0x2f97a69a0| 0 | 0 | | | | | | | | | | |- this$0 org.apache.solr.analysis.TokenizerChain @ 0x300bf6158 | 32 | 13,768 | | | | | | | | | | |- source org.apache.lucene.analysis.core.WhitespaceTokenizer @ 0x39a2c91a0 | 64 | 8,304 | | | | | | | | | * | |- sink org.apache.lucene.analysis.snowball.SnowballFilter @ 0x39a2c96a8 | 48 | 10,736* ------------------------------------------------------------------------------------------------------------------------------------------------------------- Any inputs are most welcome. -Saroj
