we are testing solr 4.1 running inside tomcat 7 and java 7 with following
options
JAVA_OPTS="-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump"
our source code looks like following:
/**** START *****/
int noOfSolrDocumentsInBatch = 0;
for(int i=0 ; i<5000 ; i++) {
SolrInputDocument solrInputDocument = getNextSolrInputDocument();
server.add(solrInputDocument);
noOfSolrDocumentsInBatch += 1;
if(noOfSolrDocumentsInBatch == 10) {
server.commit();
noOfSolrDocumentsInBatch = 0;
}
}
/**** END *****/
the method "getNextSolrInputDocument()" generates a solr document with 100
fields (average). Around 50 of the fields are of "text_general" type.
Some of the "test_general" fields consist of approx 1000 words rest
consists of few words. Ouf of total fields there are around 35-40
multivalued fields (not of type "text_general").
We are indexing all the fields but storing only 8 fields. Out of these 8
fields two are string type, five are long and one is boolean. So our index
size is only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB.
Why the memory is so high even though the index size is small?
What is being stored in the memory? Our understanding is that after every
commit documents are flushed to the disk.So nothing should remain in RAM
after commit.
We are using the following settings:
server.commit() set waitForSearcher=true and waitForFlush=true
solrConfig.xml has following properties set:
directoryFactory = solr.MMapDirectoryFactory
maxWarmingSearchers = 1
text_general data type is being used as supplied in the schema.xml with the
solr setup.
maxIndexingThreads = 8(default)
<autoCommit><maxTime>15000</maxTime><openSearcher>false</openSearcher></autoCommit>
We get Java heap Out Of Memory Error after commiting around 3990 solr
documents.Some of the snapshots of memory dump from profiler are uploaded
at following links.
http://s9.postimage.org/w7589t9e7/memorydump1.png
http://s7.postimage.org/p3abs6nuj/memorydump2.png
can somebody please suggest what should we do to minimize/optimize the
memory consumption in our case with the reasons?
also suggest what should be optimal values and reason for following
parameters of solrConfig.xml
useColdSearcher - true/false?
maxwarmingsearchers- number
spellcheck-on/off?
omitNorms=true/false?
omitTermFreqAndPositions?
mergefactor? we are using default value 10
java garbage collection tuning parameters ?
Regards
Harish Verma
