You should actually decrease solr heap size. Let me explain a bit. Solr requires very less heap memory for its operation and more memory for storing data in main memory. This is because solr uses mmap for storing the index files. Please check the link http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html for understanding how solr operates on files .
Solr has typical problem of Garbage collection once you the heap size to a large value. It will have indeterminate pauses due to GC. The amount of heap memory required is difficult to tell. However the way we tuned this parameter is setting it to a low value and increasing it by 1Gb whenever OOM is thrown. Please check the problem of having large Java Heap http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap Just for your reference, in our production setup, we have data of around 60Gb/node spread across 25 collections. We have configured 8GB as heap and the rest of the memory we will leave it to OS to manage. We do around 1000 (search + Insert)/second on the data. I hope this helps. Regards, Rahul On Tue, Dec 15, 2015 at 4:33 PM, zhenglingyun <konghuaru...@163.com> wrote: > Hi, list > > I’m new to solr. Recently I encounter a “memory leak” problem with > solrcloud. > > I have two 64GB servers running a solrcloud cluster. In the solrcloud, I > have > one collection with about 400k docs. The index size of the collection is > about > 500MB. Memory for solr is 16GB. > > Following is "ps aux | grep solr” : > > /usr/java/jdk1.7.0_67-cloudera/bin/java > -Djava.util.logging.config.file=/var/lib/solr/tomcat-deployment/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.net.preferIPv4Stack=true -Dsolr.hdfs.blockcache.enabled=true > -Dsolr.hdfs.blockcache.direct.memory.allocation=true > -Dsolr.hdfs.blockcache.blocksperbank=16384 > -Dsolr.hdfs.blockcache.slab.count=1 -Xms16608395264 -Xmx16608395264 > -XX:MaxDirectMemorySize=21590179840 -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled > -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC > -Xloggc:/var/log/solr/gc.log > -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -DzkHost= > bjzw-datacenter-hadoop-160.d.yourmall.cc:2181, > bjzw-datacenter-hadoop-163.d.yourmall.cc:2181, > bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr > -Dsolr.solrxml.location=zookeeper -Dsolr.hdfs.home=hdfs://datacenter/solr > -Dsolr.hdfs.confdir=/var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/hadoop-conf > -Dsolr.authentication.simple.anonymous.allowed=true > -Dsolr.security.proxyuser.hue.hosts=* > -Dsolr.security.proxyuser.hue.groups=* -Dhost= > bjzw-datacenter-solr-15.d.yourmall.cc -Djetty.port=8983 -Dsolr.host= > bjzw-datacenter-solr-15.d.yourmall.cc -Dsolr.port=8983 > -Dlog4j.configuration=file:///var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/log4j.properties > -Dsolr.log=/var/log/solr -Dsolr.admin.port=8984 > -Dsolr.max.connector.thread=10000 -Dsolr.solr.home=/var/lib/solr > -Djava.net.preferIPv4Stack=true -Dsolr.hdfs.blockcache.enabled=true > -Dsolr.hdfs.blockcache.direct.memory.allocation=true > -Dsolr.hdfs.blockcache.blocksperbank=16384 > -Dsolr.hdfs.blockcache.slab.count=1 -Xms16608395264 -Xmx16608395264 > -XX:MaxDirectMemorySize=21590179840 -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled > -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC > -Xloggc:/var/log/solr/gc.log > -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -DzkHost= > bjzw-datacenter-hadoop-160.d.yourmall.cc:2181, > bjzw-datacenter-hadoop-163.d.yourmall.cc:2181, > bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr > -Dsolr.solrxml.location=zookeeper -Dsolr.hdfs.home=hdfs://datacenter/solr > -Dsolr.hdfs.confdir=/var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/hadoop-conf > -Dsolr.authentication.simple.anonymous.allowed=true > -Dsolr.security.proxyuser.hue.hosts=* > -Dsolr.security.proxyuser.hue.groups=* -Dhost= > bjzw-datacenter-solr-15.d.yourmall.cc -Djetty.port=8983 -Dsolr.host= > bjzw-datacenter-solr-15.d.yourmall.cc -Dsolr.port=8983 > -Dlog4j.configuration=file:///var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/log4j.properties > -Dsolr.log=/var/log/solr -Dsolr.admin.port=8984 > -Dsolr.max.connector.thread=10000 -Dsolr.solr.home=/var/lib/solr > -Djava.endorsed.dirs=/usr/lib/bigtop-tomcat/endorsed -classpath > /usr/lib/bigtop-tomcat/bin/bootstrap.jar > -Dcatalina.base=/var/lib/solr/tomcat-deployment > -Dcatalina.home=/usr/lib/bigtop-tomcat -Djava.io.tmpdir=/var/lib/solr/ > org.apache.catalina.startup.Bootstrap start > > > solr version is solr4.4.0-cdh5.3.0 > jdk version is 1.7.0_67 > > Soft commit time is 1.5s. And we have real time indexing/partialupdating > rate about 100 docs per second. > > When fresh started, Solr will use about 500M memory(the memory show in > solr ui panel). > After several days running, Solr will meet with long time gc problems, and > no response to user query. > > During solr running, the memory used by solr is keep increasing until some > large value, and decrease to > a low level(because of gc), and keep increasing until a larger value > again, then decrease to a low level again … and keep > increasing to an more larger value … until solr has no response and i > restart it. > > > I don’t know how to solve this problem. Can you give me some advices? > > Thanks. > > > >
