Yesterday I found there are some slow join operations in another collection whose from index is the collection with many searchers opened. Those slow join operations will be auto warmed when that collection is soft committed. The auto warm time is about 120s but the soft commit time is 30s. So that collection keeps auto warming all the time. Every join operation will open a searcher of “fromindex” collection, and doesn’t close it until join is finished. So I see so many opened searchers in “fromindex” collection, and those opened searchers waste a lot of memory.
I removed those joins, now the memory used by Solr is OK. Thank you very much Erick. I have got stuck on these problem for several weeks! Thank you all. > 在 2015年12月22日，13:24，Erick Erickson <erickerick...@gmail.com> 写道： > > bq: What can we benefit from set maxWarmingSearchers to a larger value > > You really don't get _any_ value. That's in there as a safety valve to > prevent run-away resource consumption. Getting this warning in your logs > means you're mis-configuring your system. Increasing the value is almost > totally useless. It simply makes little sense to have your soft commit take > less time than your autowarming, that's a ton of wasted work for no purpose. > It's highly unlikely that your users _really_ need 1.5 second latency, my bet > is 10-15 seconds would be fine. You know best of course, but this kind of > requirement is often something that people _think_ they need but really > don't. It particularly amuses me when the time between when a document > changes and any attempt is made to send it to solr is minutes, but the > product manager insists that "Solr must show the doc within two seconds of > sending it to the index". > > It's often actually acceptable for your users to know "it may take up to a > minute for the docs to be searchable". What's usually not acceptable is > unpredictability. But again that's up to your product managers. > > bq: You mean if my customer SearchComponent open a searcher, it will exceed > the limit set by maxWarmingSearchers? > > Not at all. but if you don't close it properly (it's reference counted), then > more and more searchers will stay open, chewing up memory. So you may just be > failing to close them and seeing memory increase because of that. > > Best, > Erick > > On Mon, Dec 21, 2015 at 6:47 PM, zhenglingyun <konghuaru...@163.com > <mailto:konghuaru...@163.com>> wrote: > Yes, I do have some custom “Tokenizer"s and “SearchComponent"s. > > Here is the screenshot: > > > The number of opened searchers keeps changing. This time it’s 10. > > You mean if my customer SearchComponent open a searcher, it will exceed > the limit set by maxWarmingSearchers? I’ll check that, thanks! > > I have to do a short time commit. Our application needs a near real time > searching > service. But I’m not sure whether Solr can support NRT search in other ways. > Can > you give me some advices? > > The value of maxWarmingSearchers is copied from some example configs I think, > I’ll try to set it back to 2. > > What can we benefit from set maxWarmingSearchers to a larger value? I don't > find > the answer on google and apache-solr-ref-guide. > > > > >> 在 2015年12月22日，00:34，Erick Erickson <erickerick...@gmail.com >> <mailto:erickerick...@gmail.com>> 写道： >> >> Do you have any custom components? Indeed, you shouldn't have >> that many searchers open. But could we see a screenshot? That's >> the best way to insure that we're talking about the same thing. >> >> Your autocommit settings are really hurting you. Your commit interval >> should be as long as you can tolerate. At that kind of commit frequency, >> your caches are of very limited usefulness anyway, so you can pretty >> much shut them off. Every 1.5 seconds, they're invalidated totally. >> >> Upping maxWarmingSearchers is almost always a mistake. That's >> a safety valve that's there in order to prevent runaway resource >> consumption and almost always means the system is mis-configured. >> I'd put it back to 2 and tune the rest of the system to avoid it rather >> than bumping it up. >> >> Best, >> Erick >> >> On Sun, Dec 20, 2015 at 11:43 PM, zhenglingyun <konghuaru...@163.com >> <mailto:konghuaru...@163.com>> wrote: >>> Just now, I see about 40 "Searchers@XXXX main" displayed in Solr Web UI: >>> collection -> Plugins/Stats -> CORE >>> >>> I think it’s abnormal! >>> >>> softcommit is set to 1.5s, but warmupTime needs about 3s >>> Does it lead to so many Searchers? >>> >>> maxWarmingSearchers is set to 4 in my solrconfig.xml, >>> doesn’t it will prevent Solr from creating more than 4 Searchers? >>> >>> >>> >>>> 在 2015年12月21日，14:43，zhenglingyun <konghuaru...@163.com >>>> <mailto:konghuaru...@163.com>> 写道： >>>> >>>> Thanks Erick for pointing out the memory change in a sawtooth pattern. >>>> The problem troubles me is that the bottom point of the sawtooth keeps >>>> increasing. >>>> And when the used capacity of old generation exceeds the threshold set by >>>> CMS’s >>>> CMSInitiatingOccupancyFraction, gc keeps running and uses a lot of CPU >>>> cycle >>>> but the used old generation memory does not decrease. >>>> >>>> After I take Rahul’s advice, I decrease the Xms and Xmx from 16G to 8G, and >>>> adjust the parameters of JVM from >>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC >>>> -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 >>>> -XX:+CMSParallelRemarkEnabled >>>> to >>>> -XX:NewRatio=3 >>>> -XX:SurvivorRatio=4 >>>> -XX:TargetSurvivorRatio=90 >>>> -XX:MaxTenuringThreshold=8 >>>> -XX:+UseConcMarkSweepGC >>>> -XX:+UseParNewGC >>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 >>>> -XX:+CMSScavengeBeforeRemark >>>> -XX:PretenureSizeThreshold=64m >>>> -XX:+UseCMSInitiatingOccupancyOnly >>>> -XX:CMSInitiatingOccupancyFraction=50 >>>> -XX:CMSMaxAbortablePrecleanTime=6000 >>>> -XX:+CMSParallelRemarkEnabled >>>> -XX:+ParallelRefProcEnabled >>>> -XX:-CMSConcurrentMTEnabled >>>> which is taken from bin/solr.in.sh <http://solr.in.sh/> >>>> I hope this can reduce gc pause time and full gc times. >>>> And maybe the memory increasing problem will disappear if I’m lucky. >>>> >>>> After several day's running, the memory on one of my two servers increased >>>> to 90% again… >>>> (When solr is started, the memory used by solr is less than 1G.) >>>> >>>> Following is the output of stat -gccause -h5 <pid> 1000: >>>> >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 9.56 0.00 8.65 91.31 65.89 69379 3076.096 16563 1579.639 4655.735 >>>> Allocation Failure No GC >>>> 9.56 0.00 51.10 91.31 65.89 69379 3076.096 16563 1579.639 4655.735 >>>> Allocation Failure No GC >>>> 0.00 9.23 10.23 91.35 65.89 69380 3076.135 16563 1579.639 4655.774 >>>> Allocation Failure No GC >>>> 7.90 0.00 9.74 91.39 65.89 69381 3076.165 16564 1579.683 4655.848 >>>> CMS Final Remark No GC >>>> 7.90 0.00 67.45 91.39 65.89 69381 3076.165 16564 1579.683 4655.848 >>>> CMS Final Remark No GC >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 0.00 7.48 16.18 91.41 65.89 69382 3076.200 16565 1579.707 4655.908 >>>> CMS Initial Mark No GC >>>> 0.00 7.48 73.77 91.41 65.89 69382 3076.200 16565 1579.707 4655.908 >>>> CMS Initial Mark No GC >>>> 8.61 0.00 29.86 91.45 65.89 69383 3076.228 16565 1579.707 4655.936 >>>> Allocation Failure No GC >>>> 8.61 0.00 90.16 91.45 65.89 69383 3076.228 16565 1579.707 4655.936 >>>> Allocation Failure No GC >>>> 0.00 7.46 47.89 91.46 65.89 69384 3076.258 16565 1579.707 4655.966 >>>> Allocation Failure No GC >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 8.67 0.00 11.98 91.49 65.89 69385 3076.287 16565 1579.707 4655.995 >>>> Allocation Failure No GC >>>> 0.00 11.76 9.24 91.54 65.89 69386 3076.321 16566 1579.759 4656.081 >>>> CMS Final Remark No GC >>>> 0.00 11.76 64.53 91.54 65.89 69386 3076.321 16566 1579.759 4656.081 >>>> CMS Final Remark No GC >>>> 7.25 0.00 20.39 91.57 65.89 69387 3076.358 16567 1579.786 4656.144 >>>> CMS Initial Mark No GC >>>> 7.25 0.00 81.56 91.57 65.89 69387 3076.358 16567 1579.786 4656.144 >>>> CMS Initial Mark No GC >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 0.00 8.05 34.42 91.60 65.89 69388 3076.391 16567 1579.786 4656.177 >>>> Allocation Failure No GC >>>> 0.00 8.05 84.17 91.60 65.89 69388 3076.391 16567 1579.786 4656.177 >>>> Allocation Failure No GC >>>> 8.54 0.00 55.14 91.62 65.89 69389 3076.420 16567 1579.786 4656.205 >>>> Allocation Failure No GC >>>> 0.00 7.74 12.42 91.66 65.89 69390 3076.456 16567 1579.786 4656.242 >>>> Allocation Failure No GC >>>> 9.60 0.00 11.00 91.70 65.89 69391 3076.492 16568 1579.841 4656.333 >>>> CMS Final Remark No GC >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 9.60 0.00 69.24 91.70 65.89 69391 3076.492 16568 1579.841 4656.333 >>>> CMS Final Remark No GC >>>> 0.00 8.70 18.21 91.74 65.89 69392 3076.529 16569 1579.870 4656.400 >>>> CMS Initial Mark No GC >>>> 0.00 8.70 61.92 91.74 65.89 69392 3076.529 16569 1579.870 4656.400 >>>> CMS Initial Mark No GC >>>> 7.36 0.00 3.49 91.77 65.89 69393 3076.570 16569 1579.870 4656.440 >>>> Allocation Failure No GC >>>> 7.36 0.00 42.03 91.77 65.89 69393 3076.570 16569 1579.870 4656.440 >>>> Allocation Failure No GC >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 0.00 9.77 0.00 91.80 65.89 69394 3076.604 16569 1579.870 4656.475 >>>> Allocation Failure No GC >>>> 9.08 0.00 9.92 91.82 65.89 69395 3076.632 16570 1579.913 4656.545 >>>> CMS Final Remark No GC >>>> 9.08 0.00 58.90 91.82 65.89 69395 3076.632 16570 1579.913 4656.545 >>>> CMS Final Remark No GC >>>> 0.00 8.44 16.20 91.86 65.89 69396 3076.664 16571 1579.930 4656.594 >>>> CMS Initial Mark No GC >>>> 0.00 8.44 71.95 91.86 65.89 69396 3076.664 16571 1579.930 4656.594 >>>> CMS Initial Mark No GC >>>> S0 S1 E O P YGC YGCT FGC FGCT GCT >>>> LGCC GCC >>>> 8.11 0.00 30.59 91.90 65.89 69397 3076.694 16571 1579.930 4656.624 >>>> Allocation Failure No GC >>>> 8.11 0.00 93.41 91.90 65.89 69397 3076.694 16571 1579.930 4656.624 >>>> Allocation Failure No GC >>>> 0.00 9.77 57.34 91.96 65.89 69398 3076.724 16571 1579.930 4656.654 >>>> Allocation Failure No GC >>>> >>>> Full gc seems can’t free any garbage any more (Or the garbage produced is >>>> as fast as gc freed?) >>>> On the other hand, another replication of the collection on another >>>> server(the collection has two replications) >>>> uses 40% of old generation memory, and doesn’t trigger so many full gc. >>>> >>>> >>>> Following is the output of eclipse MAT leak suspects: >>>> >>>> Problem Suspect 1 >>>> >>>> 4,741 instances of "org.apache.lucene.index.SegmentCoreReaders", loaded by >>>> "org.apache.catalina.loader.WebappClassLoader @ 0x67d8ed978" occupy >>>> 3,743,067,520 (64.12%) bytes. These instances are referenced from one >>>> instance of "java.lang.Object[]", loaded by "<system class loader>" >>>> >>>> Keywords >>>> java.lang.Object[] >>>> org.apache.catalina.loader.WebappClassLoader @ 0x67d8ed978 >>>> org.apache.lucene.index.SegmentCoreReaders >>>> >>>> Details » >>>> Problem Suspect 2 >>>> >>>> 2,815 instances of "org.apache.lucene.index.StandardDirectoryReader", >>>> loaded by "org.apache.catalina.loader.WebappClassLoader @ 0x67d8ed978" >>>> occupy 970,614,912 (16.63%) bytes. These instances are referenced from one >>>> instance of "java.lang.Object[]", loaded by "<system class loader>" >>>> >>>> Keywords >>>> java.lang.Object[] >>>> org.apache.catalina.loader.WebappClassLoader @ 0x67d8ed978 >>>> org.apache.lucene.index.StandardDirectoryReader >>>> >>>> Details » >>>> >>>> >>>> >>>> Class structure in above “Details": >>>> >>>> java.lang.Thread @XXX >>>> <Java Local> java.util.ArrayList @XXXX >>>> elementData java.lang.Object[3141] @XXXX >>>> org.apache.lucene.search.FieldCache$CacheEntry @XXXX >>>> org.apache.lucene.search.FieldCache$CacheEntry @XXXX >>>> org.apache.lucene.search.FieldCache$CacheEntry @XXXX >>>> … >>>> a lot of org.apache.lucene.search.FieldCache$CacheEntry (1205 in Suspect >>>> 1, 2785 in Suspect 2) >>>> >>>> Does these lots of org.apache.lucene.search.FieldCache$CacheEntry normal? >>>> >>>> Thanks. >>>> >>>> >>>> >>>> >>>>> 在 2015年12月16日，00:44，Erick Erickson <erickerick...@gmail.com >>>>> <mailto:erickerick...@gmail.com>> 写道： >>>>> >>>>> Rahul's comments were spot on. You can gain more confidence that this >>>>> is normal if if you try attaching a memory reporting program (jconsole >>>>> is one) you'll see the memory grow for quite a while, then garbage >>>>> collection kicks in and you'll see it drop in a sawtooth pattern. >>>>> >>>>> Best, >>>>> Erick >>>>> >>>>> On Tue, Dec 15, 2015 at 8:19 AM, zhenglingyun <konghuaru...@163.com >>>>> <mailto:konghuaru...@163.com>> wrote: >>>>>> Thank you very much. >>>>>> I will try reduce the heap memory and check if the memory still keep >>>>>> increasing or not. >>>>>> >>>>>>> 在 2015年12月15日，19:37，Rahul Ramesh <rr.ii...@gmail.com >>>>>>> <mailto:rr.ii...@gmail.com>> 写道： >>>>>>> >>>>>>> You should actually decrease solr heap size. Let me explain a bit. >>>>>>> >>>>>>> Solr requires very less heap memory for its operation and more memory >>>>>>> for >>>>>>> storing data in main memory. This is because solr uses mmap for storing >>>>>>> the >>>>>>> index files. >>>>>>> Please check the link >>>>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html >>>>>>> <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html> >>>>>>> for >>>>>>> understanding how solr operates on files . >>>>>>> >>>>>>> Solr has typical problem of Garbage collection once you the heap size >>>>>>> to a >>>>>>> large value. It will have indeterminate pauses due to GC. The amount of >>>>>>> heap memory required is difficult to tell. However the way we tuned this >>>>>>> parameter is setting it to a low value and increasing it by 1Gb whenever >>>>>>> OOM is thrown. >>>>>>> >>>>>>> Please check the problem of having large Java Heap >>>>>>> >>>>>>> http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap >>>>>>> <http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap> >>>>>>> >>>>>>> >>>>>>> Just for your reference, in our production setup, we have data of around >>>>>>> 60Gb/node spread across 25 collections. We have configured 8GB as heap >>>>>>> and >>>>>>> the rest of the memory we will leave it to OS to manage. We do around >>>>>>> 1000 >>>>>>> (search + Insert)/second on the data. >>>>>>> >>>>>>> I hope this helps. >>>>>>> >>>>>>> Regards, >>>>>>> Rahul >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Dec 15, 2015 at 4:33 PM, zhenglingyun <konghuaru...@163.com >>>>>>> <mailto:konghuaru...@163.com>> wrote: >>>>>>> >>>>>>>> Hi, list >>>>>>>> >>>>>>>> I’m new to solr. Recently I encounter a “memory leak” problem with >>>>>>>> solrcloud. >>>>>>>> >>>>>>>> I have two 64GB servers running a solrcloud cluster. In the solrcloud, >>>>>>>> I >>>>>>>> have >>>>>>>> one collection with about 400k docs. The index size of the collection >>>>>>>> is >>>>>>>> about >>>>>>>> 500MB. Memory for solr is 16GB. >>>>>>>> >>>>>>>> Following is "ps aux | grep solr” : >>>>>>>> >>>>>>>> /usr/java/jdk1.7.0_67-cloudera/bin/java >>>>>>>> -Djava.util.logging.config.file=/var/lib/solr/tomcat-deployment/conf/logging.properties >>>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >>>>>>>> -Djava.net.preferIPv4Stack=true -Dsolr.hdfs.blockcache.enabled=true >>>>>>>> -Dsolr.hdfs.blockcache.direct.memory.allocation=true >>>>>>>> -Dsolr.hdfs.blockcache.blocksperbank=16384 >>>>>>>> -Dsolr.hdfs.blockcache.slab.count=1 -Xms16608395264 -Xmx16608395264 >>>>>>>> -XX:MaxDirectMemorySize=21590179840 -XX:+UseParNewGC >>>>>>>> -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled >>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled >>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC >>>>>>>> -Xloggc:/var/log/solr/gc.log >>>>>>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh >>>>>>>> -DzkHost= >>>>>>>> bjzw-datacenter-hadoop-160.d.yourmall.cc:2181 >>>>>>>> <http://bjzw-datacenter-hadoop-160.d.yourmall.cc:2181/>, >>>>>>>> bjzw-datacenter-hadoop-163.d.yourmall.cc:2181 >>>>>>>> <http://bjzw-datacenter-hadoop-163.d.yourmall.cc:2181/>, >>>>>>>> bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr >>>>>>>> <http://bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr> >>>>>>>> -Dsolr.solrxml.location=zookeeper >>>>>>>> -Dsolr.hdfs.home=hdfs://datacenter/solr >>>>>>>> -Dsolr.hdfs.confdir=/var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/hadoop-conf >>>>>>>> -Dsolr.authentication.simple.anonymous.allowed=true >>>>>>>> -Dsolr.security.proxyuser.hue.hosts=* >>>>>>>> -Dsolr.security.proxyuser.hue.groups=* -Dhost= >>>>>>>> bjzw-datacenter-solr-15.d.yourmall.cc >>>>>>>> <http://bjzw-datacenter-solr-15.d.yourmall.cc/> -Djetty.port=8983 >>>>>>>> -Dsolr.host= >>>>>>>> bjzw-datacenter-solr-15.d.yourmall.cc >>>>>>>> <http://bjzw-datacenter-solr-15.d.yourmall.cc/> -Dsolr.port=8983 >>>>>>>> -Dlog4j.configuration=file:///var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/log4j.properties >>>>>>>> -Dsolr.log=/var/log/solr -Dsolr.admin.port=8984 >>>>>>>> -Dsolr.max.connector.thread=10000 -Dsolr.solr.home=/var/lib/solr >>>>>>>> -Djava.net.preferIPv4Stack=true -Dsolr.hdfs.blockcache.enabled=true >>>>>>>> -Dsolr.hdfs.blockcache.direct.memory.allocation=true >>>>>>>> -Dsolr.hdfs.blockcache.blocksperbank=16384 >>>>>>>> -Dsolr.hdfs.blockcache.slab.count=1 -Xms16608395264 -Xmx16608395264 >>>>>>>> -XX:MaxDirectMemorySize=21590179840 -XX:+UseParNewGC >>>>>>>> -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled >>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled >>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC >>>>>>>> -Xloggc:/var/log/solr/gc.log >>>>>>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh >>>>>>>> -DzkHost= >>>>>>>> bjzw-datacenter-hadoop-160.d.yourmall.cc:2181 >>>>>>>> <http://bjzw-datacenter-hadoop-160.d.yourmall.cc:2181/>, >>>>>>>> bjzw-datacenter-hadoop-163.d.yourmall.cc:2181 >>>>>>>> <http://bjzw-datacenter-hadoop-163.d.yourmall.cc:2181/>, >>>>>>>> bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr >>>>>>>> <http://bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr> >>>>>>>> -Dsolr.solrxml.location=zookeeper >>>>>>>> -Dsolr.hdfs.home=hdfs://datacenter/solr >>>>>>>> -Dsolr.hdfs.confdir=/var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/hadoop-conf >>>>>>>> -Dsolr.authentication.simple.anonymous.allowed=true >>>>>>>> -Dsolr.security.proxyuser.hue.hosts=* >>>>>>>> -Dsolr.security.proxyuser.hue.groups=* -Dhost= >>>>>>>> bjzw-datacenter-solr-15.d.yourmall.cc >>>>>>>> <http://bjzw-datacenter-solr-15.d.yourmall.cc/> -Djetty.port=8983 >>>>>>>> -Dsolr.host= >>>>>>>> bjzw-datacenter-solr-15.d.yourmall.cc >>>>>>>> <http://bjzw-datacenter-solr-15.d.yourmall.cc/> -Dsolr.port=8983 >>>>>>>> -Dlog4j.configuration=file:///var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/log4j.properties >>>>>>>> -Dsolr.log=/var/log/solr -Dsolr.admin.port=8984 >>>>>>>> -Dsolr.max.connector.thread=10000 -Dsolr.solr.home=/var/lib/solr >>>>>>>> -Djava.endorsed.dirs=/usr/lib/bigtop-tomcat/endorsed -classpath >>>>>>>> /usr/lib/bigtop-tomcat/bin/bootstrap.jar >>>>>>>> -Dcatalina.base=/var/lib/solr/tomcat-deployment >>>>>>>> -Dcatalina.home=/usr/lib/bigtop-tomcat -Djava.io.tmpdir=/var/lib/solr/ >>>>>>>> org.apache.catalina.startup.Bootstrap start >>>>>>>> >>>>>>>> >>>>>>>> solr version is solr4.4.0-cdh5.3.0 >>>>>>>> jdk version is 1.7.0_67 >>>>>>>> >>>>>>>> Soft commit time is 1.5s. And we have real time >>>>>>>> indexing/partialupdating >>>>>>>> rate about 100 docs per second. >>>>>>>> >>>>>>>> When fresh started, Solr will use about 500M memory(the memory show in >>>>>>>> solr ui panel). >>>>>>>> After several days running, Solr will meet with long time gc problems, >>>>>>>> and >>>>>>>> no response to user query. >>>>>>>> >>>>>>>> During solr running, the memory used by solr is keep increasing until >>>>>>>> some >>>>>>>> large value, and decrease to >>>>>>>> a low level(because of gc), and keep increasing until a larger value >>>>>>>> again, then decrease to a low level again … and keep >>>>>>>> increasing to an more larger value … until solr has no response and i >>>>>>>> restart it. >>>>>>>> >>>>>>>> >>>>>>>> I don’t know how to solve this problem. Can you give me some advices? >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >>> >>> > >
