Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-22 Thread zhenglingyun
Yesterday I found there are some slow join operations in another collection whose from index is the collection with many searchers opened. Those slow join operations will be auto warmed when that collection is soft committed. The auto warm time is about 120s but the soft commit time is 30s. So th

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-21 Thread Erick Erickson
bq: What can we benefit from set maxWarmingSearchers to a larger value You really don't get _any_ value. That's in there as a safety valve to prevent run-away resource consumption. Getting this warning in your logs means you're mis-configuring your system. Increasing the value is almost totally us

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-21 Thread Erick Erickson
Do you have any custom components? Indeed, you shouldn't have that many searchers open. But could we see a screenshot? That's the best way to insure that we're talking about the same thing. Your autocommit settings are really hurting you. Your commit interval should be as long as you can tolerate.

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-20 Thread zhenglingyun
Just now, I see about 40 "Searchers@ main" displayed in Solr Web UI: collection -> Plugins/Stats -> CORE I think it’s abnormal! softcommit is set to 1.5s, but warmupTime needs about 3s Does it lead to so many Searchers? maxWarmingSearchers is set to 4 in my solrconfig.xml, doesn’t it will p

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-20 Thread zhenglingyun
Thanks Erick for pointing out the memory change in a sawtooth pattern. The problem troubles me is that the bottom point of the sawtooth keeps increasing. And when the used capacity of old generation exceeds the threshold set by CMS’s CMSInitiatingOccupancyFraction, gc keeps running and uses a lot

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-15 Thread Erick Erickson
Rahul's comments were spot on. You can gain more confidence that this is normal if if you try attaching a memory reporting program (jconsole is one) you'll see the memory grow for quite a while, then garbage collection kicks in and you'll see it drop in a sawtooth pattern. Best, Erick On Tue, Dec

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-15 Thread zhenglingyun
Thank you very much. I will try reduce the heap memory and check if the memory still keep increasing or not. > 在 2015年12月15日,19:37,Rahul Ramesh 写道: > > You should actually decrease solr heap size. Let me explain a bit. > > Solr requires very less heap memory for its operation and more memory f

Re: solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-15 Thread Rahul Ramesh
You should actually decrease solr heap size. Let me explain a bit. Solr requires very less heap memory for its operation and more memory for storing data in main memory. This is because solr uses mmap for storing the index files. Please check the link http://blog.thetaphi.de/2012/07/use-lucenes-mm

solrcloud used a lot of memory and memory keep increasing during long time run

2015-12-15 Thread zhenglingyun
Hi, list I’m new to solr. Recently I encounter a “memory leak” problem with solrcloud. I have two 64GB servers running a solrcloud cluster. In the solrcloud, I have one collection with about 400k docs. The index size of the collection is about 500MB. Memory for solr is 16GB. Following is "ps aux