Thanks for your suggestions. I will look into the link provided. http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
This is usually an anti-pattern. The very first thing I'd be doing is trying to not do this. See ngrams for infix queries, or shingles or ReverseWildcardFilterFactory or..... We cannot avoid multiple wildcards since that's is our user's requirement. We try to discourage it but the users insist on firing such queries. Also, ngrams etc. can be tried but our index is already huge and ngrams may further add lot to it. We are OK with such queries failing as long as other queries are not affected. Please find the details below. So, how many nodes in the cluster ? There are total 4 nodes on the cluster. How many shards and replicas for the collection ? There are 4 shards and no replica for any of them. How many items in the collection ? If I understand the question correctly there are two collection on each node and there size on each node is approximately 190GB and 130GB. What is the size of the index ? There are two collection on each node and there size on each node is approximately 190GB and 130GB. How is updated the collection (frequency, how many items per days, what is your hard commit strategy) ? It is an optimized index and read-only. There are no inter-mediate update. How are configured cache in solrconfig.xml ? Filter cache, query result cache and document cache are enabled. Auto-warming is also done. Can you provide all other JVM parameters ? -Xms20g -Xmx24g -XX:+UseConcMarkSweepGC Thanks again, Modassar On Wed, Dec 24, 2014 at 2:29 AM, Dominique Bejean <dominique.bej...@eolya.fr > wrote: > Hi, > > I agree Erick it could be a good think to have more details about your > configuration and collection. > > Your heap size is 32Gb. How many RAM on each servers ? > > By « 4 shard Solr cluster », you mean a 4 nodes Solr servers or a > collection with 4 shards ? > > So, how many nodes in the cluster ? > How many shards and replicas for the collection ? > How many items in the collection ? > What is the size of the index ? > How is updated the collection (frequency, how many items per days, what is > your hard commit strategy) ? > How are configured cache in solrconfig.xml ? > Can you provide all other JVM parameters ? > > Regards > > Dominique > > 2014-12-23 17:50 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > > > Second most important part of your message: > > "When executing a huge query with many wildcards inside it the server" > > > > This is usually an anti-pattern. The very first thing > > I'd be doing is trying to not do this. See ngrams for infix > > queries, or shingles or ReverseWildcardFilterFactory or..... > > > > And if your corpus is very large with many unique terms it's even > > worse, but you haven't really told us about that yet. > > > > Best, > > Erick > > > > On Tue, Dec 23, 2014 at 8:30 AM, Shawn Heisey <apa...@elyograg.org> > wrote: > > > On 12/23/2014 4:34 AM, Modassar Ather wrote: > > >> Hi, > > >> > > >> I have a setup of 4 shard Solr cluster with embedded zookeeper on one > of > > >> them. The zkClient time out is set to 30 seconds, -Xms is 20g and -Xms > > is > > >> 24g. > > >> When executing a huge query with many wildcards inside it the server > > >> crashes and becomes non-responsive. Even the dashboard does not > responds > > >> and shows connection lost error. This requires me to restart the > > servers. > > > > > > Here's the important part of your message: > > > > > > *Caused by: java.lang.OutOfMemoryError: Java heap space* > > > > > > > > > Your heap is not big enough for what Solr has been asked to do. You > > > need to either increase your heap size or change your configuration so > > > that it uses less memory. > > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > > > > > > Most programs have pretty much undefined behavior when an OOME occurs. > > > Lucene's IndexWriter has been hardened so that it tries extremely hard > > > to avoid index corruption when OOME strikes, and I believe that works > > > well enough that we can call it nearly bulletproof ... but the rest of > > > Lucene and Solr will make no guarantees. > > > > > > It's very difficult to have definable program behavior when an OOME > > > happens, because you simply cannot know the precise point during > program > > > execution where it will happen, or what isn't going to work because > Java > > > did not have memory space to create an object. Going unresponsive is > > > not surprising. > > > > > > If you can solve your heap problem, note that you may run into other > > > performance issues discussed on the wiki page that I linked. > > > > > > Thanks, > > > Shawn > > > > > > On Wed, Dec 24, 2014 at 2:29 AM, Dominique Bejean <dominique.bej...@eolya.fr > wrote: > Hi, > > I agree Erick it could be a good think to have more details about your > configuration and collection. > > Your heap size is 32Gb. How many RAM on each servers ? > > By « 4 shard Solr cluster », you mean a 4 nodes Solr servers or a > collection with 4 shards ? > > So, how many nodes in the cluster ? > How many shards and replicas for the collection ? > How many items in the collection ? > What is the size of the index ? > How is updated the collection (frequency, how many items per days, what is > your hard commit strategy) ? > How are configured cache in solrconfig.xml ? > Can you provide all other JVM parameters ? > > Regards > > Dominique > > 2014-12-23 17:50 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > > > Second most important part of your message: > > "When executing a huge query with many wildcards inside it the server" > > > > This is usually an anti-pattern. The very first thing > > I'd be doing is trying to not do this. See ngrams for infix > > queries, or shingles or ReverseWildcardFilterFactory or..... > > > > And if your corpus is very large with many unique terms it's even > > worse, but you haven't really told us about that yet. > > > > Best, > > Erick > > > > On Tue, Dec 23, 2014 at 8:30 AM, Shawn Heisey <apa...@elyograg.org> > wrote: > > > On 12/23/2014 4:34 AM, Modassar Ather wrote: > > >> Hi, > > >> > > >> I have a setup of 4 shard Solr cluster with embedded zookeeper on one > of > > >> them. The zkClient time out is set to 30 seconds, -Xms is 20g and -Xms > > is > > >> 24g. > > >> When executing a huge query with many wildcards inside it the server > > >> crashes and becomes non-responsive. Even the dashboard does not > responds > > >> and shows connection lost error. This requires me to restart the > > servers. > > > > > > Here's the important part of your message: > > > > > > *Caused by: java.lang.OutOfMemoryError: Java heap space* > > > > > > > > > Your heap is not big enough for what Solr has been asked to do. You > > > need to either increase your heap size or change your configuration so > > > that it uses less memory. > > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > > > > > > Most programs have pretty much undefined behavior when an OOME occurs. > > > Lucene's IndexWriter has been hardened so that it tries extremely hard > > > to avoid index corruption when OOME strikes, and I believe that works > > > well enough that we can call it nearly bulletproof ... but the rest of > > > Lucene and Solr will make no guarantees. > > > > > > It's very difficult to have definable program behavior when an OOME > > > happens, because you simply cannot know the precise point during > program > > > execution where it will happen, or what isn't going to work because > Java > > > did not have memory space to create an object. Going unresponsive is > > > not surprising. > > > > > > If you can solve your heap problem, note that you may run into other > > > performance issues discussed on the wiki page that I linked. > > > > > > Thanks, > > > Shawn > > > > > >
