Modassar, How many items in the collection ? I mean how many documents per collection ? 1 million, 10 millions, …?
How are configured cache in solrconfig.xml ? What are the size attribute value for each cache ? Can you provide a sample of the query ? Does it fail immediately after solrcloud startup or after several hours ? Dominique 2014-12-24 6:20 GMT+01:00 Modassar Ather <modather1...@gmail.com>: > Thanks for your suggestions. > > I will look into the link provided. > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > > This is usually an anti-pattern. The very first thing > I'd be doing is trying to not do this. See ngrams for infix > queries, or shingles or ReverseWildcardFilterFactory or..... > > We cannot avoid multiple wildcards since that's is our user's requirement. > We try to discourage it but the users insist on firing such queries. Also, > ngrams etc. can be tried but our index is already huge and ngrams may > further add lot to it. We are OK with such queries failing as long as other > queries are not affected. > > > Please find the details below. > > So, how many nodes in the cluster ? > There are total 4 nodes on the cluster. > > How many shards and replicas for the collection ? > There are 4 shards and no replica for any of them. > > How many items in the collection ? > If I understand the question correctly there are two collection on each > node and there size on each node is approximately 190GB and 130GB. > > What is the size of the index ? > There are two collection on each node and there size on each node is > approximately 190GB and 130GB. > > How is updated the collection (frequency, how many items per days, what is > your hard commit strategy) ? > It is an optimized index and read-only. There are no inter-mediate update. > > How are configured cache in solrconfig.xml ? > Filter cache, query result cache and document cache are enabled. > Auto-warming is also done. > > Can you provide all other JVM parameters ? > -Xms20g -Xmx24g -XX:+UseConcMarkSweepGC > > Thanks again, > Modassar > > On Wed, Dec 24, 2014 at 2:29 AM, Dominique Bejean < > dominique.bej...@eolya.fr > > wrote: > > > Hi, > > > > I agree Erick it could be a good think to have more details about your > > configuration and collection. > > > > Your heap size is 32Gb. How many RAM on each servers ? > > > > By « 4 shard Solr cluster », you mean a 4 nodes Solr servers or a > > collection with 4 shards ? > > > > So, how many nodes in the cluster ? > > How many shards and replicas for the collection ? > > How many items in the collection ? > > What is the size of the index ? > > How is updated the collection (frequency, how many items per days, what > is > > your hard commit strategy) ? > > How are configured cache in solrconfig.xml ? > > Can you provide all other JVM parameters ? > > > > Regards > > > > Dominique > > > > 2014-12-23 17:50 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > > > > > Second most important part of your message: > > > "When executing a huge query with many wildcards inside it the server" > > > > > > This is usually an anti-pattern. The very first thing > > > I'd be doing is trying to not do this. See ngrams for infix > > > queries, or shingles or ReverseWildcardFilterFactory or..... > > > > > > And if your corpus is very large with many unique terms it's even > > > worse, but you haven't really told us about that yet. > > > > > > Best, > > > Erick > > > > > > On Tue, Dec 23, 2014 at 8:30 AM, Shawn Heisey <apa...@elyograg.org> > > wrote: > > > > On 12/23/2014 4:34 AM, Modassar Ather wrote: > > > >> Hi, > > > >> > > > >> I have a setup of 4 shard Solr cluster with embedded zookeeper on > one > > of > > > >> them. The zkClient time out is set to 30 seconds, -Xms is 20g and > -Xms > > > is > > > >> 24g. > > > >> When executing a huge query with many wildcards inside it the server > > > >> crashes and becomes non-responsive. Even the dashboard does not > > responds > > > >> and shows connection lost error. This requires me to restart the > > > servers. > > > > > > > > Here's the important part of your message: > > > > > > > > *Caused by: java.lang.OutOfMemoryError: Java heap space* > > > > > > > > > > > > Your heap is not big enough for what Solr has been asked to do. You > > > > need to either increase your heap size or change your configuration > so > > > > that it uses less memory. > > > > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > > > > > > > > Most programs have pretty much undefined behavior when an OOME > occurs. > > > > Lucene's IndexWriter has been hardened so that it tries extremely > hard > > > > to avoid index corruption when OOME strikes, and I believe that works > > > > well enough that we can call it nearly bulletproof ... but the rest > of > > > > Lucene and Solr will make no guarantees. > > > > > > > > It's very difficult to have definable program behavior when an OOME > > > > happens, because you simply cannot know the precise point during > > program > > > > execution where it will happen, or what isn't going to work because > > Java > > > > did not have memory space to create an object. Going unresponsive is > > > > not surprising. > > > > > > > > If you can solve your heap problem, note that you may run into other > > > > performance issues discussed on the wiki page that I linked. > > > > > > > > Thanks, > > > > Shawn > > > > > > > > > > > > On Wed, Dec 24, 2014 at 2:29 AM, Dominique Bejean < > dominique.bej...@eolya.fr > > wrote: > > > Hi, > > > > I agree Erick it could be a good think to have more details about your > > configuration and collection. > > > > Your heap size is 32Gb. How many RAM on each servers ? > > > > By « 4 shard Solr cluster », you mean a 4 nodes Solr servers or a > > collection with 4 shards ? > > > > So, how many nodes in the cluster ? > > How many shards and replicas for the collection ? > > How many items in the collection ? > > What is the size of the index ? > > How is updated the collection (frequency, how many items per days, what > is > > your hard commit strategy) ? > > How are configured cache in solrconfig.xml ? > > Can you provide all other JVM parameters ? > > > > Regards > > > > Dominique > > > > 2014-12-23 17:50 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > > > > > Second most important part of your message: > > > "When executing a huge query with many wildcards inside it the server" > > > > > > This is usually an anti-pattern. The very first thing > > > I'd be doing is trying to not do this. See ngrams for infix > > > queries, or shingles or ReverseWildcardFilterFactory or..... > > > > > > And if your corpus is very large with many unique terms it's even > > > worse, but you haven't really told us about that yet. > > > > > > Best, > > > Erick > > > > > > On Tue, Dec 23, 2014 at 8:30 AM, Shawn Heisey <apa...@elyograg.org> > > wrote: > > > > On 12/23/2014 4:34 AM, Modassar Ather wrote: > > > >> Hi, > > > >> > > > >> I have a setup of 4 shard Solr cluster with embedded zookeeper on > one > > of > > > >> them. The zkClient time out is set to 30 seconds, -Xms is 20g and > -Xms > > > is > > > >> 24g. > > > >> When executing a huge query with many wildcards inside it the server > > > >> crashes and becomes non-responsive. Even the dashboard does not > > responds > > > >> and shows connection lost error. This requires me to restart the > > > servers. > > > > > > > > Here's the important part of your message: > > > > > > > > *Caused by: java.lang.OutOfMemoryError: Java heap space* > > > > > > > > > > > > Your heap is not big enough for what Solr has been asked to do. You > > > > need to either increase your heap size or change your configuration > so > > > > that it uses less memory. > > > > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > > > > > > > > Most programs have pretty much undefined behavior when an OOME > occurs. > > > > Lucene's IndexWriter has been hardened so that it tries extremely > hard > > > > to avoid index corruption when OOME strikes, and I believe that works > > > > well enough that we can call it nearly bulletproof ... but the rest > of > > > > Lucene and Solr will make no guarantees. > > > > > > > > It's very difficult to have definable program behavior when an OOME > > > > happens, because you simply cannot know the precise point during > > program > > > > execution where it will happen, or what isn't going to work because > > Java > > > > did not have memory space to create an object. Going unresponsive is > > > > not surprising. > > > > > > > > If you can solve your heap problem, note that you may run into other > > > > performance issues discussed on the wiki page that I linked. > > > > > > > > Thanks, > > > > Shawn > > > > > > > > > >
