Hi Yonik, Thanks for the information, but we are still facing issues related to slowness and high memory usage.
As per my understanding, the default 'FC' method suits are use case, as we have total about 1.1 million documents and no. of unique values for facet fields is quite high. We facet on 5 fields and the no. of unique values are: Field 1 : 19,000 Field 2 : 19,000 Field 3 : 55,000 Field 4: 474 Field 5 : 27 (The alphabetical faceting) All the facet fields are of type string and multivalued. As enum method , will create a bitset for all the unique values, it would be consuming more memory compared to fc method. Also even with a field value cache size of '100', the heap memory(max 6GB) is getting consumed pretty fast. With about 60 parallel requests contributing about 4 million queries, about 25% of our queries have QTime above 1 sec. The max QTime shoots upto 55 sec. Debugging deeper into the solr and lucene code, the particular method which slows us down is IndexSearcher.numDocs which internally gets the terms by loading it from the index. I have not been able to determine the root cause of this. Any other pointers/suggestions in this regard will be helpful. Thanks, Rachita On Tue, Feb 22, 2011 at 10:42 PM, Yonik Seeley <yo...@lucidimagination.com>wrote: > On Tue, Feb 22, 2011 at 9:13 AM, Rachita Choudhary > <rachita.choudh...@burrp.com> wrote: > > Hi Solr Users, > > > > We are upgrading from Solr 1.3 to Solr 1.4.1. > > While using Solr 1.3 , we were seeing multiple blocking active threads on > > "org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal() ". > > > > To utilize the benefits of NIO, on upgrading to Solr 1.4.1, we see other > > type of multiple blocking threads on > > "org.apache.solr.request.UnInvertedField.getUnInvertedField() & > > > > SegmentReader$CoreReaders.getTermsReader". > > Due to this, the QTimes shoots up from few hundreds to thousand of > > msec.. even going upto 30-40 secs for a single query. > > > > - The multiple blocking threads show up after few thousands of queries. > > - We do not have faceting and sorting on the same fields. > > - Our facet fields are multivalued text fields, but no large text values > are > > present. > > - Index size - around 10 GB > > - We have not specified any method for faceting in our schema.xml. > > - Our field value cache settings are: > > <fieldValueCache > > class="solr.FastLRUCache" > > size="175" > > autowarmCount="0" > > showItems="10" > > /> > > > > Can someone please tell us the why we are seeing these blocked threads ? > > Also if they are related to our field value cache , then a cache of size > 175 > > will be filled up with very few initial queries and right after that we > > should see multiple blocking threads ? > > What difference it will make if we have "facet.method = enum" ? > > fc method on a multivalued field instantiates an UnInvertedField (like > a multi-valued field cache) which can take some time. > Just like sorting, you may want to use some warming faceting queries > to make sure that real queries don't pay the cost of the initial entry > construction. > > From your fieldValueCache statistics, it looks like the number of > terms is low enough that the enum method may be fine here. > > -Yonik > http://lucidimagination.com > > > > Is this all related to fieldValueCache or is there some other > configuration > > which we need to set to avoid these blocking threads? > > > > Thanks, > > Rachita > > > > *Cache values example: > > *facetField1_27443 : > > > {field=facet1_27443,memSize=4214884,tindexSize=52,time=22,phase1=15,nTerms=4,bigTerms=0,termInstances=6,uses=1} > > > > facetField1_70 : > > > {field=facetField1_70,memSize=4223310,tindexSize=308,time=28,phase1=21,nTerms=636,bigTerms=0,termInstances=14404,uses=1} > > > > facetField2 : > {field=facetField2,memSize=4262644,tindexSize=3156,time=273,phase1=267,nTerms=12188,bigTerms=0,termInstances=1255522,uses=7031} >
