Hi, Thanks for the reply.
Here the solr log capture.: ****** hl.fragsize=100&spellcheck=true&spellcheck.q=XXXXX&group.limit=5&hl.simple.pre=<b>&hl.fl=content&spellcheck.collate=true&wt=javabin&hl=true&rows=20&version=2&fl=score,approved,domain,host,id,lang,mimetype,title,tstamp,url,category&hl.snippets=3&start=0&q=%2BXXXX+-"XXXXX"+-"XXXXX"+-"XXXXXX"+-"XXXXXX"+-"XXXXXX"+-XXXX+-"XXXXXX"+-XXX+-"XXXXX"+-XXXX+-XXXX+-"XXXXX"+-"XXXXX"+-"XXXXX"+-XXXX+-"XXXX"+-"XXXXX"+-"XXXXXX"+-"XXXXX"+-"XXXXXX"+-"XXXXXX"+-XXXX+-"XXXXX"+-"XXXXXX"+-XXXX+-"XXXXX"+-"XXXXX"+-XXXXX+-"XXXXX"+-"XXXXX"+-"XXXXX"+-"XXXXX"+-XXXXX+-"XXXXXX"+-"XXXXXX"+-XXXXXX+-XXXXX+-"XXXXX"+"XXXXX"+"XXXXX"+"XXXXXX"++&group.field=host&hl.simple.post=</b>&group=true&qt=search&fq=mrank:[0+TO+100]&fq=word_count:[70+TO+*] ****** XXXX is the words. All phrases "xxxxx" has two words inside. The timing from the DebugQuery: <lst name="timing"> <double name="time">8654.0</double> <lst name="prepare"> <double name="time">16.0</double> <lst name="org.apache.solr.handler.component.QueryComponent"> <double name="time">16.0</double> </lst> <lst name="org.apache.solr.handler.component.FacetComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.HighlightComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.StatsComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.SpellCheckComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.DebugComponent"> <double name="time">0.0</double> </lst> </lst> <lst name="process"> <double name="time">8638.0</double> <lst name="org.apache.solr.handler.component.QueryComponent"> <double name="time">4473.0</double> </lst> <lst name="org.apache.solr.handler.component.FacetComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.HighlightComponent"> <double name="time">42.0</double> </lst> <lst name="org.apache.solr.handler.component.StatsComponent"> <double name="time">0.0</double> </lst> <lst name="org.apache.solr.handler.component.SpellCheckComponent"> <double name="time">1.0</double> </lst> <lst name="org.apache.solr.handler.component.DebugComponent"> <double name="time">4122.0</double> </lst> The funny thing is if I removed the ShingleFilter from the below "sh_text" field and index normally the query time is half of the current shingle one !. Shouldn't be shingled index better for such heavy 2 word phrases search ? I am confused. On the other hand One of the on the shelf big FAT companies search engine doing the same query same machine 0.7 / 0.8 secs without cache . I am confident we can do better in solr but couldnt find the way at the moment. thanks for helping.. On Sat, Aug 27, 2011 at 2:46 AM, Erik Hatcher <erik.hatc...@gmail.com>wrote: > > On Aug 26, 2011, at 17:49 , Lord Khan Han wrote: > > We are indexing news document from the various sites. Currently we have > > 200K docs indexed. Total index size is 36 gig. There is also attachement > to > > the news (pdf -docs etc) So document size could be high (ie 10mb). > > > > We are using some complex queries which includes around 30 - 40 terms per > > query. %70 of this terms is two word phrases. We are using > > with conjunction + and - to pinpoint exact result. > > There is also grouping, dismax and boosting , Termvector HL . > > You're using a lot of componentry there, and have complex queries. We need > more details. > > Turn on debugQuery=true... what do the timings say for each component? > > > Our problem is query times. Currently its around 6-7 secs. I know our > query > > is little bit heavy but we want to improve query performance. I believe > we > > can make it sub second but no succes at the moment. > > Please provide an example query or two (perhaps a full line logged from > Solr itself), and then let's see what debugQuery says about your query being > parsed. > > > We tried to use shingle 2 word token it decreases the query performcen !! > We > > assumed it will help the speed up phrases search.. > > Again, we'd need to see a parsed query to understand this deeper. > > Lots of synonym expansion? A parsed query will tell us. > > > > > (using solr latest trunk and HW is pretty good, 32 core with 32 gig ram) > > > > Here the field def: > > > > <fieldType name="sh_text" class="solr.TextField" > positionIncrementGap="100" > > autoGeneratePhraseQueries="true"> > > <analyzer type="index"> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/> > > <filter class="solr.StopFilterFactory" ignoreCase="true" > > words="stopwords.txt" enablePositionIncrements="true" /> > > <filter class="solr.WordDelimiterFilterFactory" > > generateWordParts="1" generateNumberParts="1" catenateWords="1" > > catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> > > <!--<filter class="solr.LowerCaseFilterFactory"/>--> > > <filter class="solr.KeywordMarkerFilterFactory" > > protected="protwords.txt"/> > > <filter class="solr.ShingleFilterFactory" maxShingleSize="2" > > outputUnigrams="true"/> > > </analyzer> > > <analyzer type="query"> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/> > > <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" > > ignoreCase="true" expand="true"/> > > <filter class="solr.StopFilterFactory" ignoreCase="true" > > words="stopwords.txt" enablePositionIncrements="true" /> > > <filter class="solr.WordDelimiterFilterFactory" > > generateWordParts="1" generateNumberParts="1" catenateWords="0" > > catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> > > <!--<filter class="solr.LowerCaseFilterFactory"/>--> > > <filter class="solr.KeywordMarkerFilterFactory" > > protected="protwords.txt"/> > > <filter class="solr.ShingleFilterFactory" maxShingleSize="2" > > outputUnigrams="true"/> > > </analyzer> > > </fieldType> > > > > and > > > > <field name="content" type="sh_text" stored="true" indexed="true" > > termVectors="true" termPositions="true" termOffsets="true"/> > >
