On the other hand We couldnt use the cache for below types queries. I think its caused from grouping. Anyway we need to be sub second without cache.
On Sat, Aug 27, 2011 at 2:18 PM, Lord Khan Han <khanuniver...@gmail.com>wrote: > Hi, > > Thanks for the reply. > > Here the solr log capture.: > > ****** > > hl.fragsize=100&spellcheck=true&spellcheck.q=XXXXX&group.limit=5&hl.simple.pre=<b>&hl.fl=content&spellcheck.collate=true&wt=javabin&hl=true&rows=20&version=2&fl=score,approved,domain,host,id,lang,mimetype,title,tstamp,url,category&hl.snippets=3&start=0&q=%2BXXXX+-"XXXXX"+-"XXXXX"+-"XXXXXX"+-"XXXXXX"+-"XXXXXX"+-XXXX+-"XXXXXX"+-XXX+-"XXXXX"+-XXXX+-XXXX+-"XXXXX"+-"XXXXX"+-"XXXXX"+-XXXX+-"XXXX"+-"XXXXX"+-"XXXXXX"+-"XXXXX"+-"XXXXXX"+-"XXXXXX"+-XXXX+-"XXXXX"+-"XXXXXX"+-XXXX+-"XXXXX"+-"XXXXX"+-XXXXX+-"XXXXX"+-"XXXXX"+-"XXXXX"+-"XXXXX"+-XXXXX+-"XXXXXX"+-"XXXXXX"+-XXXXXX+-XXXXX+-"XXXXX"+"XXXXX"+"XXXXX"+"XXXXXX"++&group.field=host&hl.simple.post=</b>&group=true&qt=search&fq=mrank:[0+TO+100]&fq=word_count:[70+TO+*] > ****** > > XXXX is the words. All phrases "xxxxx" has two words inside. > > The timing from the DebugQuery: > > <lst name="timing"> > <double name="time">8654.0</double> > <lst name="prepare"> > <double name="time">16.0</double> > <lst name="org.apache.solr.handler.component.QueryComponent"> > <double name="time">16.0</double> > </lst> > <lst name="org.apache.solr.handler.component.FacetComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.HighlightComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.StatsComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.SpellCheckComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.DebugComponent"> > <double name="time">0.0</double> > </lst> > </lst> > <lst name="process"> > <double name="time">8638.0</double> > <lst name="org.apache.solr.handler.component.QueryComponent"> > <double name="time">4473.0</double> > </lst> > <lst name="org.apache.solr.handler.component.FacetComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.HighlightComponent"> > <double name="time">42.0</double> > </lst> > <lst name="org.apache.solr.handler.component.StatsComponent"> > <double name="time">0.0</double> > </lst> > <lst name="org.apache.solr.handler.component.SpellCheckComponent"> > <double name="time">1.0</double> > </lst> > <lst name="org.apache.solr.handler.component.DebugComponent"> > <double name="time">4122.0</double> > </lst> > > > The funny thing is if I removed the ShingleFilter from the below "sh_text" > field and index normally the query time is half of the current shingle one > !. Shouldn't be shingled index better for such heavy 2 word phrases search > ? I am confused. > > On the other hand One of the on the shelf big FAT companies search engine > doing the same query same machine 0.7 / 0.8 secs without cache . I am > confident we can do better in solr but couldnt find the way at the moment. > > thanks for helping.. > > > > > On Sat, Aug 27, 2011 at 2:46 AM, Erik Hatcher <erik.hatc...@gmail.com>wrote: > >> >> On Aug 26, 2011, at 17:49 , Lord Khan Han wrote: >> > We are indexing news document from the various sites. Currently we have >> > 200K docs indexed. Total index size is 36 gig. There is also >> attachement to >> > the news (pdf -docs etc) So document size could be high (ie 10mb). >> > >> > We are using some complex queries which includes around 30 - 40 terms >> per >> > query. %70 of this terms is two word phrases. We are using >> > with conjunction + and - to pinpoint exact result. >> > There is also grouping, dismax and boosting , Termvector HL . >> >> You're using a lot of componentry there, and have complex queries. We >> need more details. >> >> Turn on debugQuery=true... what do the timings say for each component? >> >> > Our problem is query times. Currently its around 6-7 secs. I know our >> query >> > is little bit heavy but we want to improve query performance. I believe >> we >> > can make it sub second but no succes at the moment. >> >> Please provide an example query or two (perhaps a full line logged from >> Solr itself), and then let's see what debugQuery says about your query being >> parsed. >> >> > We tried to use shingle 2 word token it decreases the query performcen >> !! We >> > assumed it will help the speed up phrases search.. >> >> Again, we'd need to see a parsed query to understand this deeper. >> >> Lots of synonym expansion? A parsed query will tell us. >> >> >> >> > (using solr latest trunk and HW is pretty good, 32 core with 32 gig >> ram) >> > >> > Here the field def: >> > >> > <fieldType name="sh_text" class="solr.TextField" >> positionIncrementGap="100" >> > autoGeneratePhraseQueries="true"> >> > <analyzer type="index"> >> > <tokenizer class="solr.WhitespaceTokenizerFactory"/> >> > <filter class="solr.StopFilterFactory" ignoreCase="true" >> > words="stopwords.txt" enablePositionIncrements="true" /> >> > <filter class="solr.WordDelimiterFilterFactory" >> > generateWordParts="1" generateNumberParts="1" catenateWords="1" >> > catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> >> > <!--<filter class="solr.LowerCaseFilterFactory"/>--> >> > <filter class="solr.KeywordMarkerFilterFactory" >> > protected="protwords.txt"/> >> > <filter class="solr.ShingleFilterFactory" maxShingleSize="2" >> > outputUnigrams="true"/> >> > </analyzer> >> > <analyzer type="query"> >> > <tokenizer class="solr.WhitespaceTokenizerFactory"/> >> > <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" >> > ignoreCase="true" expand="true"/> >> > <filter class="solr.StopFilterFactory" ignoreCase="true" >> > words="stopwords.txt" enablePositionIncrements="true" /> >> > <filter class="solr.WordDelimiterFilterFactory" >> > generateWordParts="1" generateNumberParts="1" catenateWords="0" >> > catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> >> > <!--<filter class="solr.LowerCaseFilterFactory"/>--> >> > <filter class="solr.KeywordMarkerFilterFactory" >> > protected="protwords.txt"/> >> > <filter class="solr.ShingleFilterFactory" maxShingleSize="2" >> > outputUnigrams="true"/> >> > </analyzer> >> > </fieldType> >> > >> > and >> > >> > <field name="content" type="sh_text" stored="true" indexed="true" >> > termVectors="true" termPositions="true" termOffsets="true"/> >> >> >
