If I don't do highlighting, it's really fast. Optimize has no effect. -Peter
On Jul 29, 2010, at 11:54 AM, dc tech wrote: > Are you storing the entire log file text in SOLR? That's almost 3gb of > text that you are storing in the SOLR. Try to > 1) Is this first time performance or on repaat queries with the same fields? > 2) Optimze the index and test performance again > 3) index without storing the text and see what the performance looks like. > > > On 7/29/10, Peter Spam <ps...@mac.com> wrote: >> Any ideas? I've got 5000 documents with an average size of 850k each, and >> it sometimes takes 2 minutes for a query to come back when highlighting is >> turned on! Help! >> >> >> -Pete >> >> On Jul 21, 2010, at 2:41 PM, Peter Spam wrote: >> >>> From the mailing list archive, Koji wrote: >>> >>>> 1. Provide another field for highlighting and use copyField to copy >>>> plainText to the highlighting field. >>> >>> and Lance wrote: >>> http://www.mail-archive.com/solr-user@lucene.apache.org/msg35548.html >>> >>>> If you want to highlight field X, doing the >>>> termOffsets/termPositions/termVectors will make highlighting that field >>>> faster. You should make a separate field and apply these options to that >>>> field. >>>> >>>> Now: doing a copyfield adds a "value" to a multiValued field. For a text >>>> field, you get a multi-valued text field. You should only copy one value >>>> to the highlighted field, so just copyField the document to your special >>>> field. To enforce this, I would add multiValued="false" to that field, >>>> just to avoid mistakes. >>>> >>>> So, all_text should be indexed without the term* attributes, and should >>>> not be stored. Then your document stored in a separate field that you use >>>> for highlighting and has the term* attributes. >>> >>> I've been experimenting with this, and here's what I've tried: >>> >>> <field name="body" type="text_pl" indexed="true" stored="false" >>> multiValued="true" termVectors="true" termPositions="true" termOff >>> sets="true" /> >>> <field name="body_all" type="text_pl" indexed="false" stored="true" >>> multiValued="true" /> >>> <copyField source="body" dest="body_all"/> >>> >>> ... but it's still very slow (10+ seconds). Why is it better to have two >>> fields (one indexed but not stored, and the other not indexed but stored) >>> rather than just one field that's both indexed and stored? >>> >>> >>> From the Perf wiki page http://wiki.apache.org/solr/SolrPerformanceFactors >>> >>>> If you aren't always using all the stored fields, then enabling lazy >>>> field loading can be a huge boon, especially if compressed fields are >>>> used. >>> >>> What does this mean? How do you load a field lazily? >>> >>> Thanks for your time, guys - this has started to become frustrating, since >>> it works so well, but is very slow! >>> >>> >>> -Pete >>> >>> On Jul 20, 2010, at 5:36 PM, Peter Spam wrote: >>> >>>> Data set: About 4,000 log files (will eventually grow to millions). >>>> Average log file is 850k. Largest log file (so far) is about 70MB. >>>> >>>> Problem: When I search for common terms, the query time goes from under >>>> 2-3 seconds to about 60 seconds. TermVectors etc are enabled. When I >>>> disable highlighting, performance improves a lot, but is still slow for >>>> some queries (7 seconds). Thanks in advance for any ideas! >>>> >>>> >>>> -Peter >>>> >>>> >>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>> >>>> 4GB RAM server >>>> % java -Xms2048M -Xmx3072M -jar start.jar >>>> >>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>> >>>> schema.xml changes: >>>> >>>> <fieldType name="text_pl" class="solr.TextField"> >>>> <analyzer> >>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/> >>>> <filter class="solr.LowerCaseFilterFactory"/> >>>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="0" >>>> generateNumberParts="0" catenateWords="0" catenateNumbers="0" >>>> catenateAll="0" splitOnCaseChange="0"/> >>>> </analyzer> >>>> </fieldType> >>>> >>>> ... >>>> >>>> <field name="body" type="text_pl" indexed="true" stored="true" >>>> multiValued="false" termVectors="true" termPositions="true" >>>> termOffsets="true" /> >>>> <field name="timestamp" type="date" indexed="true" stored="true" >>>> default="NOW" multiValued="false"/> >>>> <field name="version" type="string" indexed="true" stored="true" >>>> multiValued="false"/> >>>> <field name="device" type="string" indexed="true" stored="true" >>>> multiValued="false"/> >>>> <field name="filename" type="string" indexed="true" stored="true" >>>> multiValued="false"/> >>>> <field name="filesize" type="long" indexed="true" stored="true" >>>> multiValued="false"/> >>>> <field name="pversion" type="int" indexed="true" stored="true" >>>> multiValued="false"/> >>>> <field name="first2md5" type="string" indexed="false" stored="true" >>>> multiValued="false"/> >>>> <field name="ckey" type="string" indexed="true" stored="true" >>>> multiValued="false"/> >>>> >>>> ... >>>> >>>> <dynamicField name="*" type="ignored" multiValued="true" /> >>>> <defaultSearchField>body</defaultSearchField> >>>> <solrQueryParser defaultOperator="AND"/> >>>> >>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>> >>>> solrconfig.xml changes: >>>> >>>> <maxFieldLength>2147483647</maxFieldLength> >>>> <ramBufferSizeMB>128</ramBufferSizeMB> >>>> >>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>> >>>> The query: >>>> >>>> rowStr = "&rows=10" >>>> facet = >>>> "&facet=true&facet.limit=10&facet.field=device&facet.field=ckey&facet.field=version" >>>> fields = "&fl=id,score,filename,version,device,first2md5,filesize,ckey" >>>> termvectors = "&tv=true&qt=tvrh&tv.all=true" >>>> hl = "&hl=true&hl.fl=body&hl.snippets=1&hl.fragsize=400" >>>> regexv = "(?m)^.*\n.*\n.*$" >>>> hl_regex = "&hl.regex.pattern=" + CGI::escape(regexv) + >>>> "&hl.regex.slop=1&hl.fragmenter=regex&hl.regex.maxAnalyzedChars=2147483647&hl.maxAnalyzedChars=2147483647" >>>> justq = '&q=' + CGI::escape('body:' + fuzzy + p['q'].to_s.gsub(/\\/, >>>> '').gsub(/([:~!<>="])/,'\\\\\1') + fuzzy + minLogSizeStr) >>>> >>>> thequery = '/solr/select?timeAllowed=5000&wt=ruby' + (p['fq'].empty? ? '' >>>> : ('&fq='+p['fq'].to_s) ) + justq + rowStr + facet + fields + termvectors >>>> + hl + hl_regex >>>> >>>> baseurl = '/cgi-bin/search.rb?q=' + CGI::escape(p['q'].to_s) + '&rows=' + >>>> p['rows'].to_s + '&minLogSize=' + p['minLogSize'].to_s >>>> >>> >> >> > > -- > Sent from my mobile device
