Hi,
I'm using SOLR(1.4) to search among about 3,500,000 documents. After the
server kernel was updated to 64bit system has started to suffer.
Our server has 8G of RAM and double Intel Core 2 DUO.
We used to have average loads around 2-2,5. It was not as good as it
should but as long HTTP response times was acceptable we do not care to
much ;-)
Since few days avg loads are usually around 6, sometimes goes even to
20. PHP, Mysql and Postgresql based application is rather fine, but when
tries to access SOLR it takes ages to load page. In top java process
(Jetty) takes 200-250% of CPU, iotop shows that most of the disk
operations are done by SOLR threads as well.
When we do shut down Jetty load goes down to 1,5 or even less than 1.
My index has ~12G below is a part of my solrconf.xml:
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache
class="solr.LRUCache"
size="16384"
initialSize="4096"
autowarmCount="4096"/>
<queryResultCache
class="solr.LRUCache"
size="16384"
initialSize="4096"
autowarmCount="1024"/>
<documentCache
class="solr.LRUCache"
size="16384"
initialSize="16384"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<useFilterForSortedQuery>true</useFilterForSortedQuery>
<queryResultWindowSize>40</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<HashDocSet maxSize="3000" loadFactor="0.75"/>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst> <str name="q">solr</str> <str name="start">0</str> <str
name="rows">10</str> </lst>
<lst> <str name="q">solr</str> <str name="sort">price</str> <str
name="start">0</str> <str name="rows">10</str> </lst>
<lst> <str name="q">solr</str> <str
name="sort">rekomendacja</str> <str name="start">0</str> <str
name="rows">10</str> </lst>
<lst><str name="q">static newSearcher warming query from
solrconfig.xml</str></lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst> <str name="q">fast_warm</str> <str name="start">0</str>
<str name="rows">10</str> </lst>
<lst><str name="q">static firstSearcher warming query from
solrconfig.xml</str></lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
</query>
<requestHandler name="dismax" class="solr.SearchHandler" >
<lst name="defaults">
<str name="defType">dismax</str>
<str name="echoParams">explicit</str>
<float name="tie">0.01</float>
<str name="qf">
name^90.0 scategory^450.0 brand^90.0 text^0.01 description^30
</str>
<str name="pf">
</str>
<str name="bf">
</str>
<str name="fl">
brand,description,id,name,price,score
</str>
<str name="mm">
4<100% 5<90%
</str>
<int name="ps">100</int>
<str name="q.alt">*:*</str>
</lst>
</requestHandler>
sample query parameters from log looks like this:
2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true}
hits=3784 status=0 QTime=83
2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/spellCheckCompRH
params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true}
hits=3784 status=0 QTime=16
And at last the question ;-)
How to speed up the search?
Which parameters should I check first to find out what is the bottleneck?
Sorry for verbose entry but I would like to give as clear point of view
as possible
Thanks in advance,
Tom
