Tom, It looks like the machine might simply be running too many things. If the load is around 1 when Solr is not running, and this is a dual-core server, it shows its already relatively busy (cca 50% idle). Your caches are not small, so I am guessing you either have to have a relatively big heap, or your heap is not large enough and it's the GC that's causing high CPU load. If you are seeing Solr causing lots of IO, that's a sign the box doesn't have enough memory for all those servers running comfortably on it.
Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR ----- Original Message ---- > From: Tomasz Kępski <tom...@kepski.pl> > To: solr <solr-user@lucene.apache.org> > Sent: Fri, November 20, 2009 3:15:08 PM > Subject: Huge load and long response times during search > > Hi, > > I'm using SOLR(1.4) to search among about 3,500,000 documents. After the > server > kernel was updated to 64bit system has started to suffer. > Our server has 8G of RAM and double Intel Core 2 DUO. > We used to have average loads around 2-2,5. It was not as good as it should > but > as long HTTP response times was acceptable we do not care to much ;-) > > Since few days avg loads are usually around 6, sometimes goes even to 20. > PHP, > Mysql and Postgresql based application is rather fine, but when tries to > access > SOLR it takes ages to load page. In top java process (Jetty) takes 200-250% > of > CPU, iotop shows that most of the disk operations are done by SOLR threads as > well. > > When we do shut down Jetty load goes down to 1,5 or even less than 1. > > My index has ~12G below is a part of my solrconf.xml: > > > 1024 > > class="solr.LRUCache" > size="16384" > initialSize="4096" > autowarmCount="4096"/> > > class="solr.LRUCache" > size="16384" > initialSize="4096" > autowarmCount="1024"/> > > class="solr.LRUCache" > size="16384" > initialSize="16384" > autowarmCount="0"/> > true > true > 40 > 200 > > > > solr 0 > name="rows">10 > solr price > name="start">0 10 > solr rekomendacja > name="start">0 10 > static newSearcher warming query from > solrconfig.xml > > > > > fast_warm 0 > name="rows">10 > static firstSearcher warming query from > solrconfig.xml > > > false > > > > > dismax > explicit > 0.01 > > name^90.0 scategory^450.0 brand^90.0 text^0.01 description^30 > > > > > > > brand,description,id,name,price,score > > > 4<100% 5<90% > > 100 > *:* > > > > sample query parameters from log looks like this: > > 2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/select > params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true} > > hits=3784 status=0 QTime=83 > 2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute > INFO: [] webapp=/solr path=/spellCheckCompRH > params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true} > > hits=3784 status=0 QTime=16 > > And at last the question ;-) > How to speed up the search? > Which parameters should I check first to find out what is the bottleneck? > > Sorry for verbose entry but I would like to give as clear point of view as > possible > > Thanks in advance, > Tom
