First, thanks very much for a prompt reply. Here is more info:
===============
a) What operating system?
Debian GNU/Linux 5.0
b) What Java container (Tomcat/Jetty)
Jetty
c) What JAVA_OPTIONS? I.e. memory, garbage collection etc.
-Xmx9000m -DDEBUG -Djava.awt.headless=true
-Dorg.mortbay.log.class=org.mortbay.log.StdErrLog
-Dcom.sun.management.jmxremote.port=3000
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
-javaagent:/usr/local/lib/newrelic/newrelic.jar
d) Example queries? I.e. what features, how many facets, sort fields etc
/select?start=0&rows=20&fl=id&hl=true&hl.fl=title%2Cabstract%2Cauthors&hl.fragsize=300&hl.simple.pre=<strong>&hl.simple.post=<%2Fstrong>&qt=dismax&q=gene
therapy
We also get queries with filters examples:
/select?start=0&rows=20&fl=id&hl=true&hl.fl=title%2Cabstract%2Cauthors&hl.fragsize=300&hl.simple.pre=<strong>&hl.simple.post=<%2Fstrong>&qt=dismax&q=gene
therapy&fq=meshterm:(gene)&fq=author:(david)
e) How do you load balance queries between the slaves?
proxy based load balance
f) What is your search latency now and @ what QPS? Also, where do you
measure time - on the API or on the end-user page?
Average response time: 2600 - 3000 ms with average throughput: 4-6 rpm
(from 'new relic RPM' solr performance monitor)
g) How often do you replicate?
Daily (indexer runs each night) and replicates after indexing completes at
master. However lately we are experiencing problems right after replication,
and have to restart jetty (its most likely that slaves are running out of
memory).
h) Are you using warm-up-queries?
Yes, using autoWarmCount variable in cache configuration/ these are
specified as:
<filterCache class="solr.FastLRUCache" size="5000" initialSize="1000"
autowarmCount="500"/>
<queryResultCache class="solr.LRUCache" size="10000" initialSize="20000"
autowarmCount="20000"/>
<documentCache class="solr.LRUCache" size="10000" initialSize="10000"
autowarmCount="5000"/>
i) Are you ever optimizing your index?
Yes, daily after indexing. We are not doing dynamic updates to index, so I
guess its not needed to be done multiple times.
j) Are you using highlighting? If so, are you using the fast vector
highlighter or the regex?
Yes, we are using the default highlight component, with default fragmenter
called 'gap' and not regex. solr.highlight.GapFragmenter, with fragsize=300.
k) What other search components are you using?
spellcheck component, we will be using faceting in future soon.
i) Are you using RAID setup for the disks? If so, what kind of RAID, what
stripe-size and block size?
Yes, RAID-0:
$> cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 sda1[0] sdb1[1]
449225344 blocks 64k chunks
==============
I havn't benchmarked it yet as such, however here is the debugQuery
<section> from query results:
<lst name="debug">
<str name="rawquerystring">case study research</str>
<str name="querystring">case study research</str>
−
<str name="parsedquery">
+(DisjunctionMaxQuery((tags:case^1.2 | authors:case^7.5 | title:case^65.5 |
matchAll:case | keywords:case^2.5 | meshterm:case^3.2 |
abstract1:case^9.5)~0.01) DisjunctionMaxQuery((tags:studi^1.2 |
authors:study^7.5 | title:study^65.5 | matchAll:study | keywords:studi^2.5 |
meshterm:studi^3.2 | abstract1:studi^9.5)~0.01)
DisjunctionMaxQuery((tags:research^1.2 | authors:research^7.5 |
title:research^65.5 | matchAll:research | keywords:research^2.5 |
meshterm:research^3.2 | abstract1:research^9.5)~0.01))
DisjunctionMaxQuery((tags:"case studi research"~50^1.2 | authors:"case study
research"~50^7.5 | title:"case study research"~50^65.5 | matchAll:case study
research | keywords:"case studi research"~50^2.5 | meshterm:"case studi
research"~50^3.2 | abstract1:"case studi research"~50^9.5)~0.01)
FunctionQuery((sum(sdouble(yearScore)))^1.1)
FunctionQuery((sum(sdouble(readerScore)))^2.0)
</str>
−
<str name="parsedquery_toString">
+((tags:case^1.2 | authors:case^7.5 | title:case^65.5 | matchAll:case |
keywords:case^2.5 | meshterm:case^3.2 | abstract1:case^9.5)~0.01
(tags:studi^1.2 | authors:study^7.5 | title:study^65.5 | matchAll:study |
keywords:studi^2.5 | meshterm:studi^3.2 | abstract1:studi^9.5)~0.01
(tags:research^1.2 | authors:research^7.5 | title:research^65.5 |
matchAll:research | keywords:research^2.5 | meshterm:research^3.2 |
abstract1:research^9.5)~0.01) (tags:"case studi research"~50^1.2 |
authors:"case study research"~50^7.5 | title:"case study research"~50^65.5 |
matchAll:case study research | keywords:"case studi research"~50^2.5 |
meshterm:"case studi research"~50^3.2 | abstract1:"case studi
research"~50^9.5)~0.01 (sum(sdouble(yearScore)))^1.1
(sum(sdouble(readerScore)))^2.0
</str>
−
<lst name="explain">
−
<str name="7644c450-6d00-11df-a2b2-0026b95e3eb7">
9.473454 = (MATCH) sum of:
2.247054 = (MATCH) sum of:
0.7535966 = (MATCH) max plus 0.01 times others of:
0.7535966 = (MATCH) weight(title:case^65.5 in 6557735), product of:
0.29090396 = queryWeight(title:case^65.5), product of:
65.5 = boost
5.181068 = idf(docFreq=204956, maxDocs=13411507)
8.5721357E-4 = queryNorm
2.590534 = (MATCH) fieldWeight(title:case in 6557735), product of:
1.0 = tf(termFreq(title:case)=1)
5.181068 = idf(docFreq=204956, maxDocs=13411507)
0.5 = fieldNorm(field=title, doc=6557735)
0.5454388 = (MATCH) max plus 0.01 times others of:
0.5454388 = (MATCH) weight(title:study^65.5 in 6557735), product of:
0.24748746 = queryWeight(title:study^65.5), product of:
65.5 = boost
4.4078097 = idf(docFreq=444103, maxDocs=13411507)
8.5721357E-4 = queryNorm
2.2039049 = (MATCH) fieldWeight(title:study in 6557735), product of:
1.0 = tf(termFreq(title:study)=1)
4.4078097 = idf(docFreq=444103, maxDocs=13411507)
0.5 = fieldNorm(field=title, doc=6557735)
0.9480188 = (MATCH) max plus 0.01 times others of:
0.9480188 = (MATCH) weight(title:research^65.5 in 6557735), product
of:
0.32627863 = queryWeight(title:research^65.5), product of:
65.5 = boost
5.8110995 = idf(docFreq=109154, maxDocs=13411507)
8.5721357E-4 = queryNorm
2.9055498 = (MATCH) fieldWeight(title:research in 6557735), product
of:
1.0 = tf(termFreq(title:research)=1)
5.8110995 = idf(docFreq=109154, maxDocs=13411507)
0.5 = fieldNorm(field=title, doc=6557735)
6.6579494 = (MATCH) max plus 0.01 times others of:
6.6579494 = weight(title:"case study research"~50^65.5 in 6557735),
product of:
0.86467004 = queryWeight(title:"case study research"~50^65.5), product
of:
65.5 = boost
15.399977 = idf(title: case=204956 study=444103 research=109154)
8.5721357E-4 = queryNorm
7.6999884 = fieldWeight(title:"case study research" in 6557735),
product of:
1.0 = tf(phraseFreq=1.0)
15.399977 = idf(title: case=204956 study=444103 research=109154)
0.5 = fieldNorm(field=title, doc=6557735)
0.053200547 = (MATCH) FunctionQuery(sum(sdouble(yearScore))), product of:
56.420166 = sum(sdouble(yearScore)=56.42016783216783)
1.1 = boost
8.5721357E-4 = queryNorm
0.5152504 = (MATCH) FunctionQuery(sum(sdouble(readerScore))), product of:
300.53793 = sum(sdouble(readerScore)=300.5379289983797)
2.0 = boost
8.5721357E-4 = queryNorm
</str>
−
...
...
...
−
<str name="e3542c60-6d06-11df-afb8-0026b95d30b2">
9.212496 = (MATCH) sum of:
2.247054 = (MATCH) sum of:
0.7535966 = (MATCH) max plus 0.01 times others of:
0.7535966 = (MATCH) weight(title:case^65.5 in 12274669), product of:
0.29090396 = queryWeight(title:case^65.5), product of:
65.5 = boost
5.181068 = idf(docFreq=204956, maxDocs=13411507)
8.5721357E-4 = queryNorm
2.590534 = (MATCH) fieldWeight(title:case in 12274669), product of:
1.0 = tf(termFreq(title:case)=1)
5.181068 = idf(docFreq=204956, maxDocs=13411507)
0.5 = fieldNorm(field=title, doc=12274669)
0.5454388 = (MATCH) max plus 0.01 times others of:
0.5454388 = (MATCH) weight(title:study^65.5 in 12274669), product of:
0.24748746 = queryWeight(title:study^65.5), product of:
65.5 = boost
4.4078097 = idf(docFreq=444103, maxDocs=13411507)
8.5721357E-4 = queryNorm
2.2039049 = (MATCH) fieldWeight(title:study in 12274669), product
of:
1.0 = tf(termFreq(title:study)=1)
4.4078097 = idf(docFreq=444103, maxDocs=13411507)
0.5 = fieldNorm(field=title, doc=12274669)
0.9480188 = (MATCH) max plus 0.01 times others of:
0.9480188 = (MATCH) weight(title:research^65.5 in 12274669), product
of:
0.32627863 = queryWeight(title:research^65.5), product of:
65.5 = boost
5.8110995 = idf(docFreq=109154, maxDocs=13411507)
8.5721357E-4 = queryNorm
2.9055498 = (MATCH) fieldWeight(title:research in 12274669), product
of:
1.0 = tf(termFreq(title:research)=1)
5.8110995 = idf(docFreq=109154, maxDocs=13411507)
0.5 = fieldNorm(field=title, doc=12274669)
6.6579494 = (MATCH) max plus 0.01 times others of:
6.6579494 = weight(title:"case study research"~50^65.5 in 12274669),
product of:
0.86467004 = queryWeight(title:"case study research"~50^65.5), product
of:
65.5 = boost
15.399977 = idf(title: case=204956 study=444103 research=109154)
8.5721357E-4 = queryNorm
7.6999884 = fieldWeight(title:"case study research" in 12274669),
product of:
1.0 = tf(phraseFreq=1.0)
15.399977 = idf(title: case=204956 study=444103 research=109154)
0.5 = fieldNorm(field=title, doc=12274669)
0.030677302 = (MATCH) FunctionQuery(sum(sdouble(yearScore))), product of:
32.533848 = sum(sdouble(yearScore)=32.533846153846156)
1.1 = boost
8.5721357E-4 = queryNorm
0.27681494 = (MATCH) FunctionQuery(sum(sdouble(readerScore))), product of:
161.46207 = sum(sdouble(readerScore)=161.46207100162033)
2.0 = boost
8.5721357E-4 = queryNorm
</str>
</lst>
<str name="QParser">DisMaxQParser</str>
<null name="altquerystring"/>
−
<arr name="boostfuncs">
−
<str>
sum(readerScore)^2 sum(yearScore)^1.1
</str>
</arr>
−
<lst name="timing">
<double name="time">5468.0</double>
−
<lst name="prepare">
<double name="time">1.0</double>
−
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">1.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.SpellCheckComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
−
<lst name="process">
<double name="time">5467.0</double>
−
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">4734.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">231.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.SpellCheckComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">501.0</double>
</lst>
</lst>
</lst>
</lst>
=====================
Thanks for your help.
-Muneeb
--
View this message in context:
http://lucene.472066.n3.nabble.com/improving-search-response-time-tp1204491p1207596.html
Sent from the Solr - User mailing list archive at Nabble.com.
