On Tue, Nov 29, 2011 at 9:13 PM, Chris Hostetter
<hossman_luc...@fucit.org> wrote:
Let's back up a minute and cover some basics...
1) You said that you built a brand new index on a brand new master server,
using Solr 3.5 -- how do you build your indexes? did the source data
change at all? does your new index have the same number of docs as your
previous Solr 1.4 index? what does a directory listing (including file
sizes) look like for both your old and new indexes?
Yes, both indexes have same data. Indexes are build using some C++
programm which reads data from database and inserts it into Solr
(using XML). Both indexes have about 8GB size and 18milions documents.
2) Did you try using your Solr 1.4 index (and configs) directly in Solr
3.5 w/o rebuilding from scratch?
Yes I used the same configs in solr 1.4 and solr 3.5 (adding only line
about "luceneMatchVersion")
As I see in example of solr 3.5 in repository (solrconfig.xml) there
are not many diffrences.
3) You said you build the new index on a new mmachine, but then you said
you used a slave where the performanne was worse then Solr 1.4 "on the
same machine" ... are you running both the Solr 1.4 and Solr 3.5 instances
concurrently on your slave machine? How much physical ram is on that
machine? what JVM options are using when running the Solr 3.5 instance?
what servlet container are you using?
Mayby I didn't wrote precisely enough. I have some machine on which
there is master node. I have second machine on which there is slave. I
tested solr 1.4 on that machine, then turned it off and turned on
solr-3.5. I have 36GB RAM on that machine.
On both - solr 1.4 and 3.5 configuration of JVM is the same, and the
same servlet container ... jetty-6
JVM options: -server -Xms12000m -Xmx12000m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:NewSize=1500m -XX:ParallelGCThreads=8
-XX:CMSInitiatingOccupancyFraction=60
4) what does your request handler configuration look like? do you have
any default/invariant/appended request params?
<requestHandler name="standard" class="solr.SearchHandler" default="true">
<lst name="defaults">
<str name="echoParams">explicit</str>
</lst>
</requestHandler>
<requestHandler name="/admin/"
class="org.apache.solr.handler.admin.AdminHandlers" />
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="slave">
<!--fully qualified url for the replication handler of
master . It
is possible to pass on this as a request param for the
fetchindexommand-->
<str
name="masterUrl">http://${masterHost}:${masterPort}/solr-3.5/${solr.core.instanceDir}replication</str>
<str name="pollInterval">00:00:02</str>
<str name="httpConnTimeout">5000</str>
<str name="httpReadTimeout">10000</str>
</lst>
</requestHandler>
5) The descriptions youve given of how the performance has changed sound
like you are doing concurrent load testing -- did you do cache warming before
you
started your testing? how many client threads are hitting the solr server
at one time?
Maybe I wasn't precise enough again. CPU on solr 1.4 was 200% and on
solr 3.5 1200%
yes there is cache warming. There are 50-100 client threads on both
1.4 and 3.5. There are about 60 requests per second on 3.5 and on 1.4,
but on 3.5 responses are slower and CPU usage much higher.
6) have you tried doing some basic manual testing to see how individual
requests performe? ie: single client at a time, loading a URL, then
request the same URL again to verify that your Solr caches are in use and
the QTime is low. If you see slow respone times even when manually
executing single requests at a time, have you tried using "debug=timing"
to see which serach components are contributing the most to the slow
QTimes?
Most time is in org.apache.solr.handler.component.QueryComponent and
org.apache.solr.handler.component.DebugComponent in process. I didn't
comare individual request performance.
7) What do the cache stats look like on your Solr 3.5 instance after
you've done some of this timing testing? the output of...
http://localhost:8983/solr/admin/mbeans?cat=CACHE&stats=true&wt=json&indent=true
...would be helpful. NOTE: you may need to add this to your solrconfig.xml
for that URL to work...
<requestHandler name="/admin/" class="solr.admin.AdminHandlers" />'
Will check it :)
: in my last pos i mean
: default operation AND
: promoted - int
: ending - int
: b_count - int
: name - text
: cat1 - int
: cat2 - int
:
: On Tue, Nov 29, 2011 at 7:54 PM, Pawel Rog<pawelro...@gmail.com> wrote:
:> examples
:>
:>
facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=0&q=name:(kurtka+skóry+brazowe42)&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50
:>
:>
facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=1350&q=name:naczepa&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50
:>
:>
facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=0&q=it_name:(miłosz+giedroyc)&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50
:>
:> default operation ANDpromoted - intending - intb_count - intname -
:> textcat1 - intcat2 -int
:> these are only few examples. almost all queries are much slower. there
:> was about 60 searches per second on old and new version of solr. solr
:> 1.4 reached 200% cpu utilization and solr 3.5 reached 1200% cpu
:> utilization on same machine
:>
:> On Tue, Nov 29, 2011 at 7:05 PM, Yonik Seeley
:> <yo...@lucidimagination.com> wrote:
:>> On Tue, Nov 29, 2011 at 12:25 PM, Pawel<pawelmis...@gmail.com> wrote:
:>>> I've build index on solr 1.4 some time ago (about 18milions documents,
:>>> about 8GB). I need new features from newer version of solr, so i
:>>> decided to upgrade solr version from 1.4 to 3.5.
:>>>
:>>> * I created new solr master on new physical machine
:>>> * then I created new index using the same schema as in earlier version
:>>> * then I indexed some slave, and start sending the same requests as
:>>> earlier but to newer version of solr (3.5, but the same situation is
:>>> on solr 3.4).
:>>>
:>>> The CPU went from 200% to 1200% and load went from 3 to 15. Avarage
:>>> QTime went from 15ms to 180ms and median went from 1ms to 150ms
:>>> I didn't change any parameters in solrconfig and schema.
:>>
:>> What are the requests that look slower?
:>>
:>> -Yonik
:>> http://www.lucidimagination.com
:
-Hoss
