IO waits about 0-2% Didn't see any suspicious activity in logs, but I can check it again
On Tue, Nov 29, 2011 at 11:40 PM, Darren Govoni <dar...@ontrenet.com> wrote: > Any suspicous activity in the logs? what about disk activity? > > > On 11/29/2011 05:22 PM, Pawel Rog wrote: >> >> On Tue, Nov 29, 2011 at 9:13 PM, Chris Hostetter >> <hossman_luc...@fucit.org> wrote: >>> >>> Let's back up a minute and cover some basics... >>> >>> 1) You said that you built a brand new index on a brand new master >>> server, >>> using Solr 3.5 -- how do you build your indexes? did the source data >>> change at all? does your new index have the same number of docs as your >>> previous Solr 1.4 index? what does a directory listing (including file >>> sizes) look like for both your old and new indexes? >> >> Yes, both indexes have same data. Indexes are build using some C++ >> programm which reads data from database and inserts it into Solr >> (using XML). Both indexes have about 8GB size and 18milions documents. >> >> >>> 2) Did you try using your Solr 1.4 index (and configs) directly in Solr >>> 3.5 w/o rebuilding from scratch? >> >> Yes I used the same configs in solr 1.4 and solr 3.5 (adding only line >> about "luceneMatchVersion") >> As I see in example of solr 3.5 in repository (solrconfig.xml) there >> are not many diffrences. >> >>> 3) You said you build the new index on a new mmachine, but then you said >>> you used a slave where the performanne was worse then Solr 1.4 "on the >>> same machine" ... are you running both the Solr 1.4 and Solr 3.5 >>> instances >>> concurrently on your slave machine? How much physical ram is on that >>> machine? what JVM options are using when running the Solr 3.5 instance? >>> what servlet container are you using? >> >> Mayby I didn't wrote precisely enough. I have some machine on which >> there is master node. I have second machine on which there is slave. I >> tested solr 1.4 on that machine, then turned it off and turned on >> solr-3.5. I have 36GB RAM on that machine. >> On both - solr 1.4 and 3.5 configuration of JVM is the same, and the >> same servlet container ... jetty-6 >> >> JVM options: -server -Xms12000m -Xmx12000m -XX:+UseParNewGC >> -XX:+UseConcMarkSweepGC -XX:NewSize=1500m -XX:ParallelGCThreads=8 >> -XX:CMSInitiatingOccupancyFraction=60 >> >>> 4) what does your request handler configuration look like? do you have >>> any default/invariant/appended request params? >> >> <requestHandler name="standard" class="solr.SearchHandler" default="true"> >> <lst name="defaults"> >> <str name="echoParams">explicit</str> >> </lst> >> </requestHandler> >> <requestHandler name="/admin/" >> class="org.apache.solr.handler.admin.AdminHandlers" /> >> <requestHandler name="/replication" class="solr.ReplicationHandler"> >> <lst name="slave"> >> <!--fully qualified url for the replication handler >> of master . It >> is possible to pass on this as a request param for the >> fetchindexommand--> >> <str >> name="masterUrl">http://${masterHost}:${masterPort}/solr-3.5/${solr.core.instanceDir}replication</str> >> <str name="pollInterval">00:00:02</str> >> <str name="httpConnTimeout">5000</str> >> <str name="httpReadTimeout">10000</str> >> </lst> >> </requestHandler> >> >> >>> 5) The descriptions youve given of how the performance has changed sound >>> like you are doing concurrent load testing -- did you do cache warming >>> before you >>> started your testing? how many client threads are hitting the solr >>> server >>> at one time? >> >> Maybe I wasn't precise enough again. CPU on solr 1.4 was 200% and on >> solr 3.5 1200% >> yes there is cache warming. There are 50-100 client threads on both >> 1.4 and 3.5. There are about 60 requests per second on 3.5 and on 1.4, >> but on 3.5 responses are slower and CPU usage much higher. >> >>> 6) have you tried doing some basic manual testing to see how individual >>> requests performe? ie: single client at a time, loading a URL, then >>> request the same URL again to verify that your Solr caches are in use and >>> the QTime is low. If you see slow respone times even when manually >>> executing single requests at a time, have you tried using "debug=timing" >>> to see which serach components are contributing the most to the slow >>> QTimes? >> >> Most time is in org.apache.solr.handler.component.QueryComponent and >> org.apache.solr.handler.component.DebugComponent in process. I didn't >> comare individual request performance. >> >>> 7) What do the cache stats look like on your Solr 3.5 instance after >>> you've done some of this timing testing? the output of... >>> >>> http://localhost:8983/solr/admin/mbeans?cat=CACHE&stats=true&wt=json&indent=true >>> ...would be helpful. NOTE: you may need to add this to your >>> solrconfig.xml >>> for that URL to work... >>> <requestHandler name="/admin/" class="solr.admin.AdminHandlers" />' >>> >> Will check it :) >> >>> : in my last pos i mean >>> : default operation AND >>> : promoted - int >>> : ending - int >>> : b_count - int >>> : name - text >>> : cat1 - int >>> : cat2 - int >>> : >>> : On Tue, Nov 29, 2011 at 7:54 PM, Pawel Rog<pawelro...@gmail.com> >>> wrote: >>> :> examples >>> :> >>> :> >>> facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=0&q=name:(kurtka+skóry+brazowe42)&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50 >>> :> >>> :> >>> facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=1350&q=name:naczepa&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50 >>> :> >>> :> >>> facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=0&q=it_name:(miłosz+giedroyc)&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50 >>> :> >>> :> default operation ANDpromoted - intending - intb_count - intname - >>> :> textcat1 - intcat2 -int >>> :> these are only few examples. almost all queries are much slower. >>> there >>> :> was about 60 searches per second on old and new version of solr. solr >>> :> 1.4 reached 200% cpu utilization and solr 3.5 reached 1200% cpu >>> :> utilization on same machine >>> :> >>> :> On Tue, Nov 29, 2011 at 7:05 PM, Yonik Seeley >>> :> <yo...@lucidimagination.com> wrote: >>> :>> On Tue, Nov 29, 2011 at 12:25 PM, Pawel<pawelmis...@gmail.com> >>> wrote: >>> :>>> I've build index on solr 1.4 some time ago (about 18milions >>> documents, >>> :>>> about 8GB). I need new features from newer version of solr, so i >>> :>>> decided to upgrade solr version from 1.4 to 3.5. >>> :>>> >>> :>>> * I created new solr master on new physical machine >>> :>>> * then I created new index using the same schema as in earlier >>> version >>> :>>> * then I indexed some slave, and start sending the same requests as >>> :>>> earlier but to newer version of solr (3.5, but the same situation >>> is >>> :>>> on solr 3.4). >>> :>>> >>> :>>> The CPU went from 200% to 1200% and load went from 3 to 15. Avarage >>> :>>> QTime went from 15ms to 180ms and median went from 1ms to 150ms >>> :>>> I didn't change any parameters in solrconfig and schema. >>> :>> >>> :>> What are the requests that look slower? >>> :>> >>> :>> -Yonik >>> :>> http://www.lucidimagination.com >>> : >>> >>> -Hoss > >
