Hi, Perhaps you could try to remove the luceneMatchVersion from the 3.5 solrconfig again and use same schema version as for 1.4; to more closely emulate behavior of 1.4. LuceneMatchVersion will modify several defaults. Quick way to see if any of these new defaults make a change.
-- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com Solr Training - www.solrtraining.com On 29. nov. 2011, at 23:22, Pawel Rog wrote: > On Tue, Nov 29, 2011 at 9:13 PM, Chris Hostetter > <hossman_luc...@fucit.org> wrote: >> >> Let's back up a minute and cover some basics... >> >> 1) You said that you built a brand new index on a brand new master server, >> using Solr 3.5 -- how do you build your indexes? did the source data >> change at all? does your new index have the same number of docs as your >> previous Solr 1.4 index? what does a directory listing (including file >> sizes) look like for both your old and new indexes? > > Yes, both indexes have same data. Indexes are build using some C++ > programm which reads data from database and inserts it into Solr > (using XML). Both indexes have about 8GB size and 18milions documents. > > >> 2) Did you try using your Solr 1.4 index (and configs) directly in Solr >> 3.5 w/o rebuilding from scratch? > > Yes I used the same configs in solr 1.4 and solr 3.5 (adding only line > about "luceneMatchVersion") > As I see in example of solr 3.5 in repository (solrconfig.xml) there > are not many diffrences. > >> 3) You said you build the new index on a new mmachine, but then you said >> you used a slave where the performanne was worse then Solr 1.4 "on the >> same machine" ... are you running both the Solr 1.4 and Solr 3.5 instances >> concurrently on your slave machine? How much physical ram is on that >> machine? what JVM options are using when running the Solr 3.5 instance? >> what servlet container are you using? > > Mayby I didn't wrote precisely enough. I have some machine on which > there is master node. I have second machine on which there is slave. I > tested solr 1.4 on that machine, then turned it off and turned on > solr-3.5. I have 36GB RAM on that machine. > On both - solr 1.4 and 3.5 configuration of JVM is the same, and the > same servlet container ... jetty-6 > > JVM options: -server -Xms12000m -Xmx12000m -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:NewSize=1500m -XX:ParallelGCThreads=8 > -XX:CMSInitiatingOccupancyFraction=60 > >> 4) what does your request handler configuration look like? do you have >> any default/invariant/appended request params? > > <requestHandler name="standard" class="solr.SearchHandler" default="true"> > <lst name="defaults"> > <str name="echoParams">explicit</str> > </lst> > </requestHandler> > <requestHandler name="/admin/" > class="org.apache.solr.handler.admin.AdminHandlers" /> > <requestHandler name="/replication" class="solr.ReplicationHandler" > > <lst name="slave"> > <!--fully qualified url for the replication handler of > master . It > is possible to pass on this as a request param for the > fetchindexommand--> > <str > name="masterUrl">http://${masterHost}:${masterPort}/solr-3.5/${solr.core.instanceDir}replication</str> > <str name="pollInterval">00:00:02</str> > <str name="httpConnTimeout">5000</str> > <str name="httpReadTimeout">10000</str> > </lst> > </requestHandler> > > >> 5) The descriptions youve given of how the performance has changed sound >> like you are doing concurrent load testing -- did you do cache warming >> before you >> started your testing? how many client threads are hitting the solr server >> at one time? > > Maybe I wasn't precise enough again. CPU on solr 1.4 was 200% and on > solr 3.5 1200% > yes there is cache warming. There are 50-100 client threads on both > 1.4 and 3.5. There are about 60 requests per second on 3.5 and on 1.4, > but on 3.5 responses are slower and CPU usage much higher. > >> 6) have you tried doing some basic manual testing to see how individual >> requests performe? ie: single client at a time, loading a URL, then >> request the same URL again to verify that your Solr caches are in use and >> the QTime is low. If you see slow respone times even when manually >> executing single requests at a time, have you tried using "debug=timing" >> to see which serach components are contributing the most to the slow >> QTimes? > > Most time is in org.apache.solr.handler.component.QueryComponent and > org.apache.solr.handler.component.DebugComponent in process. I didn't > comare individual request performance. > >> 7) What do the cache stats look like on your Solr 3.5 instance after >> you've done some of this timing testing? the output of... >> http://localhost:8983/solr/admin/mbeans?cat=CACHE&stats=true&wt=json&indent=true >> ...would be helpful. NOTE: you may need to add this to your solrconfig.xml >> for that URL to work... >> <requestHandler name="/admin/" class="solr.admin.AdminHandlers" />' >> > > Will check it :) > >> >> : in my last pos i mean >> : default operation AND >> : promoted - int >> : ending - int >> : b_count - int >> : name - text >> : cat1 - int >> : cat2 - int >> : >> : On Tue, Nov 29, 2011 at 7:54 PM, Pawel Rog <pawelro...@gmail.com> wrote: >> : > examples >> : > >> : > >> facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=0&q=name:(kurtka+skóry+brazowe42)&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50 >> : > >> : > >> facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=1350&q=name:naczepa&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50 >> : > >> : > >> facet=true&sort=promoted+desc,ending+asc,b_count+desc&facet.mincount=1&start=0&q=it_name:(miłosz+giedroyc)&facet.limit=500&facet.field=cat1&facet.field=cat2&wt=json&rows=50 >> : > >> : > default operation ANDpromoted - intending - intb_count - intname - >> : > textcat1 - intcat2 -int >> : > these are only few examples. almost all queries are much slower. there >> : > was about 60 searches per second on old and new version of solr. solr >> : > 1.4 reached 200% cpu utilization and solr 3.5 reached 1200% cpu >> : > utilization on same machine >> : > >> : > On Tue, Nov 29, 2011 at 7:05 PM, Yonik Seeley >> : > <yo...@lucidimagination.com> wrote: >> : >> On Tue, Nov 29, 2011 at 12:25 PM, Pawel <pawelmis...@gmail.com> wrote: >> : >>> I've build index on solr 1.4 some time ago (about 18milions documents, >> : >>> about 8GB). I need new features from newer version of solr, so i >> : >>> decided to upgrade solr version from 1.4 to 3.5. >> : >>> >> : >>> * I created new solr master on new physical machine >> : >>> * then I created new index using the same schema as in earlier version >> : >>> * then I indexed some slave, and start sending the same requests as >> : >>> earlier but to newer version of solr (3.5, but the same situation is >> : >>> on solr 3.4). >> : >>> >> : >>> The CPU went from 200% to 1200% and load went from 3 to 15. Avarage >> : >>> QTime went from 15ms to 180ms and median went from 1ms to 150ms >> : >>> I didn't change any parameters in solrconfig and schema. >> : >> >> : >> What are the requests that look slower? >> : >> >> : >> -Yonik >> : >> http://www.lucidimagination.com >> : >> >> -Hoss
