We're noticing a performance delta between 3.6 and 4.1 too. We're transitioning some textural fields from indexed only to stored as well.
One of the reasons why we're testing 4.1 in this particular case is more efficient storage use, which would eliminate some iowait behavior we were noticing. That seems to have worked, but still the 4.1 tests seem to be slower than we would expect. It's a little hard to say though, because we have two independent variables (3.6 -> 4.1, stored="false" -> stored="true"). If it were field compression, we'd expect to see some CPUs pegged, but it doesn't look that way to us.... No iowait (once the index is in cache), less than 50% CPU usage. Michael Della Bitta ------------------------------------------------ Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Thu, Feb 28, 2013 at 2:37 AM, Otis Gospodnetic <otis.gospodne...@gmail.com> wrote: > Hi, > > Could it be field compression that, I believe, is on by default? > > Otis > -- > SOLR Performance Monitoring - http://sematext.com/spm/index.html > > > > > > > On Wed, Feb 27, 2013 at 9:51 PM, adityab <aditya_ba...@yahoo.com> wrote: > >> Hi, >> A little history before i tell the actual issue. Please bare with me. >> In Dev lab with a single VM (2vCPU, 4gb RAM, 3Gb to JVM, JBoss5.1, >> JDK1.6.0.30) we used Solr 4.1 and indexed 250K documents with each document >> of avg size 22Kb - Totla index size is 3.6GB. >> Every thing works good. The only query that will be hitting this cluster is >> with *q=masterId:XXX* where masterId is our unique key defined in schema. >> All our fields are stored >> >> Now for performance testing we requested our perf. lab to upgrade their >> Solr >> 3.5 to Solr 4.1. Performance Lab has 1 Master and 2 slaves. After upgrading >> To Solr 4.1 the first thing they observed is index size shrieked to 1/2 of >> Solr3.5 (This is good) They have the same machine Spec as Dev. >> >> As soon as we started hitting the Slaves with queries we saw CPU Spike upto >> 100% withing 2mins at 15QPS and the Application becomes sluggish to >> unresponsive. >> 1. The same machine had Solr 3.5 earlier word like charm and we took the >> QPS >> upto 150. But with 4.1 we are stuck below 15. >> 2. Even after we stop the load runner we still see CPU 100% for long time. >> 3. We also performed same query test by reverting 1 slave to Solr3.5 and >> second slave to 4.1. 3.5 had no issue and 4.1 was causing CPU 100% (Master >> was turned off so no replication) >> 4. Took thread dumps as to check whats consuming CPU. There were no dead >> locks and all threads are in Block State. of which most are as seen below >> this message. >> >> Burning ourselves for several hours so far we couldn't identify why 3.5 >> performs better than 4.1. Where as the only difference is solr and the >> related change required for 4.1 in schema and config file. Also we use >> MMapdirectory for both. >> >> Any advice or direction would be great. >> >> thanks >> >> >> Thread 32403: (state = BLOCKED) >> - java.util.AbstractList$Itr.hasNext() @bci=8, line=339 (Compiled frame; >> information may be imprecise) >> - org.apache.lucene.document.Document.getFields(java.lang.String) @bci=19, >> line=175 (Compiled frame) >> - org.apache.lucene.document.LazyDocument$LazyField.stringValue() @bci=38, >> line=107 (Compiled frame) >> - >> >> org.apache.solr.schema.FieldType.toExternal(org.apache.lucene.index.IndexableField) >> @bci=1, line=330 (Compiled frame) >> - >> >> org.apache.solr.schema.FieldType.toObject(org.apache.lucene.index.IndexableField) >> @bci=2, line=339 (Compiled frame) >> - >> >> org.apache.solr.response.BinaryResponseWriter$Resolver.getValue(org.apache.solr.schema.SchemaField, >> org.apache.lucene.index.IndexableField) @bci=120, line=223 (Compiled frame) >> - >> >> org.apache.solr.response.BinaryResponseWriter$Resolver.getDoc(org.apache.lucene.document.Document) >> @bci=76, line=186 (Compiled frame) >> - >> >> org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(org.apache.solr.response.ResultContext, >> org.apache.solr.common.util.JavaBinCodec) @bci=205, line=147 (Compiled >> frame) >> - >> >> org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(org.apache.solr.response.ResultContext, >> org.apache.solr.common.util.JavaBinCodec) @bci=126, line=173 (Compiled >> frame) >> - >> >> org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(java.lang.Object, >> org.apache.solr.common.util.JavaBinCodec) @bci=13, line=86 (Compiled frame) >> - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object) >> @bci=24, line=154 (Compiled frame) >> - >> >> org.apache.solr.common.util.JavaBinCodec.writeNamedList(org.apache.solr.common.util.NamedList) >> @bci=53, line=144 (Compiled frame) >> - >> org.apache.solr.common.util.JavaBinCodec.writeKnownType(java.lang.Object) >> @bci=22, line=234 (Compiled frame) >> - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object) >> @bci=2, line=149 (Compiled frame) >> - org.apache.solr.common.util.JavaBinCodec.marshal(java.lang.Object, >> java.io.OutputStream) @bci=20, line=92 (Compiled frame) >> - >> org.apache.solr.response.BinaryResponseWriter.write(java.io.OutputStream, >> org.apache.solr.request.SolrQueryRequest, >> org.apache.solr.response.SolrQueryResponse) @bci=68, line=50 (Compiled >> frame) >> - >> >> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(org.apache.solr.response.SolrQueryResponse, >> javax.servlet.ServletResponse, >> org.apache.solr.response.QueryResponseWriter, >> org.apache.solr.request.SolrQueryRequest, >> org.apache.solr.servlet.cache.Method) @bci=104, line=397 (Interpreted >> frame) >> - >> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, >> javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=867, >> line=282 >> (Interpreted frame) >> - >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, >> javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame) >> - >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, >> javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame) >> - >> >> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(javax.servlet.ServletRequest, >> javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=61, line=96 >> (Compiled frame) >> - >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, >> javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame) >> - >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, >> javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame) >> - >> >> org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=800, line=235 (Interpreted >> frame) >> - >> >> org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=363, line=191 (Compiled frame) >> - >> >> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=386, line=190 (Compiled frame) >> - >> >> org.jboss.web.tomcat.security.JaccContextValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=117, line=92 (Interpreted >> frame) >> - >> >> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response, org.jboss.servlet.http.HttpEvent) >> @bci=80, line=126 (Interpreted frame) >> - >> >> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=4, line=70 (Interpreted frame) >> - >> >> org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=64, line=127 (Interpreted >> frame) >> - >> >> org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=6, line=102 (Compiled frame) >> - >> >> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=35, line=158 (Compiled frame) >> - >> >> org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, >> org.apache.catalina.connector.Response) @bci=42, line=109 (Compiled frame) >> - >> >> org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, >> org.apache.coyote.Response) @bci=157, line=330 (Interpreted frame) >> - org.apache.coyote.ajp.AjpProcessor.process(java.net.Socket) @bci=322, >> line=436 (Compiled frame) >> - >> >> org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(java.net.Socket) >> @bci=39, line=384 (Interpreted frame) >> - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=447 >> (Interpreted frame) >> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) >> >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Solr3-5-Vs-Solr4-1-Help-please-tp4043543.html >> Sent from the Solr - User mailing list archive at Nabble.com. >>
