Hi, Could it be field compression that, I believe, is on by default?
Otis -- SOLR Performance Monitoring - http://sematext.com/spm/index.html On Wed, Feb 27, 2013 at 9:51 PM, adityab <aditya_ba...@yahoo.com> wrote: > Hi, > A little history before i tell the actual issue. Please bare with me. > In Dev lab with a single VM (2vCPU, 4gb RAM, 3Gb to JVM, JBoss5.1, > JDK1.6.0.30) we used Solr 4.1 and indexed 250K documents with each document > of avg size 22Kb - Totla index size is 3.6GB. > Every thing works good. The only query that will be hitting this cluster is > with *q=masterId:XXX* where masterId is our unique key defined in schema. > All our fields are stored > > Now for performance testing we requested our perf. lab to upgrade their > Solr > 3.5 to Solr 4.1. Performance Lab has 1 Master and 2 slaves. After upgrading > To Solr 4.1 the first thing they observed is index size shrieked to 1/2 of > Solr3.5 (This is good) They have the same machine Spec as Dev. > > As soon as we started hitting the Slaves with queries we saw CPU Spike upto > 100% withing 2mins at 15QPS and the Application becomes sluggish to > unresponsive. > 1. The same machine had Solr 3.5 earlier word like charm and we took the > QPS > upto 150. But with 4.1 we are stuck below 15. > 2. Even after we stop the load runner we still see CPU 100% for long time. > 3. We also performed same query test by reverting 1 slave to Solr3.5 and > second slave to 4.1. 3.5 had no issue and 4.1 was causing CPU 100% (Master > was turned off so no replication) > 4. Took thread dumps as to check whats consuming CPU. There were no dead > locks and all threads are in Block State. of which most are as seen below > this message. > > Burning ourselves for several hours so far we couldn't identify why 3.5 > performs better than 4.1. Where as the only difference is solr and the > related change required for 4.1 in schema and config file. Also we use > MMapdirectory for both. > > Any advice or direction would be great. > > thanks > > > Thread 32403: (state = BLOCKED) > - java.util.AbstractList$Itr.hasNext() @bci=8, line=339 (Compiled frame; > information may be imprecise) > - org.apache.lucene.document.Document.getFields(java.lang.String) @bci=19, > line=175 (Compiled frame) > - org.apache.lucene.document.LazyDocument$LazyField.stringValue() @bci=38, > line=107 (Compiled frame) > - > > org.apache.solr.schema.FieldType.toExternal(org.apache.lucene.index.IndexableField) > @bci=1, line=330 (Compiled frame) > - > > org.apache.solr.schema.FieldType.toObject(org.apache.lucene.index.IndexableField) > @bci=2, line=339 (Compiled frame) > - > > org.apache.solr.response.BinaryResponseWriter$Resolver.getValue(org.apache.solr.schema.SchemaField, > org.apache.lucene.index.IndexableField) @bci=120, line=223 (Compiled frame) > - > > org.apache.solr.response.BinaryResponseWriter$Resolver.getDoc(org.apache.lucene.document.Document) > @bci=76, line=186 (Compiled frame) > - > > org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(org.apache.solr.response.ResultContext, > org.apache.solr.common.util.JavaBinCodec) @bci=205, line=147 (Compiled > frame) > - > > org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(org.apache.solr.response.ResultContext, > org.apache.solr.common.util.JavaBinCodec) @bci=126, line=173 (Compiled > frame) > - > > org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(java.lang.Object, > org.apache.solr.common.util.JavaBinCodec) @bci=13, line=86 (Compiled frame) > - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object) > @bci=24, line=154 (Compiled frame) > - > > org.apache.solr.common.util.JavaBinCodec.writeNamedList(org.apache.solr.common.util.NamedList) > @bci=53, line=144 (Compiled frame) > - > org.apache.solr.common.util.JavaBinCodec.writeKnownType(java.lang.Object) > @bci=22, line=234 (Compiled frame) > - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object) > @bci=2, line=149 (Compiled frame) > - org.apache.solr.common.util.JavaBinCodec.marshal(java.lang.Object, > java.io.OutputStream) @bci=20, line=92 (Compiled frame) > - > org.apache.solr.response.BinaryResponseWriter.write(java.io.OutputStream, > org.apache.solr.request.SolrQueryRequest, > org.apache.solr.response.SolrQueryResponse) @bci=68, line=50 (Compiled > frame) > - > > org.apache.solr.servlet.SolrDispatchFilter.writeResponse(org.apache.solr.response.SolrQueryResponse, > javax.servlet.ServletResponse, > org.apache.solr.response.QueryResponseWriter, > org.apache.solr.request.SolrQueryRequest, > org.apache.solr.servlet.cache.Method) @bci=104, line=397 (Interpreted > frame) > - > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=867, > line=282 > (Interpreted frame) > - > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, > javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame) > - > > org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, > javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame) > - > > org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(javax.servlet.ServletRequest, > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=61, line=96 > (Compiled frame) > - > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, > javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame) > - > > org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, > javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame) > - > > org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=800, line=235 (Interpreted > frame) > - > > org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=363, line=191 (Compiled frame) > - > > org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=386, line=190 (Compiled frame) > - > > org.jboss.web.tomcat.security.JaccContextValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=117, line=92 (Interpreted > frame) > - > > org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response, org.jboss.servlet.http.HttpEvent) > @bci=80, line=126 (Interpreted frame) > - > > org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=4, line=70 (Interpreted frame) > - > > org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=64, line=127 (Interpreted > frame) > - > > org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=6, line=102 (Compiled frame) > - > > org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=35, line=158 (Compiled frame) > - > > org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, > org.apache.catalina.connector.Response) @bci=42, line=109 (Compiled frame) > - > > org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, > org.apache.coyote.Response) @bci=157, line=330 (Interpreted frame) > - org.apache.coyote.ajp.AjpProcessor.process(java.net.Socket) @bci=322, > line=436 (Compiled frame) > - > > org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(java.net.Socket) > @bci=39, line=384 (Interpreted frame) > - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=447 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr3-5-Vs-Solr4-1-Help-please-tp4043543.html > Sent from the Solr - User mailing list archive at Nabble.com. >
