Hi, A little history before i tell the actual issue. Please bare with me. In Dev lab with a single VM (2vCPU, 4gb RAM, 3Gb to JVM, JBoss5.1, JDK1.6.0.30) we used Solr 4.1 and indexed 250K documents with each document of avg size 22Kb - Totla index size is 3.6GB. Every thing works good. The only query that will be hitting this cluster is with *q=masterId:XXX* where masterId is our unique key defined in schema. All our fields are stored
Now for performance testing we requested our perf. lab to upgrade their Solr 3.5 to Solr 4.1. Performance Lab has 1 Master and 2 slaves. After upgrading To Solr 4.1 the first thing they observed is index size shrieked to 1/2 of Solr3.5 (This is good) They have the same machine Spec as Dev. As soon as we started hitting the Slaves with queries we saw CPU Spike upto 100% withing 2mins at 15QPS and the Application becomes sluggish to unresponsive. 1. The same machine had Solr 3.5 earlier word like charm and we took the QPS upto 150. But with 4.1 we are stuck below 15. 2. Even after we stop the load runner we still see CPU 100% for long time. 3. We also performed same query test by reverting 1 slave to Solr3.5 and second slave to 4.1. 3.5 had no issue and 4.1 was causing CPU 100% (Master was turned off so no replication) 4. Took thread dumps as to check whats consuming CPU. There were no dead locks and all threads are in Block State. of which most are as seen below this message. Burning ourselves for several hours so far we couldn't identify why 3.5 performs better than 4.1. Where as the only difference is solr and the related change required for 4.1 in schema and config file. Also we use MMapdirectory for both. Any advice or direction would be great. thanks Thread 32403: (state = BLOCKED) - java.util.AbstractList$Itr.hasNext() @bci=8, line=339 (Compiled frame; information may be imprecise) - org.apache.lucene.document.Document.getFields(java.lang.String) @bci=19, line=175 (Compiled frame) - org.apache.lucene.document.LazyDocument$LazyField.stringValue() @bci=38, line=107 (Compiled frame) - org.apache.solr.schema.FieldType.toExternal(org.apache.lucene.index.IndexableField) @bci=1, line=330 (Compiled frame) - org.apache.solr.schema.FieldType.toObject(org.apache.lucene.index.IndexableField) @bci=2, line=339 (Compiled frame) - org.apache.solr.response.BinaryResponseWriter$Resolver.getValue(org.apache.solr.schema.SchemaField, org.apache.lucene.index.IndexableField) @bci=120, line=223 (Compiled frame) - org.apache.solr.response.BinaryResponseWriter$Resolver.getDoc(org.apache.lucene.document.Document) @bci=76, line=186 (Compiled frame) - org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(org.apache.solr.response.ResultContext, org.apache.solr.common.util.JavaBinCodec) @bci=205, line=147 (Compiled frame) - org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(org.apache.solr.response.ResultContext, org.apache.solr.common.util.JavaBinCodec) @bci=126, line=173 (Compiled frame) - org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(java.lang.Object, org.apache.solr.common.util.JavaBinCodec) @bci=13, line=86 (Compiled frame) - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object) @bci=24, line=154 (Compiled frame) - org.apache.solr.common.util.JavaBinCodec.writeNamedList(org.apache.solr.common.util.NamedList) @bci=53, line=144 (Compiled frame) - org.apache.solr.common.util.JavaBinCodec.writeKnownType(java.lang.Object) @bci=22, line=234 (Compiled frame) - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object) @bci=2, line=149 (Compiled frame) - org.apache.solr.common.util.JavaBinCodec.marshal(java.lang.Object, java.io.OutputStream) @bci=20, line=92 (Compiled frame) - org.apache.solr.response.BinaryResponseWriter.write(java.io.OutputStream, org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=68, line=50 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.writeResponse(org.apache.solr.response.SolrQueryResponse, javax.servlet.ServletResponse, org.apache.solr.response.QueryResponseWriter, org.apache.solr.request.SolrQueryRequest, org.apache.solr.servlet.cache.Method) @bci=104, line=397 (Interpreted frame) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=867, line=282 (Interpreted frame) - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame) - org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame) - org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=61, line=96 (Compiled frame) - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame) - org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame) - org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=800, line=235 (Interpreted frame) - org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=363, line=191 (Compiled frame) - org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=386, line=190 (Compiled frame) - org.jboss.web.tomcat.security.JaccContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=117, line=92 (Interpreted frame) - org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response, org.jboss.servlet.http.HttpEvent) @bci=80, line=126 (Interpreted frame) - org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=4, line=70 (Interpreted frame) - org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=64, line=127 (Interpreted frame) - org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=6, line=102 (Compiled frame) - org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=35, line=158 (Compiled frame) - org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=42, line=109 (Compiled frame) - org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response) @bci=157, line=330 (Interpreted frame) - org.apache.coyote.ajp.AjpProcessor.process(java.net.Socket) @bci=322, line=436 (Compiled frame) - org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(java.net.Socket) @bci=39, line=384 (Interpreted frame) - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=447 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) -- View this message in context: http://lucene.472066.n3.nabble.com/Solr3-5-Vs-Solr4-1-Help-please-tp4043543.html Sent from the Solr - User mailing list archive at Nabble.com.
