Could this be just a simple case of a socket timeout? Can you raise the timout on request threads in Tomcat? It's a lot easier to reproduce/diagnose stuff like this when people use the stock jetty server shipped with Solr.
-Yonik http://lucidimagination.com On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > A signicant detail is the batch size which we set to 64 documents due to > earlier memory limitations. We index segments of roughly 300-500k records > each time. Lowering the batch size to 32 lead to an early internal server > error and the stack trace below. Increasing it to 128 allowed us to index > some more records but it still throws the error after 200k+ indexed records. > > Increasing it even more to 256 records per batch allowed us to index an > entire segment without errors. > > Another detail is that we do not restart the cluster between indexing > attempts so it seems that something only builds up during indexing (nothing > seems to leak afterwards) and throws an error. > > Any hints? > > Thanks, > Markus > > > > -----Original message----- >> From:Markus Jelsma <markus.jel...@openindex.io> >> Sent: Tue 07-Aug-2012 20:08 >> To: solr-user@lucene.apache.org >> Subject: null:java.lang.RuntimeException: [was class >> java.net.SocketTimeoutException] null >> >> Hello, >> >> We sometimes see the error below in our `master` when indexing. Our master >> is currently the node we send documents to - we've not yet implemented >> CloudSolrServer in Apache Nutch. This causes the indexer to crash when using >> Nutch locally, the task is retried when running on Hadoop. We're running it >> locally in this test set up so there's only one indexing thread. >> >> Anyway, for me it's quite a cryptic error because i don't know what >> connection has timed out, i assume a connection from the indexing node to >> some other node in the cluster when it passes a document to the correct >> leader? Each node of the 10 node cluster has the same configuration, Tomcat >> is configured with maxThreads=512 and a time out of one second. >> >> We're using today's trunk in this test set up and we cannot reliably >> reproduce the error. We've seen the error before so it's not a very recent >> issue. No errors are found in the other node's logs. >> >> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - >> [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class >> java.net.SocketTimeoutException] null >> at >> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18) >> at >> com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731) >> at >> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657) >> at >> com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809) >> at >> org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376) >> at >> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229) >> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157) >> at >> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) >> at >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) >> at >> io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) >> at >> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) >> at >> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) >> at >> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: java.net.SocketTimeoutException >> at >> org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185) >> at >> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229) >> at >> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210) >> at >> org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643) >> at >> org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945) >> at >> org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR >> ead(InternalNioInputBuffer.java:969) >> at >> org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268) >> at >> org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167) >> at >> org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916) >> at org.apache.coyote.Request.doRead(Request.java:427) >> at >> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304) >> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419) >> at >> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327) >> at >> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162) >> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) >> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) >> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101) >> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) >> at >> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) >> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) >> at >> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) >> at >> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) >> at >> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) >> at >> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649) >> ... 24 more >> >> Any thoughts to share? Is this a bug? A misconfiguration? >> >> Thanks, >> Markus >>
