A signicant detail is the batch size which we set to 64 documents due to earlier memory limitations. We index segments of roughly 300-500k records each time. Lowering the batch size to 32 lead to an early internal server error and the stack trace below. Increasing it to 128 allowed us to index some more records but it still throws the error after 200k+ indexed records.
Increasing it even more to 256 records per batch allowed us to index an entire segment without errors. Another detail is that we do not restart the cluster between indexing attempts so it seems that something only builds up during indexing (nothing seems to leak afterwards) and throws an error. Any hints? Thanks, Markus -----Original message----- > From:Markus Jelsma <markus.jel...@openindex.io> > Sent: Tue 07-Aug-2012 20:08 > To: solr-user@lucene.apache.org > Subject: null:java.lang.RuntimeException: [was class > java.net.SocketTimeoutException] null > > Hello, > > We sometimes see the error below in our `master` when indexing. Our master is > currently the node we send documents to - we've not yet implemented > CloudSolrServer in Apache Nutch. This causes the indexer to crash when using > Nutch locally, the task is retried when running on Hadoop. We're running it > locally in this test set up so there's only one indexing thread. > > Anyway, for me it's quite a cryptic error because i don't know what > connection has timed out, i assume a connection from the indexing node to > some other node in the cluster when it passes a document to the correct > leader? Each node of the 10 node cluster has the same configuration, Tomcat > is configured with maxThreads=512 and a time out of one second. > > We're using today's trunk in this test set up and we cannot reliably > reproduce the error. We've seen the error before so it's not a very recent > issue. No errors are found in the other node's logs. > > 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - > [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class > java.net.SocketTimeoutException] null > at > com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18) > at > com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731) > at > com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657) > at > com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809) > at > org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376) > at > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229) > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157) > at > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) > at > io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > at > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.net.SocketTimeoutException > at > org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185) > at > org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229) > at > org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210) > at > org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643) > at > org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945) > at > org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR > ead(InternalNioInputBuffer.java:969) > at > org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268) > at > org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167) > at > org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916) > at org.apache.coyote.Request.doRead(Request.java:427) > at > org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304) > at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419) > at > org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327) > at > org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162) > at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) > at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) > at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101) > at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) > at > com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) > at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) > at > com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) > at > com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) > at > com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) > at > com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649) > ... 24 more > > Any thoughts to share? Is this a bug? A misconfiguration? > > Thanks, > Markus >
