Jack, There are no peculiarities in the JVM graphs. Only increase in used threads and GC time. Heap space is collected quickly and doesn't suddenly increase. There's only 256MB available for the heap but it's fine.
Yonik, I'll increase the time out to five seconds tomorrow and try to reproduce it with a low batch size of 32. Juding from what i've seen it should throw an error quickly with such a low batch size. However, what is timing out here? My client connection to the indexing node or something else that i don't see? Unfortunately no Jetty here (yet). Thanks Markus -----Original message----- > From:Yonik Seeley <yo...@lucidimagination.com> > Sent: Tue 07-Aug-2012 23:54 > To: solr-user@lucene.apache.org > Subject: Re: null:java.lang.RuntimeException: [was class > java.net.SocketTimeoutException] null > > Could this be just a simple case of a socket timeout? Can you raise > the timout on request threads in Tomcat? > It's a lot easier to reproduce/diagnose stuff like this when people > use the stock jetty server shipped with Solr. > > -Yonik > http://lucidimagination.com > > > On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma > <markus.jel...@openindex.io> wrote: > > A signicant detail is the batch size which we set to 64 documents due to > > earlier memory limitations. We index segments of roughly 300-500k records > > each time. Lowering the batch size to 32 lead to an early internal server > > error and the stack trace below. Increasing it to 128 allowed us to index > > some more records but it still throws the error after 200k+ indexed records. > > > > Increasing it even more to 256 records per batch allowed us to index an > > entire segment without errors. > > > > Another detail is that we do not restart the cluster between indexing > > attempts so it seems that something only builds up during indexing (nothing > > seems to leak afterwards) and throws an error. > > > > Any hints? > > > > Thanks, > > Markus > > > > > > > > -----Original message----- > >> From:Markus Jelsma <markus.jel...@openindex.io> > >> Sent: Tue 07-Aug-2012 20:08 > >> To: solr-user@lucene.apache.org > >> Subject: null:java.lang.RuntimeException: [was class > >> java.net.SocketTimeoutException] null > >> > >> Hello, > >> > >> We sometimes see the error below in our `master` when indexing. Our master > >> is currently the node we send documents to - we've not yet implemented > >> CloudSolrServer in Apache Nutch. This causes the indexer to crash when > >> using Nutch locally, the task is retried when running on Hadoop. We're > >> running it locally in this test set up so there's only one indexing thread. > >> > >> Anyway, for me it's quite a cryptic error because i don't know what > >> connection has timed out, i assume a connection from the indexing node to > >> some other node in the cluster when it passes a document to the correct > >> leader? Each node of the 10 node cluster has the same configuration, > >> Tomcat is configured with maxThreads=512 and a time out of one second. > >> > >> We're using today's trunk in this test set up and we cannot reliably > >> reproduce the error. We've seen the error before so it's not a very recent > >> issue. No errors are found in the other node's logs. > >> > >> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - > >> [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class > >> java.net.SocketTimeoutException] null > >> at > >> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18) > >> at > >> com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731) > >> at > >> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657) > >> at > >> com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809) > >> at > >> org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376) > >> at > >> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229) > >> at > >> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157) > >> at > >> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) > >> at > >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) > >> at > >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) > >> at > >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) > >> at > >> io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219) > >> at > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > >> at > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > >> at > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > >> at > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > >> at > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > >> at > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > >> at > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > >> at > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > >> at > >> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > >> at > >> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) > >> at > >> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:662) > >> Caused by: java.net.SocketTimeoutException > >> at > >> org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185) > >> at > >> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229) > >> at > >> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210) > >> at > >> org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643) > >> at > >> org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945) > >> at > >> org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR > >> ead(InternalNioInputBuffer.java:969) > >> at > >> org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268) > >> at > >> org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167) > >> at > >> org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916) > >> at org.apache.coyote.Request.doRead(Request.java:427) > >> at > >> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304) > >> at > >> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419) > >> at > >> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327) > >> at > >> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162) > >> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) > >> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) > >> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101) > >> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) > >> at > >> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) > >> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) > >> at > >> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) > >> at > >> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) > >> at > >> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) > >> at > >> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649) > >> ... 24 more > >> > >> Any thoughts to share? Is this a bug? A misconfiguration? > >> > >> Thanks, > >> Markus > >> >
