Erick Thanks - My batch size was 30 and thread size also 30. Thanks On 8/17/16, 3:48 PM, "Erick Erickson" <erickerick...@gmail.com> wrote:
>What this probably indicates is that the size of the packets you send >to Solr is large enough that it exceeds the transport protocol's >limit. This is reinforced by your statement that reducing the batch >size fixes the problem even though it increases indexing time. > >So the place I'd be looking is the jetty configurations for any limits >there. > >That said, what is your batch size? In my testing I pretty quickly get >into diminishing returns, here's a writeup from some time ago: >https://lucidworks.com/blog/2015/10/05/really-batch-updates-solr-2/ > >Best, >Erick > >On Wed, Aug 17, 2016 at 12:03 PM, Jaspal Sawhney <jsawh...@sapient.com> >wrote: >> Bump ! >> >> On 8/16/16, 10:53 PM, "Jaspal Sawhney" <jsawh...@sapient.com> wrote: >> >>>Hello >>>We are running solr 4.6 in master-slave configuration where in our >>>master >>>is used entirely for indexing. No search traffic comes to master ever. >>>Off late we have started to get the early EOF error on the solr Master >>>which results in a Broken Pipe error on the commerce application from >>>where Indexing was kicked off from. >>> >>>Things to mention >>> >>> 1. We have a couple of sites each of which has the same document >>>size but diff document count. >>> 2. This error is being observed in the site which has the most number >>>of document count I.e. 2204743 >>> 3. The way I have understood solr to work is that irrespective of >>>number of document the throughput is controlled by the ŒNumber of >>>Threads¹ and ŒBatch size¹ - Am I correct? >>> * In our case we have not touched the batch size and Number of >>>Threads when this error started coming >>> * However when I do touch these parameters (specifically reduce >>>them) the error does not come however indexing time increases a lot. >>> 4. We have to index overnight daily because we put product prices in >>>the Index which get updated nightly >>> 5. Solr master is running with a 20 GB Heap >>> >>>What we have tried >>> >>> 1. I disabled autoCommit (I.e. Hard commit) and put the >>>autoSoftCommit >>>as 5 mins >>> * I realized afterwards that this was a wrong test because my >>>understanding of soft commit was incorrect, My understanding now is that >>>hard commit just truncate the Tlog do hardCommit should be better >>>indexing performance. >>> * This test failed for lack of space reason however because >>>disable autoCommit did not make sense I did not retry this test yet. >>> 2. Increased the RAMBufferSizeMB from 100MB to 1000MB >>> * This test did not yield anything favorable the master gave >>>the >>>early EOF exception >>> 3. Increased the merge factor from 20 ‹> 100 >>> * This test did not yield anything favorable the master gave >>>the >>>early EOF exception >>> 4. Flipped the autoCommit to 15 secs and disabled auto commit >>> * This test did not yield anything favorable the master gave >>>the >>>early EOF exception >>> * I got the input for this from >>>https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-so >>>ft >>>commit-and-commit-in-sorlcloud/ - Heavy (Bulk) Indexing section >>> 5. Tried to bypass transaction log all together This test is >>>underway currently >>> >>>Questions >>> >>> 1. Since we are not using solrCloud I want to understand the impact >>>of bypassing transaction log >>> 2. How does solr take documents which are sent to it to storage as in >>>what is the journey of a document from segment to tlog to storage. >>> >>>It would be great If there are any pointers which you can share. >>> >>>Thanks >>>J./ >>> >>>The actual Error Log >>>ERROR - 2016-08-16 22:59:55.988; org.apache.solr.common.SolrException; >>>org.apache.solr.common.SolrException: early EOF >>> at >>>org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176) >>> at >>>org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler >>>.j >>>ava:92) >>> at >>>org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Conte >>>nt >>>StreamHandlerBase.java:74) >>> at >>>org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB >>>as >>>e.java:135) >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) >>> at >>>org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja >>>va >>>:721) >>> at >>>org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j >>>av >>>a:417) >>> at >>>org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j >>>av >>>a:201) >>> at >>>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHan >>>dl >>>er.java:1419) >>> at >>>org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:45 >>>5) >>> at >>>org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java >>>:1 >>>37) >>> at >>>org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:5 >>>57 >>>) >>> at >>>org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler. >>>ja >>>va:231) >>> at >>>org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler. >>>ja >>>va:1075) >>> at >>>org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384 >>>) >>> at >>>org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.j >>>av >>>a:193) >>> at >>>org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.j >>>av >>>a:1009) >>> at >>>org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java >>>:1 >>>35) >>> at >>>org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Context >>>Ha >>>ndlerCollection.java:255) >>> at >>>org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollect >>>io >>>n.java:154) >>> at >>>org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.ja >>>va >>>:116) >>> at org.eclipse.jetty.server.Server.handle(Server.java:368) >>> at >>>org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHt >>>tp >>>Connection.java:489) >>> at >>>org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHt >>>tp >>>Connection.java:53) >>> at >>>org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConn >>>ec >>>tion.java:953) >>> at >>>org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(A >>>bs >>>tractHttpConnection.java:1014) >>> at >>>org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953) >>> at >>>org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) >>> at >>>org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConne >>>ct >>>ion.java:72) >>> at >>>org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(Socke >>>tC >>>onnector.java:264) >>> at >>>org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j >>>av >>>a:608) >>> at >>>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja >>>va >>>:543) >>> at java.lang.Thread.run(Thread.java:745) >>>Caused by: com.ctc.wstx.exc.WstxIOException: early EOF >>> at >>>com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708) >>> at >>>com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086) >>> at >>>org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:389) >>> at >>>org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:24 >>>6) >>> at >>>org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174) >>> ... 32 more >>>Caused by: org.eclipse.jetty.io.EofException: early EOF >>> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65) >>> at java.io.InputStream.read(InputStream.java:101) >>> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) >>> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) >>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101) >>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) >>> at >>>com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.jav >>>a: >>>57) >>> at >>>com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:104 >>>6) >>> at >>>com.ctc.wstx.sr.StreamScanner.parseLocalName2(StreamScanner.java:1796) >>> at >>>com.ctc.wstx.sr.StreamScanner.parseLocalName(StreamScanner.java:1756) >>> at >>>com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:2 >>>98 >>>1) >>> at >>>com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java >>>:2 >>>936) >>> at >>>com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:28 >>>48 >>>) >>> at >>>com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019) >>> ... 35 more >>> >>
