Bump ! On 8/16/16, 10:53 PM, "Jaspal Sawhney" <jsawh...@sapient.com> wrote:
>Hello >We are running solr 4.6 in master-slave configuration where in our master >is used entirely for indexing. No search traffic comes to master ever. >Off late we have started to get the early EOF error on the solr Master >which results in a Broken Pipe error on the commerce application from >where Indexing was kicked off from. > >Things to mention > > 1. We have a couple of sites each of which has the same document >size but diff document count. > 2. This error is being observed in the site which has the most number >of document count I.e. 2204743 > 3. The way I have understood solr to work is that irrespective of >number of document the throughput is controlled by the ŒNumber of >Threads¹ and ŒBatch size¹ - Am I correct? > * In our case we have not touched the batch size and Number of >Threads when this error started coming > * However when I do touch these parameters (specifically reduce >them) the error does not come however indexing time increases a lot. > 4. We have to index overnight daily because we put product prices in >the Index which get updated nightly > 5. Solr master is running with a 20 GB Heap > >What we have tried > > 1. I disabled autoCommit (I.e. Hard commit) and put the autoSoftCommit >as 5 mins > * I realized afterwards that this was a wrong test because my >understanding of soft commit was incorrect, My understanding now is that >hard commit just truncate the Tlog do hardCommit should be better >indexing performance. > * This test failed for lack of space reason however because >disable autoCommit did not make sense I did not retry this test yet. > 2. Increased the RAMBufferSizeMB from 100MB to 1000MB > * This test did not yield anything favorable the master gave the >early EOF exception > 3. Increased the merge factor from 20 ‹> 100 > * This test did not yield anything favorable the master gave the >early EOF exception > 4. Flipped the autoCommit to 15 secs and disabled auto commit > * This test did not yield anything favorable the master gave the >early EOF exception > * I got the input for this from >https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-soft >commit-and-commit-in-sorlcloud/ - Heavy (Bulk) Indexing section > 5. Tried to bypass transaction log all together This test is >underway currently > >Questions > > 1. Since we are not using solrCloud I want to understand the impact >of bypassing transaction log > 2. How does solr take documents which are sent to it to storage as in >what is the journey of a document from segment to tlog to storage. > >It would be great If there are any pointers which you can share. > >Thanks >J./ > >The actual Error Log >ERROR - 2016-08-16 22:59:55.988; org.apache.solr.common.SolrException; >org.apache.solr.common.SolrException: early EOF > at >org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176) > at >org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.j >ava:92) > at >org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Content >StreamHandlerBase.java:74) > at >org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBas >e.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) > at >org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java >:721) > at >org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav >a:417) > at >org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav >a:201) > at >org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl >er.java:1419) > at >org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > at >org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:1 >37) > at >org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557 >) > at >org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.ja >va:231) > at >org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.ja >va:1075) > at >org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > at >org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.jav >a:193) > at >org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.jav >a:1009) > at >org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:1 >35) > at >org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHa >ndlerCollection.java:255) > at >org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollectio >n.java:154) > at >org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java >:116) > at org.eclipse.jetty.server.Server.handle(Server.java:368) > at >org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttp >Connection.java:489) > at >org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttp >Connection.java:53) > at >org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnec >tion.java:953) > at >org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(Abs >tractHttpConnection.java:1014) > at >org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953) > at >org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) > at >org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnect >ion.java:72) > at >org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketC >onnector.java:264) > at >org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.jav >a:608) > at >org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java >:543) > at java.lang.Thread.run(Thread.java:745) >Caused by: com.ctc.wstx.exc.WstxIOException: early EOF > at >com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708) > at >com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086) > at >org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:389) > at >org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246) > at >org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174) > ... 32 more >Caused by: org.eclipse.jetty.io.EofException: early EOF > at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65) > at java.io.InputStream.read(InputStream.java:101) > at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) > at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) > at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101) > at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) > at >com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java: >57) > at >com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046) > at >com.ctc.wstx.sr.StreamScanner.parseLocalName2(StreamScanner.java:1796) > at >com.ctc.wstx.sr.StreamScanner.parseLocalName(StreamScanner.java:1756) > at >com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:298 >1) > at >com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:2 >936) > at >com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2848 >) > at >com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019) > ... 35 more >
