Hello
We are running solr 4.6 in master-slave configuration where in our master is
used entirely for indexing. No search traffic comes to master ever.
Off late we have started to get the early EOF error on the solr Master which
results in a Broken Pipe error on the commerce application from where Indexing
was kicked off from.
Things to mention
1. We have a couple of sites – each of which has the same document size but
diff document count.
2. This error is being observed in the site which has the most number of
document count I.e. 2204743
3. The way I have understood solr to work is that irrespective of number of
document – the throughput is controlled by the ‘Number of Threads’ and ‘Batch
size’ - Am I correct?
* In our case we have not touched the batch size and Number of Threads
when this error started coming
* However when I do touch these parameters (specifically reduce them)
the error does not come – however indexing time increases a lot.
4. We have to index overnight daily because we put product prices in the
Index which get updated nightly
5. Solr master is running with a 20 GB Heap
What we have tried
1. I disabled autoCommit (I.e. Hard commit) and put the autoSoftCommit as 5
mins
* I realized afterwards that this was a wrong test because my
understanding of soft commit was incorrect, My understanding now is that hard
commit just truncate the Tlog do hardCommit should be better indexing
performance.
* This test failed for lack of space reason however because disable
autoCommit did not make sense – I did not retry this test yet.
2. Increased the RAMBufferSizeMB from 100MB to 1000MB
* This test did not yield anything favorable – the master gave the early
EOF exception
3. Increased the merge factor from 20 —> 100
* This test did not yield anything favorable – the master gave the early
EOF exception
4. Flipped the autoCommit to 15 secs and disabled auto commit
* This test did not yield anything favorable – the master gave the early
EOF exception
* I got the input for this from
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
- Heavy (Bulk) Indexing section
5. Tried to bypass transaction log all together – This test is underway
currently
Questions
1. Since we are not using solrCloud – I want to understand the impact of
bypassing transaction log
2. How does solr take documents which are sent to it to storage as in what
is the journey of a document from segment to tlog to storage.
It would be great If there are any pointers which you can share.
Thanks
J./
The actual Error Log
ERROR - 2016-08-16 22:59:55.988; org.apache.solr.common.SolrException;
org.apache.solr.common.SolrException: early EOF
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ctc.wstx.exc.WstxIOException: early EOF
at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:389)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
... 32 more
Caused by: org.eclipse.jetty.io.EofException: early EOF
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
at java.io.InputStream.read(InputStream.java:101)
at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at
com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
at
com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
at
com.ctc.wstx.sr.StreamScanner.parseLocalName2(StreamScanner.java:1796)
at com.ctc.wstx.sr.StreamScanner.parseLocalName(StreamScanner.java:1756)
at
com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:2981)
at
com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:2936)
at
com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2848)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
... 35 more
