I really think you need to think about firing successive page requests at the index and reporting in chunks.
Best Erick On Mon, Aug 27, 2012 at 2:56 PM, neosky <neosk...@yahoo.com> wrote: > I am using Solr 3.5 and Jetty 8.12 > I need to pull out huge query results at a time(for example, 1 million > documents, probably a couple gigabytes size) and my machine is about 64 G > memory. > I use the java bin and SolrJ as my client. And I use a Servelt to provide a > query down service for the end user. However, when I pull out the result at > a time, it fails. > solrQuery.setStart(0); > solrQuery.setRows(totalNumber);// the totalNumber sometimes is 1 million) > logs: > Aug 27, 2012 2:34:35 PM org.apache.solr.common.SolrException log > SEVERE: org.eclipse.jetty.io.EofException > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:422) > at > org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:512) > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:159) > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:101) > at > org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) > at > org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) > at > org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) > at > org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) > ... > > I am not sure where is the bottleneck. I tried to increase the timeout > solrServer.setSoTimeout(300000); > solrServer.setConnectionTimeout(3000000); > solrServer.setDefaultMaxConnectionsPerHost(100); > solrServer.setMaxTotalConnections(300); > > I also tried to increase the cached documents in the Solr configuration > <queryResultMaxDocsCached>20000</queryResultMaxDocsCached> > > It doesn't work at all. Any advice will be appreciated! > > Btw: I want to use the compression, but I don't know how it works. Because > after my Java client pull out the result, I need to printer out to the end > user as a download file. > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Fail-to-huge-collection-extraction-tp4003559.html > Sent from the Solr - User mailing list archive at Nabble.com.
