I am using Solr 3.5 and Jetty 8.12
I need to pull out huge query results at a time(for example, 1 million
documents, probably a couple gigabytes size) and my machine is about 64 G
memory.
I use the java bin and SolrJ as my client. And I use a Servelt to provide a
query down service for the end user. However, when I pull out the result at
a time, it fails.
solrQuery.setStart(0);
solrQuery.setRows(totalNumber);// the totalNumber sometimes is 1 million)
logs:
Aug 27, 2012 2:34:35 PM org.apache.solr.common.SolrException log
SEVERE: org.eclipse.jetty.io.EofException
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:422)
at
org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:512)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:159)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:101)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184)
at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89)
at
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336)
...
I am not sure where is the bottleneck. I tried to increase the timeout
solrServer.setSoTimeout(300000);
solrServer.setConnectionTimeout(3000000);
solrServer.setDefaultMaxConnectionsPerHost(100);
solrServer.setMaxTotalConnections(300);
I also tried to increase the cached documents in the Solr configuration
<queryResultMaxDocsCached>20000</queryResultMaxDocsCached>
It doesn't work at all. Any advice will be appreciated!
Btw: I want to use the compression, but I don't know how it works. Because
after my Java client pull out the result, I need to printer out to the end
user as a download file.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Fail-to-huge-collection-extraction-tp4003559.html
Sent from the Solr - User mailing list archive at Nabble.com.
