: I have a requirement where I need to retrieve 10000 to 15000 records at a : time from SOLR. : With 20 or 100 records everything happens in milliseconds. : When it goes to 1000, 10000 it is taking more time... like even 30 seconds.
so far all you've really told us about your setup is that some queries with "rows=1000" are slow -- but you haven't really told us anything else we can help you with -- for example it's not obvious if you mean that you are using start=0 in all ofthose queries andthey are slow, or if you mean you are paginating through results (ie: increasing start param) 1000 at a time nad it starts getting slow as you page deeply. you also haven't told us anything about the fields you are returning -- how many are there?, what data types are they? are they large string values? how are you measuring the time? are you sure network lag, or client side processing of the data as solr returns it isn't the bulk of the time you are measuring? what does the QTime in the solr responses for these slow queries say? my best guesses are that either: you are doing deep paging and conflating the increased response time for deep results with an increase in response time for large rows params (because you are getting "deeper" faster with a large rows#) or you are seeing an increase in processing time on the client due ot the large volume of data being returned -- possibly even with SolrJ which is designed to parse the entire response into java data structures by default before returning to the client. w/o more concrete information, it's hard to give you advice beyond guesses. potentially helpful links... https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results https://lucidworks.com/blog/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/ https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions https://lucene.apache.org/solr/5_4_0/solr-solrj/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.html -Hoss http://www.lucidworks.com/
