Thanks Joel and Chris! I have around 121 fields out of which 12 of them are indexed and almost all 121 are stored. Average size of a doc is 10KB.
I was checking for start=0, rows=1000. We were querying a Solr instance which was on another server and I think network lag might have come into the picture also. I did not go for any caching as I wanted good response time in the first time querying itself. Thanks much for the links and suggestions. I will go thru each of them. Best, Mark. On Wed, Feb 17, 2016 at 5:26 PM, Chris Hostetter <hossman_luc...@fucit.org> wrote: > > : I have a requirement where I need to retrieve 10000 to 15000 records at a > : time from SOLR. > : With 20 or 100 records everything happens in milliseconds. > : When it goes to 1000, 10000 it is taking more time... like even 30 > seconds. > > so far all you've really told us about your setup is that some > queries with "rows=1000" are slow -- but you haven't really told us > anything else we can help you with -- for example it's not obvious if you > mean that you are using start=0 in all ofthose queries andthey are slow, > or if you mean you are paginating through results (ie: increasing start > param) 1000 at a time nad it starts getting slow as you page deeply. > > you also haven't told us anything about the fields you are returning -- > how many are there?, what data types are they? are they large string > values? > > how are you measuring the time? are you sure network lag, or client side > processing of the data as solr returns it isn't the bulk of the time you > are measuring? what does the QTime in the solr responses for these slow > queries say? > > my best guesses are that either: you are doing deep paging and conflating > the increased response time for deep results with an increase in response > time for large rows params (because you are getting "deeper" faster with a > large rows#) or you are seeing an increase in processing time on the > client due ot the large volume of data being returned -- possibly even > with SolrJ which is designed to parse the entire response into java > data structures by default before returning to the client. > > w/o more concrete information, it's hard to give you advice beyond > guesses. > > > potentially helpful links... > > https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results > > https://lucidworks.com/blog/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/ > > https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets > > https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions > > https://lucene.apache.org/solr/5_4_0/solr-solrj/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.html > > > > -Hoss > http://www.lucidworks.com/ >
