Yes, expected. When it does a search for the first, say, 10 results, it must scan through all docs, recording just the highest ten scoring ones.
To find documents 1000 to 1010, it must scan through all docs, recording the best scoring 1010 documents, and then discard the first 1000. This is much more expensive. Try it on google, they won't let you go beyond around 900 pages or such (or is it 900 results?) Upayavira On Sat, Dec 8, 2012, at 01:10 AM, Petersen, Robert wrote: > Hi guys, > > > Sometimes we get a bot crawling our search function on our retail web > site. The ebay crawler loves to do this (Request.UserAgent: > Terapeakbot). They just do a star search and then iterate through page > after page. I've noticed that when they get to higher page numbers like > page 9000, the searches are taking more than 20 seconds. Is this > expected behavior? We're requesting standard facets with the search as > well as incorporating boosting by function query. Our index is almost 15 > million docs now and we're on Solr 3.6.1, this isn't causing any errors > to occur at the solr layer but our web layer times out the search after > 20 seconds and logs the exception. > > > > Thanks > > Robi >
