Peter - Aaron just commented on a recent Solr issue (reading large result sets) and mentioned his patch. So far he has 2 x +1 from Grant and me to stick his patch in JIRA.
Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch ----- Original Message ---- > From: Peter Wolanin <peter.wola...@acquia.com> > To: solr-user@lucene.apache.org > Sent: Sun, January 3, 2010 3:37:01 PM > Subject: Re: SOLR Performance Tuning: Pagination > > At the NOVA Apache Lucene/Solr Meetup last May, one of the speakers > from Near Infinity (Aaron McCurry I think) mentioned that he had a > patch for lucene that enabled unlimited depth memory-efficient paging. > Is anyone in contact with him? > > -Peter > > On Thu, Dec 24, 2009 at 11:27 AM, Grant Ingersoll wrote: > > > > On Dec 24, 2009, at 11:09 AM, Fuad Efendi wrote: > > > >> I used pagination for a while till found this... > >> > >> > >> I have filtered query ID:[* TO *] returning 20 millions results (no > >> faceting), and pagination always seemed to be fast. However, fast only with > >> low values for start=12345. Queries like start=28838540 take 40-60 seconds, > >> and even cause OutOfMemoryException. > > > > Yeah, deep pagination in Lucene/Solr can be problematic due to the Priority > Queue management. See http://issues.apache.org/jira/browse/LUCENE-2127 and > the > linked discussion on java-dev. > > > >> > >> I use highlight, faceting on nontokenized "Country" field, standard > >> handler. > >> > >> > >> It even seems to be a bug... > >> > >> > >> Fuad Efendi > >> +1 416-993-2060 > >> http://www.linkedin.com/in/liferay > >> > >> Tokenizer Inc. > >> http://www.tokenizer.ca/ > >> Data Mining, Vertical Search > >> > >> > >> > >> > > > > -------------------------- > > Grant Ingersoll > > http://www.lucidimagination.com/ > > > > Search the Lucene ecosystem using Solr/Lucene: > http://www.lucidimagination.com/search > > > > > > > > -- > Peter M. Wolanin, Ph.D. > Momentum Specialist, Acquia. Inc. > peter.wola...@acquia.com
