: so coming back to the issue .. even if am sorting it by _docid_ i need to do : paging( 2 million docs in result) : how is it internally doing it ? : when sorted by docid, don we have deep pagin issue ? (getting all the : previous pages into memory to get the next page) : so whats the main difference we are gaining by sorting lucene docids and : normal fields ?
when you sort on score (or some field) and ask for rows N - M, Lucene has to "collect" a priority queue of all the first" M docs in order to give you back N-M. This is where the "deep paging" issue can use up a lot of memory. I *think* that when you sort or _docid_ the Sort code doesn't build up a priority queue of all M docs, it just knows it should ignore the first N docs it sees and start keping track of them after N until it finds a total of M-N. But like i said before... : > >> *sort* on _docid_ asc, which should make all inherient issues with deep : > >> paging go away (as far as i know). At no point with the internal lucene ..."as far as i know" this should work, but i'm not certain. there may be other issues. (or i may be smocking crack that the sorting code works this way) -Hoss
