One technique I've used to page through huge result sets that could
help: if you have a sortable key (like an id), you can just fetch all
docs, sorted by the key, and then on subsequent page requests use the
last value from the previous page as a filter in a range term like:
id:[<last-id> TO *]
where you substitute for <last-id>
there may be a better approach though...
-Mike
On 6/19/2011 6:02 PM, Hiller, Dean x66079 wrote:
As you probably know, using Query in hibernate/JPA gets slower and slower each
page since it starts all over on the index tree :( WHILE ScrollableResultSet
does NOT because the database maintains a cursor into the index that just picks
up where it left off so as you go to the next page, next page, the speed stays
linearly the same!!!!
Does something like that exist in solr?
I was looking at the api and all the examples are just for returning all
results from what I could tell.
I went into Lucene and it looks like it can do it kind of if you code up your
own Collector and unfortunately make the Collector.collect(int doc) block on a
lock while waiting for the client to ask for the next page(or ask to release
the resource since it is complete).
Ie. ScrollableResultSet obviously has to be closed when complete and so would
this method as well.
Any ideas on how to achieve this as my client is a computer not a webapp with a
human clicking next page and we want the resultset paging to be linear as it
really hurts our performance.
Thanks,
Dean
This message and any attachments are intended only for the use of the addressee
and
may contain information that is privileged and confidential. If the reader of
the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
