Thanks for replying
I tried this "poor mans" cursor approach out ad-hoc, but I get OOM.
Pretty sure this is because you need all uniqueKey-values in FieldCache
in order to be able to sort on it. We do not have memory for that - and
never will. Our uniqueKey field is not DocValue.
Just out of curiosity
* Will I have the same OOM problem using the CURSOR-feature in later Solrs?
* Will the "poor mans" cursor approach still be efficient if my
uniqueKey was DocValued, knowing that all values for uniqueKey (the
DocValue file) cannot fit in memory (OS file cache)?
Regards, Per Steffensen
On 23/07/14 23:57, Chris Hostetter wrote:
: billions of documents (not enough memory). Please note that we are on 4.4,
: which does not contain the new CURSOR-feature. Please also note that speed is
: an important factor for us.
for situations where you know you will be processing every doc and order
doesn't matter you can use a "poor mans" cursor by filtering on sccessive
ranges of your uniqueKey field as described in the "Is There A
Workaround?" section of this blog post...
http://searchhub.org/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/
* sort on uniqueKey
* leave start=0 on every requets
* add an fq to each request based on the last uniqueKey value from
the previous request.
-Hoss
http://www.lucidworks.com/
