Re: How to migrate content of a collection to a new collection

2014-07-24 Thread Chris Hostetter
: I tried this "poor mans" cursor approach out ad-hoc, but I get OOM. Pretty : sure this is because you need all uniqueKey-values in FieldCache in order to : be able to sort on it. We do not have memory for that - and never will. Our : uniqueKey field is not DocValue. : Just out of curiosity : * W

Re: How to migrate content of a collection to a new collection

2014-07-24 Thread Per Steffensen
Thanks for replying I tried this "poor mans" cursor approach out ad-hoc, but I get OOM. Pretty sure this is because you need all uniqueKey-values in FieldCache in order to be able to sort on it. We do not have memory for that - and never will. Our uniqueKey field is not DocValue. Just out of

Re: How to migrate content of a collection to a new collection

2014-07-24 Thread Per Steffensen
On 23/07/14 17:13, Erick Erickson wrote: Per: Given that you said that the field redefinition also includes routing info Exactly. It would probably be much faster to make sure that the new collection have the same number of shards on each Solr-machine and that the routing-ranges are identi

Re: How to migrate content of a collection to a new collection

2014-07-23 Thread Chris Hostetter
: billions of documents (not enough memory). Please note that we are on 4.4, : which does not contain the new CURSOR-feature. Please also note that speed is : an important factor for us. for situations where you know you will be processing every doc and order doesn't matter you can use a "poor m

Re: How to migrate content of a collection to a new collection

2014-07-23 Thread Erick Erickson
Per: Given that you said that the field redefinition also includes routing info I don't see any other way than re-indexing each collection. That said, could you use the collection aliasing and do one collection at a time? Best, Erick On Tue, Jul 22, 2014 at 11:45 PM, Per Steffensen wrote:

How to migrate content of a collection to a new collection

2014-07-22 Thread Per Steffensen
Hi We have numerous collections each with numerous shards spread across numerous machines. We just discovered that all documents have a field with a wrong value and besides that we would like to add a new field to all documents * The field with the wrong value is a long, DocValued, Indexed and