Hello, Karl. Please check these: https://lucene.apache.org/solr/guide/6_6/pagination-of-results.html#constraints-when-using-cursors
https://lucene.apache.org/solr/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html#solrentityprocessor cursorMark="true" Good luck. On Wed, Feb 5, 2020 at 10:06 PM Karl Stoney <karl.sto...@autotrader.co.uk.invalid> wrote: > Hey All, > I'm trying to implement a simplistic reindex strategy to copy all of the > data out of one collection, into another, on a single node (no distributed > queries). > > It's approx 4 million documents, with an index size of 26gig. Based on > your experience, I'm wondering what people feel sensible values for the > SolrEntityProcessor are (to give me a sensible starting point, to save me > iterating over loads of them). > > This is where I'm at right now. I know `rows` would increase memory > pressure but speed up the copy, I can't really find anywhere online where > people have benchmarked different values for rows and the default (50) > seems quite low. > > <dataConfig> > <document> > <entity name="solr_doc" processor="SolrEntityProcessor" > query="*:*" > rows="100" > fl="*,old_version:_version_" > wt="javabin" > url="http://127.0.0.1/solr/at-uk";> > </entity> > </document> > </dataConfig> > > Any suggestions are welcome. > Thanks > This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: > 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England > No. 9439967). This email and any files transmitted with it are confidential > and may be legally privileged, and intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > email in error please notify the sender. This email message has been swept > for the presence of computer viruses. > -- Sincerely yours Mikhail Khludnev
