Let me clarify - DIH is running on Solr 6.5.0 that calls a different solr instance running on 4.5.0, which has 150M documents. If we try fetch them using DIH onto new solr cluster, wouldn't it result in deep paging on solr 4.5.0 and drastically slow down indexing on solr 6.5.0?
On Thu, Apr 27, 2017 at 4:40 PM, Erick Erickson <erickerick...@gmail.com> wrote: > I'm unclear why DIH an deep paging are mixed. DIH is > indexing and deep paging is querying. > > If it's querying, consider cursorMark or the /export handler. > https://lucidworks.com/2013/12/12/coming-soon-to-solr- > efficient-cursor-based-iteration-of-large-result-sets/ > > If it's DIH, please explain a bit more. > > Best, > Erick > > On Thu, Apr 27, 2017 at 3:37 PM, Vijay Kokatnur > <kokatnur.vi...@gmail.com> wrote: > > We have a new solr 6.5.0 cluster, for which data is being imported via > DIH > > from another Solr cluster running version 4.5.0. > > > > This question comes back to deep paging, but we have observed that after > 30 > > minutes of querying the rate of processing goes down from 400/s to about > > 120/s. At that point it has processed only 500K of 1.3M docs. Is there > > any way to speed this up? > > > > And, I can't go back to the source for the data. > > > > -- > -- Best, Vijay
