Re: DIH Speed

2017-04-27 Thread Vijay Kokatnur
​Let me clarify - DIH is running on Solr 6.5.0 that calls a different solr instance running​ on 4.5.0, which has 150M documents. If we try fetch them using DIH onto new solr cluster, wouldn't it result in deep paging on solr 4.5.0 and drastically slow down indexing on solr 6.5.0? On Thu, Apr 27,

Re: DIH Speed

2017-04-27 Thread Shawn Heisey
On 4/27/2017 9:15 PM, Vijay Kokatnur wrote: > Hey Shawn, Unfortunately, we can't upgrade the existing cluster. That > was my first approach as well. Yes, SolrEntityProcessor is used so it > results in deep paging after certain rows. I have observed that > instead of importing for a larger period, i

Re: DIH Speed

2017-04-27 Thread Vijay Kokatnur
:07 PM *To:* solr-user@lucene.apache.org *Subject:* Re: DIH Speed On 4/27/2017 5:40 PM, Erick Erickson wrote: > I'm unclear why DIH an deep paging are mixed. DIH is indexing and deep paging is querying. > > If it's querying, consider cursorMark or the /export handler. https://luc

Re: DIH Speed

2017-04-27 Thread Shawn Heisey
On 4/27/2017 5:40 PM, Erick Erickson wrote: > I'm unclear why DIH an deep paging are mixed. DIH is indexing and deep paging > is querying. > > If it's querying, consider cursorMark or the /export handler. > https://lucidworks.com/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-

Re: DIH Speed

2017-04-27 Thread Erick Erickson
I'm unclear why DIH an deep paging are mixed. DIH is indexing and deep paging is querying. If it's querying, consider cursorMark or the /export handler. https://lucidworks.com/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/ If it's DIH, please explain a bit