Re: Solr queries slow down over time

2020-09-25 Thread Goutham Tholpadi
Hi Mark, Thanks for confirming Dwane's advice from your own experience. I will shift to a streaming expressions implementation. Best Goutham On Fri, Sep 25, 2020 at 7:03 PM Mark H. Wood wrote: > On Fri, Sep 25, 2020 at 11:49:22AM +0530, Goutham Tholpadi wrote: > > I have around 30M documents in

Re: Solr queries slow down over time

2020-09-25 Thread Goutham Tholpadi
Thanks a ton, Dwane. I went through the article and the documentation link. This corresponds exactly to my use case. Best Goutham On Fri, Sep 25, 2020 at 2:59 PM Dwane Hall wrote: > Goutham I suggest you read Hossman's excellent article on deep paging and > why returning rows=(some large number

Re: Solr queries slow down over time

2020-09-25 Thread Mark H. Wood
On Fri, Sep 25, 2020 at 11:49:22AM +0530, Goutham Tholpadi wrote: > I have around 30M documents in Solr, and I am doing repeated *:* queries > with rows=1, and changing start to 0, 1, 2, and so on, in a > loop in my script (using pysolr). > > At the start of the iteration, the calls to

Re: Solr queries slow down over time

2020-09-25 Thread Dwane Hall
Goutham I suggest you read Hossman's excellent article on deep paging and why returning rows=(some large number) is a bad idea. It provides an thorough overview of the concept and will explain it better than I ever could (https://lucidworks.com/post/coming-soon-to-solr-efficient-cursor-based-ite