Re: Too many segments flushed (was: Too many commits)

2018-05-09 Thread Patrick Recchia
Thanks for the great info. Both of you. Yes, I know we have configuration issue with commitWithin and autoCommit. I cannot change our config freely, so this will probably stay like this for some time. I will try to indeed introduce autoSoftCommit (60 seconds is our maximus, as we try to achieve 'n

Re: Too many commits

2018-05-02 Thread Shawn Heisey
On 5/2/2018 11:45 AM, Patrick Recchia wrote: > Is there any logging I can turn on to know when a commit happens and/or > when a segment is flushed? The normal INFO-level logging that Solr ships with will log all commits.  It probably doesn't log segment flushes unless they happen as a result of a

Re: Too many commits

2018-05-02 Thread Erick Erickson
wn Heisey wrote: > >> On 5/2/2018 4:54 AM, Patrick Recchia wrote: >> > I'm seeing way too many commits on our solr cluster, and I don't know >> why. >> >> Are you sure there are commits happening? Do you have logs actually >> saying that a commi

Re: Too many commits

2018-05-02 Thread Patrick Recchia
Swawn, thanks you very much for your answer. On Wed, May 2, 2018 at 6:27 PM, Shawn Heisey wrote: > On 5/2/2018 4:54 AM, Patrick Recchia wrote: > > I'm seeing way too many commits on our solr cluster, and I don't know > why. > > Are you sure there are commits

Re: Too many commits

2018-05-02 Thread Shawn Heisey
On 5/2/2018 4:54 AM, Patrick Recchia wrote: > I'm seeing way too many commits on our solr cluster, and I don't know why. Are you sure there are commits happening?  Do you have logs actually saying that a commit is occurring?  The creation of a new segment does not necessarily

Re: Too many commits

2018-05-02 Thread Erick Erickson
, 2018 at 3:54 AM, Patrick Recchia wrote: > Hello, > > I'm seeing way too many commits on our solr cluster, and I don't know why. > > Here is the landscape: > - Each collection we create (one per day) is created with 10 shards with 2 > replicas each. > - we send li

Too many commits

2018-05-02 Thread Patrick Recchia
Hello, I'm seeing way too many commits on our solr cluster, and I don't know why. Here is the landscape: - Each collection we create (one per day) is created with 10 shards with 2 replicas each. - we send live data, 2B records / day. so on average 200M records/shard per day - for