Two possibilities:
1> you have multiple replicas in the same JVM and are seeing commits
happen withall of them.
2> ramBufferSizeMB. when you index docs, segments are flushed when the
in-memory structures exceed this limit, is this perhaps what you're
seeing?
Best,
Erick
On Wed, May 2, 2018 at 3:54 AM, Patrick Recchia
<patrick.recc...@gmail.com> wrote:
> Hello,
>
> I'm seeing way too many commits on our solr cluster, and I don't know why.
>
> Here is the landscape:
> - Each collection we create (one per day) is created with 10 shards with 2
> replicas each.
> - we send live data, 2B records / day. so on average 200M records/shard per
> day - for a size of approx 180GB/sahrd*Day.
> on peak hours that makes approx 10M records/hour;
> - so approx. 150000 records/minute. For a size of ~115MB/Minute?
>
> - IndexConfig is set to autoCommit every minute:
>
> <autoCommit> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime> <
> openSearcher>true</openSearcher> </autoCommit>
>
> (solr.autoCommit.maxTime is not set)
>
> There is nothing else customized (when it comes to IndexWriter, at least)
> within solrconfig.xml
>
> The data is sent without commit, but with commitWithin=500000 ms.
>
> All that said, I would have expected a rate of about 1 segment created epr
> minute; of about 100MB.
>
> Instead of that, I a lot of very small segments (between a few KB to a few
> MB) with a very high rate.
>
> And I have no idea why this would happen.
> Where I can look to explain such a rate of segments being written?
>
>
>
>
>
> --
> One way of describing a computer is as an electric box which hums.
> Never ascribe to malice what can be explained by stupidity
> --
> Patrick Recchia
> GSM (BE): +32 486 828311
> GSM(IT): +39 347 2300830
