On 5/2/2018 4:54 AM, Patrick Recchia wrote:
> I'm seeing way too many commits on our solr cluster, and I don't know why.
Are you sure there are commits happening? Do you have logs actually
saying that a commit is occurring? The creation of a new segment does
not necessarily mean a commit happened -- this can happen even without a
commit.
> - IndexConfig is set to autoCommit every minute:
>
> <autoCommit> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime> <
> openSearcher>true</openSearcher> </autoCommit>
>
> (solr.autoCommit.maxTime is not set)
It's recommended to set openSearcher to false on autoCommit. Do you
have autoSoftCommit configured?
> There is nothing else customized (when it comes to IndexWriter, at least)
> within solrconfig.xml
>
> The data is sent without commit, but with commitWithin=500000 ms.
>
> All that said, I would have expected a rate of about 1 segment created epr
> minute; of about 100MB.
One of the events that can cause a new segment to be flushed is the ram
buffer filling up. Solr defaults to a ramBufferSizeMB value of 100.
But that does not translate to a segment size of 100MB -- it's merely
the size of the ram buffer that Lucene uses for all the work related to
building a segment. A segment resulting from a full memory buffer is
going to be smaller than the buffer. I do not know how MUCH smaller, or
what causes variations in that size.
The general advice is to leave the buffer size alone. But with the high
volume you've got, you might want to increase it so segments are not
flushed as frequently. Be aware that increasing it will have an impact
on how much heap memory gets used. Every Solr core (shard replica in
SolrCloud terminology) that does indexing is going to need one of these
ram buffers.
Thanks,
Shawn
