Hello,
I'm seeing way too many commits on our solr cluster, and I don't know why.
Here is the landscape:
- Each collection we create (one per day) is created with 10 shards with 2
replicas each.
- we send live data, 2B records / day. so on average 200M records/shard per
day - for a size of approx 180GB/sahrd*Day.
on peak hours that makes approx 10M records/hour;
- so approx. 150000 records/minute. For a size of ~115MB/Minute?
- IndexConfig is set to autoCommit every minute:
<autoCommit> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime> <
openSearcher>true</openSearcher> </autoCommit>
(solr.autoCommit.maxTime is not set)
There is nothing else customized (when it comes to IndexWriter, at least)
within solrconfig.xml
The data is sent without commit, but with commitWithin=500000 ms.
All that said, I would have expected a rate of about 1 segment created epr
minute; of about 100MB.
Instead of that, I a lot of very small segments (between a few KB to a few
MB) with a very high rate.
And I have no idea why this would happen.
Where I can look to explain such a rate of segments being written?
--
One way of describing a computer is as an electric box which hums.
Never ascribe to malice what can be explained by stupidity
--
Patrick Recchia
GSM (BE): +32 486 828311
GSM(IT): +39 347 2300830
