So I've read a bunch of stuff on hard/soft commits and tlogs. As I understand,
after a hard commit, solr is supposed to delete old tlogs depending on the
numRecordsToKeep and maxNumLogsToKeep values in the autocommit settings in
solrconfig.xml. I am occasionally seeing solr fail to do this and the tlogs
just build up over time and eventually we run out of disk space on the VM and
this causes problems for us. This does not happen all the time, only sometimes.
I currently have a tlog directory that has 123G worth of tlogs. The last hard
commit on this node was 10 minutes ago but these tlogs date back to 3 days ago.
We have sometimes found that restarting solr on the node will get it to clean
up the old tlogs, but we really want to find the root cause and fix it if
possible so we don't keep getting disk space alerts and have to adhoc restart
nodes. Has anyone seen an issue like this before?
My update handler settings look like this:
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog class="solr.CdcrUpdateLog">
<str name="dir">${solr.ulog.dir:}</str>
<int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
</updateLog>
<autoCommit>
<maxTime>600000</maxTime>
<maxDocs>250000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>120000</maxTime>
</autoSoftCommit>
<shardHandlerFactory class="HttpShardHandlerFactory">
<int name="maxConnectionsPerHost">100</int>
</shardHandlerFactory>
</updateHandler>
