Hi all,
I'm working with Solr 4.6.1 and I'm trying to tune my ingestion process.
The ingestion runs a big DB query and then does some ETL on it and inserts
via SolrJ.
I have a 4 node cluster with 1 shard per node running in Tomcat with
-Xmx=4096M. Each node has a separate instance of Zookeeper on it, plus the
ingestion server has one as well. The Solr servers have 8 cores and 64 Gb
of total RAM. The ingestion server is a VM with 8 Gb and 2 cores.
My ingestion code uses a few settings to control concurrency and batch size.
solr.update.batchSize=500
solr.threadCount=4
With this setup, I'm getting a lot of errors and the ingestion is taking
much longer than it should.
Every so often during the ingestion I get these errors on the Solr servers:
WARN shard1 - 2014-02-25 11:18:34.341;
org.apache.solr.update.UpdateLog$LogReplayer; Starting log replay
tlog{file=/usr/local/solr_shard1/productCatalog/data/tlog/tlog.0000000000000014074
refcount=2} active=true starting pos=776774
WARN shard1 - 2014-02-25 11:18:37.275;
org.apache.solr.update.UpdateLog$LogReplayer; Log replay finished.
recoveryInfo=RecoveryInfo{adds=4065 deletes=0 deleteByQuery=0 errors=0
positionOfStart=776774}
WARN shard1 - 2014-02-25 11:18:37.960; org.apache.solr.core.SolrCore;
[productCatalog] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
WARN shard1 - 2014-02-25 11:18:37.961; org.apache.solr.core.SolrCore;
[productCatalog] Error opening new searcher. exceeded limit of
maxWarmingSearchers=2, try again later.
WARN shard1 - 2014-02-25 11:18:37.961; org.apache.solr.core.SolrCore;
[productCatalog] Error opening new searcher. exceeded limit of
maxWarmingSearchers=2, try again later.
ERROR shard1 - 2014-02-25 11:18:37.961;
org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try
again later.
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1575)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1346)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:592)
I cut threads down to 1 and batchSize down to 100 and the errors go away,
but the upload time jumps up by a factor of 25.
My solrconfig.xml has:
<autoCommit>
<maxDocs>${solr.autoCommit.maxDocs:10000}</maxDocs>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:1000}</maxTime>
</autoSoftCommit>
I turned autowarmCount down to 0 for all the caches. What else can I tune
to allow me to run bigger batch sizes and more threads in my upload script?
--
joel cohen, senior system engineer
e joel.co...@bluefly.com p 212.944.8000 x276
bluefly, inc. 42 w. 39th st. new york, ny 10018
www.bluefly.com <http://www.bluefly.com/?referer=autosig> | *fly since
2013...*
