I read that blog too! Great info. I've bumped up the commit times and
turned the ingestion up a bit as well. I've upped hard commit to 5 minutes
and the soft commit to 60 seconds.
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:300000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:60000}</maxTime>
</autoSoftCommit>
I'm still getting the same issue. After speaking to the engineer working on
the ingestion code, he's told me that he's doing commits in his SolrJ code
every 1000 items (configurable). Does that override my Solr server settings?
On Tue, Feb 25, 2014 at 3:27 PM, Erick Erickson <erickerick...@gmail.com>wrote:
> Gopal: I'm glad somebody noticed that blog!
>
> Joel:
> For bulk loads it's a Good Thing to lengthen out
> your soft autocommit interval. A lot. Every second
> poor Solr is trying to open up a new searcher while
> you're throwing lots of documents at it. That's what's
> generating the "too many searchers" problem I'd
> guess. Soft commits are less expensive than hard
> commits with openSearcher=true (you're not doing this,
> and you shouldn't be). But soft commits aren't free.
> All the top-level caches are thrown away and autowarming
> is performed.....
>
> Also, I'd probably consider just leaving off the bit about
> maxDocs in your hard commit, I find it rarely does all
> that much good. After all, even if you have to replay the
> transaction log, you're only talking 15 seconds here.
>
> Best,
> Erick
>
>
> On Tue, Feb 25, 2014 at 12:08 PM, Gopal Patwa <gopalpa...@gmail.com>
> wrote:
>
> > This blog by Eric will help you to understand different commit option and
> > transaction logs and it does provide some recommendation for ingestion
> > process.
> >
> >
> >
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >
> >
> > On Tue, Feb 25, 2014 at 11:40 AM, Furkan KAMACI <furkankam...@gmail.com
> > >wrote:
> >
> > > Hi;
> > >
> > > You should read here:
> > >
> > >
> >
> http://wiki.apache.org/solr/FAQ#What_does_.22exceeded_limit_of_maxWarmingSearchers.3DX.22_mean.3F
> > >
> > > On the other hand do you have 4 Zookeeper instances as a quorum?
> > >
> > > Thanks;
> > > Furkan KAMACI
> > >
> > >
> > > 2014-02-25 20:31 GMT+02:00 Joel Cohen <joel.co...@bluefly.com>:
> > >
> > > > Hi all,
> > > >
> > > > I'm working with Solr 4.6.1 and I'm trying to tune my ingestion
> > process.
> > > > The ingestion runs a big DB query and then does some ETL on it and
> > > inserts
> > > > via SolrJ.
> > > >
> > > > I have a 4 node cluster with 1 shard per node running in Tomcat with
> > > > -Xmx=4096M. Each node has a separate instance of Zookeeper on it,
> plus
> > > the
> > > > ingestion server has one as well. The Solr servers have 8 cores and
> 64
> > Gb
> > > > of total RAM. The ingestion server is a VM with 8 Gb and 2 cores.
> > > >
> > > > My ingestion code uses a few settings to control concurrency and
> batch
> > > > size.
> > > >
> > > > solr.update.batchSize=500
> > > > solr.threadCount=4
> > > >
> > > > With this setup, I'm getting a lot of errors and the ingestion is
> > taking
> > > > much longer than it should.
> > > >
> > > > Every so often during the ingestion I get these errors on the Solr
> > > servers:
> > > >
> > > > WARN shard1 - 2014-02-25 11:18:34.341;
> > > > org.apache.solr.update.UpdateLog$LogReplayer; Starting log replay
> > > >
> > > >
> > >
> >
> tlog{file=/usr/local/solr_shard1/productCatalog/data/tlog/tlog.0000000000000014074
> > > > refcount=2} active=true starting pos=776774
> > > > WARN shard1 - 2014-02-25 11:18:37.275;
> > > > org.apache.solr.update.UpdateLog$LogReplayer; Log replay finished.
> > > > recoveryInfo=RecoveryInfo{adds=4065 deletes=0 deleteByQuery=0
> errors=0
> > > > positionOfStart=776774}
> > > > WARN shard1 - 2014-02-25 11:18:37.960;
> org.apache.solr.core.SolrCore;
> > > > [productCatalog] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
> > > > WARN shard1 - 2014-02-25 11:18:37.961;
> org.apache.solr.core.SolrCore;
> > > > [productCatalog] Error opening new searcher. exceeded limit of
> > > > maxWarmingSearchers=2, try again later.
> > > > WARN shard1 - 2014-02-25 11:18:37.961;
> org.apache.solr.core.SolrCore;
> > > > [productCatalog] Error opening new searcher. exceeded limit of
> > > > maxWarmingSearchers=2, try again later.
> > > > ERROR shard1 - 2014-02-25 11:18:37.961;
> > > > org.apache.solr.common.SolrException;
> > > org.apache.solr.common.SolrException:
> > > > Error opening new searcher. exceeded limit of maxWarmingSearchers=2,
> > try
> > > > again later.
> > > > at
> > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1575)
> > > > at
> > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1346)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:592)
> > > >
> > > > I cut threads down to 1 and batchSize down to 100 and the errors go
> > away,
> > > > but the upload time jumps up by a factor of 25.
> > > >
> > > > My solrconfig.xml has:
> > > >
> > > > <autoCommit>
> > > > <maxDocs>${solr.autoCommit.maxDocs:10000}</maxDocs>
> > > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> > > > <openSearcher>false</openSearcher>
> > > > </autoCommit>
> > > >
> > > > <autoSoftCommit>
> > > > <maxTime>${solr.autoSoftCommit.maxTime:1000}</maxTime>
> > > > </autoSoftCommit>
> > > >
> > > > I turned autowarmCount down to 0 for all the caches. What else can I
> > tune
> > > > to allow me to run bigger batch sizes and more threads in my upload
> > > script?
> > > >
> > > > --
> > > >
> > > > joel cohen, senior system engineer
> > > >
> > > > e joel.co...@bluefly.com p 212.944.8000 x276
> > > > bluefly, inc. 42 w. 39th st. new york, ny 10018
> > > > www.bluefly.com <http://www.bluefly.com/?referer=autosig> | *fly
> since
> > > > 2013...*
> > > >
> > >
> >
>
--
joel cohen, senior system engineer
e joel.co...@bluefly.com p 212.944.8000 x276
bluefly, inc. 42 w. 39th st. new york, ny 10018
www.bluefly.com <http://www.bluefly.com/?referer=autosig> | *fly since
2013...*
