Re: SolrCloud commit process is too time consuming, even if documents are light

2013-07-30 Thread Mark Miller
I don't seem to be seeing a signifigant slowdown over time when I use the old defaults for merge threads and max merges. - Mark On Jul 25, 2013, at 10:17 AM, Mark Miller wrote: > I'm looking into some possible slow down after long indexing issues when I > get back from vacation. This could be

Re: SolrCloud commit process is too time consuming, even if documents are light

2013-07-25 Thread Mark Miller
I'm looking into some possible slow down after long indexing issues when I get back from vacation. This could be related. Very early guess though. Another thing you might try - Lucene recently changed the merge scheduler policy defaults (in 4.1) - it used to use up 3 threads to merge and have a

Re: SolrCloud commit process is too time consuming, even if documents are light

2013-07-25 Thread Radu Ghita
Forgot to attach server and solr configurations: SolrCloud 4.1, internal Zookeeper, 16 shards, custom java importer. Server: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, 32 cores, 192gb RAM, 10tb SSD and 50tb SAS memory On Thu, Jul 25, 2013 at 3:20 PM, Radu Ghita wrote: > > Hi, > > We are having

Re: SolrCloud commit process is too time consuming, even if documents are light

2013-07-25 Thread Jack Krupansky
Auto soft commit is great for real time access, but you need to do hard commits periodically or else the transaction log (which is what assures that soft commits are durable) gets too big - it needs to be replayed on startup and is used for real-time search. So, set the auto soft commit to the