Re: Solr Cloud Bulk Indexing Questions

2014-01-23 Thread Software Dev
Does maxWriteMBPerSec apply to NRTCachingDirectoryFactory? I only see maxMergeSizeMB and maxCachedMB as configuration values. On Thu, Jan 23, 2014 at 11:05 AM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > Have you tried maxWriteMBPerSec? > > http://search-lucene.com/?q=maxWrit

Re: Solr Cloud Bulk Indexing Questions

2014-01-23 Thread Otis Gospodnetic
Hi, Have you tried maxWriteMBPerSec? http://search-lucene.com/?q=maxWriteMBPerSec&fc_project=Solr Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jan 20, 2014 at 4:00 PM, Software Dev wrote: > We are testing our shi

Re: Solr Cloud Bulk Indexing Questions

2014-01-23 Thread Shawn Heisey
On 1/23/2014 11:01 AM, Software Dev wrote: Is there any way to configure autoCommit, softCommit values on a per request basis? The majority of the time we have small flow of updates coming in and we would like to see them in ASAP. However we occasionally need to do some bulk indexing (once a week

Re: Solr Cloud Bulk Indexing Questions

2014-01-23 Thread Software Dev
Also, any suggestions on debugging? What should I look for and how? Thanks On Thu, Jan 23, 2014 at 10:01 AM, Software Dev wrote: > Thanks for suggestions. After reading that document I feel even more > confused though because I always thought that hard commits should be less > frequent that hard

Re: Solr Cloud Bulk Indexing Questions

2014-01-23 Thread Software Dev
Thanks for suggestions. After reading that document I feel even more confused though because I always thought that hard commits should be less frequent that hard commits. Is there any way to configure autoCommit, softCommit values on a per request basis? The majority of the time we have small flow

Re: Solr Cloud Bulk Indexing Questions

2014-01-22 Thread Erick Erickson
When you're doing hard commits, is it with openSeacher = true or false? It should probably be false... Here's a rundown of the soft/hard commit consequences: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ I suspect (but, of course, can't prove)

Re: Solr Cloud Bulk Indexing Questions

2014-01-22 Thread Software Dev
A suggestion would be to hard commit much less often, ie every 10 minutes, and see if there is a change. - Will try this How much system RAM ? JVM Heap ? Enough space in RAM for system disk cache ? - We have 18G of ram 12 dedicated to Solr but as of right now the total index size is only 5GB Ah

Re: Solr Cloud Bulk Indexing Questions

2014-01-22 Thread Andre Bois-Crettez
1 node having more load should be the leader (because of the extra work of receiving and distributing updates, but my experiences show only a bit more CPU usage, and no difference in disk IO). A suggestion would be to hard commit much less often, ie every 10 minutes, and see if there is a change.

Re: Solr Cloud Bulk Indexing Questions

2014-01-21 Thread Software Dev
Any other suggestions? On Mon, Jan 20, 2014 at 2:49 PM, Software Dev wrote: > 4.6.0 > > > On Mon, Jan 20, 2014 at 2:47 PM, Mark Miller wrote: > >> What version are you running? >> >> - Mark >> >> On Jan 20, 2014, at 5:43 PM, Software Dev >> wrote: >> >> > We also noticed that disk IO shoots up

Re: Solr Cloud Bulk Indexing Questions

2014-01-20 Thread Software Dev
4.6.0 On Mon, Jan 20, 2014 at 2:47 PM, Mark Miller wrote: > What version are you running? > > - Mark > > On Jan 20, 2014, at 5:43 PM, Software Dev > wrote: > > > We also noticed that disk IO shoots up to 100% on 1 of the nodes. Do all > > updates get sent to one machine or something? > > > > >

Re: Solr Cloud Bulk Indexing Questions

2014-01-20 Thread Mark Miller
What version are you running? - Mark On Jan 20, 2014, at 5:43 PM, Software Dev wrote: > We also noticed that disk IO shoots up to 100% on 1 of the nodes. Do all > updates get sent to one machine or something? > > > On Mon, Jan 20, 2014 at 2:42 PM, Software Dev > wrote: > >> We commit have a

Re: Solr Cloud Bulk Indexing Questions

2014-01-20 Thread Software Dev
We also noticed that disk IO shoots up to 100% on 1 of the nodes. Do all updates get sent to one machine or something? On Mon, Jan 20, 2014 at 2:42 PM, Software Dev wrote: > We commit have a soft commit every 5 seconds and hard commit every 30. As > far as docs/second it would guess around 200/s

Re: Solr Cloud Bulk Indexing Questions

2014-01-20 Thread Software Dev
We commit have a soft commit every 5 seconds and hard commit every 30. As far as docs/second it would guess around 200/sec which doesn't seem that high. On Mon, Jan 20, 2014 at 2:26 PM, Erick Erickson wrote: > Questions: How often do you commit your updates? What is your > indexing rate in docs/

Re: Solr Cloud Bulk Indexing Questions

2014-01-20 Thread Erick Erickson
Questions: How often do you commit your updates? What is your indexing rate in docs/second? In a SolrCloud setup, you should be using a CloudSolrServer. If the server is having trouble keeping up with updates, switching to CUSS probably wouldn't help. So I suspect there's something not optimal ab

Solr Cloud Bulk Indexing Questions

2014-01-20 Thread Software Dev
We are testing our shiny new Solr Cloud architecture but we are experiencing some issues when doing bulk indexing. We have 5 solr cloud machines running and 3 indexing machines (separate from the cloud servers). The indexing machines pull off ids from a queue then they index and ship over a docume