Lots of questions indeed :) 1. Total virtual machines: 3 2. Replication factor: 0 (don't have any replicas yet) 3. Each machine has 1 shard which has 20GB of data. So data for a collection is spread across 3 machines totalling to 60GB 4. Start solr: java -Xmx10000m -javaagent:newrelic/newrelic.jar -Dsolr.clustering.enabled=true -Dsolr.solr.home=multicore -Djetty.class.path=lib/ext/* " -Dbootstrap_conf=true -DnumShards=3 -DzkHost=localhost:2181 -jar start.jar" 5. Yes, all machines have 24GB RAM and 9GB heap. Separate process of ZK is running on these machine. 6. top screenshot: http://i.imgur.com/g6w9Bim.png
Thanks! On Tue, Apr 8, 2014 at 4:48 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 4/8/2014 5:30 PM, Utkarsh Sengar wrote: > > I see sudden drop in throughput once every 3-4 days. The "downtime" is > for > > about 2-6minutes and things stabilize after that. > > > > But I am not sure what is causing it the problem. > > > > I have 3 shards with 20GB of data on each shard. > > Solr dashboard: http://i.imgur.com/6RWT2Dj.png > > Newrelic graphs when during the downtime of about 4hours: > > http://i.imgur.com/9vhKiB2.png > > JVM memory graph says its normal: http://i.imgur.com/pAycgdC.png > > > > I thought it was GC pauses but it should be in the newrelic logs. > > > > How can I go about investigating this problem? I am running solr 4.4.0, I > > don't see a strong reason to upgrade yet. > > Lots of questions: > > How many total machines? What is your replicationFactor? Does each > machine have one shard replica and therefore 20GB of total index data, > or if you add up all the index directories for the cores on each > machine, is there more than 20GB of data? > > What options are you passing to your JVM when you start the servlet > container that runs Solr? > > The dashboard says that this machine has 24GB of RAM and a 9GB heap. Is > this the case for all machines? Is there any software other than Solr > on the machine? > > If it's a linux/unix machine, can you run top, press shift-M to sort by > memory, and grab a screenshot? If it's a Windows machine, a similar > list should be available in the task manager, but it must include all > processes for all users on the whole machine, and it would be best if it > showed virtual memory as well as private. > > Thanks, > Shawn > > -- Thanks, -Utkarsh