Two ideas: 1) monitor the GC activity with jvisualvm (comes with Oracle JDK), install a VisualGC plugin, it is quite helpful. The idea is to try to find the GC stop-the-world activities. If any found, look at tweaking the GC parameters. Some insight: http://wiki.apache.org/solr/ShawnHeisey Some more on tools for GC monitoring: http://architects.dzone.com/articles/how-monitor-java-garbage
2) monitor the network latency. Any possibility of the network being periodically congested? Can you plot a graph with number of concurrent (per second) queries versus their Qtimes ? On Tue, Jun 24, 2014 at 3:26 PM, RadhaJayalakshmi < rlakshminaraya...@inautix.co.in> wrote: > I am running Solr 4.5.1. Here is how my setup looks: > > Have 2 modest sized Collections. > Collection 1 - 2 shards, 3 replicas (Size of Shard 1 - 115 > MB, Size of Shard 2 - 55 MB) > Collection 2 - 2 shards, 3 replicas (Size of Shard 2 - 3.5 > GB, Size of Shard 2 - 1 GB) > These two collections are distributed across: > 6 Tomcat Nodes setup on 3 VMs (2 Nodes per VM) > Each of the 6 Tomcat nodes has a XmS / XmX setting of 2 GB > Each of the 3 VMs have a Physical Memory (RAM) of 32 GB > > As you can see my Collections are pretty small - This is actually a test > environment (and NOT Production), However my users (only have a handful of > testers) are complaining of sporadic performances issues on the Search. > > Here are my observations from the application logs: > 1) Out of 200 sample searches across both collections - 13 requests are > slow > (3 slow responses on Collection 1 and 10 slow responses on Collection 2). > > 2) When things run fast - they are really fast (Qtimes of 25 - 100 > milliseconds) - but when things are slow - I can see that the QTime > consistently hovers around the 5 second (or 5000 millisecond mark). I am > seeing responses of the order of 5024, 5094, 5035 ms - as though something > just hung for 5 seconds. I am observing this 5 second delay on both > Collections - which I feel is unusual - because both contain very different > data sets. I am unable to figure out whats causing the QTime to be so > consistent around the 5 second mark. > > 3) I build my index only once. I did try running an optimize on both > Collection 1 and Collection 2 after the users complained - I did notice > that > post the optimize the segment count on each of the four shards did come > down > - but that still didn't resolve the slowness on the searches (I was hoping > it would). > > 4) I am looking at the Solr Dashboard for more clues - My TomCat nodes are > definitely NOT running out of memory - the 6 nodes are consuming anywhere > between 500 MB - 1 GB RAM > > 5) The File Descriptor counts are under control - can only see a maximum of > 100 file descriptors being used of a total of 4096 > > 6) The Solr dashboard is however showing that 0.2% (or 9.8MB) of Swap Space > being consumed on one of the 3 VMs. Is this a concern ? > > 7) Also looked at the Plugin / Stats for every core on the Solr Dashboard. > I > can't see any evictions happening in any of the caches - Its always ZERO. > > Has anyone encountered such an issue ? What else should I be looking for to > debug my problem ? > > Thanks > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Slow-QTimes-5-seconds-for-Small-sized-Collections-tp4143681.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Dmitry Kan Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan SemanticAnalyzer: www.semanticanalyzer.info
