Hi Shawn We tried one more round of testing after We increased CPU cores on each server from 4 to 16 which amounts to 64 cores in total against 4 slaves.But Still CPU Usage was higher.So We took the thread dumps on one of our slaves and found that threads were blocked. Have also attached them.
As far as document size is considered, we have only 1.5 GB index(only one) size amounting to 4 lakh docs on each server and query load in span of 10 minutes on each of slaves was distributed as slave 1- 1258 slave 2 - 512 slave 3 - 256 slave 4 - 1384 We are using Linux OS. As threads are blocked , our cpu is reaching to 90% even even when heap or os memory is not getting used at all. One of the BLOCKED Thread Snippet Most of the threads are blocked either on Jetty Connector or FieldCacheIMPL for DOC values. Slave1ThreadDump5.txt <http://lucene.472066.n3.nabble.com/file/t493418/Slave1ThreadDump5.txt> Slave1ThreadDump4.txt <http://lucene.472066.n3.nabble.com/file/t493418/Slave1ThreadDump4.txt> "qtp1205044462-26-acceptor-1@706f9d47-ServerConnector@7ea87d3a{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #26 prio=5 os_prio=0 tid=0x00007f2530501800 nid=0x4f9c waiting for monitor entry [0x00007f2505eb8000] java.lang.Thread.State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233) - waiting to lock <0x000000064067fc68> (a java.lang.Object) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
