A lot of collection time, we also found that there are a lot of time_wait thread, mainly committed submit thread and search thread, which led to the rapid decline in the speed of solr, the number of these threads up to more than 2,000. I didn't have a solution to this problem, but I found that reloading would reduce the number of commit threads.
<https://mailtrack.io/> Sent with Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality&;> 2018-04-04 13:46 GMT+08:00 Doss <itsmed...@gmail.com>: > We have SOLR(7.0.1) cloud 3 VM Linux instances wit 4 CPU, 90 GB RAM with > zookeeper (3.4.11) ensemble running on the same machines. We have 130 cores > of overall size of 45GB. No Sharding, almost all VMs has the same copy of > data. These nodes are under LB. > > Index Config: > ============= > > <ramBufferSizeMB>300</ramBufferSizeMB> > <mergePolicyFactory class="org.apache.solr.index. > TieredMergePolicyFactory"> > <int name="maxMergeAtOnce">30</int> > <int name="maxMergeAtOnceExplicit">100</int> > <double name="segmentsPerTier">30.0</double> > </mergePolicyFactory> > > <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> > <int name="maxMergeCount">18</int> > <int name="maxThreadCount">6</int> > </mergeScheduler> > > Commit Configs: > =============== > <autoCommit> > <maxTime>${solr.autoCommit.maxTime:600000}</maxTime> > <openSearcher>false</openSearcher> > </autoCommit> > > <autoSoftCommit> > <maxTime>${solr.autoSoftCommit.maxTime:60000}</maxTime> > </autoSoftCommit> > > > We do 3500 Insert / Updates per second spread across all 130 cores, We yet > to start using selects effectively. > > The problem what we are facing is at times suddenly the thread count > increase heavily which results SOLR non responsive or throwing 503 response > for client (PHP HTTP CURL) requests. > > Today 04-04-2018 the thread dump shows that the peak went upto 13000+ > > Please hlep me in fixing this issue. Thanks! > > > Sample Threads: > =============== > > 1.updateExecutor-2-thread-25746-processing-http://// > 172.10.2.19:8983//solr//profileviews x:profileviews r:core_node2 > n:172.10.2.18:8983_solr s:shard1 c:profileviews", "state":"TIMED_WAITING", > "lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject@297be1d5", > "cpuTime":"162.4371ms", "userTime":"120.0000ms", > "stackTrace":["sun.misc.Unsafe.park(Native Method)", > "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", > "java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)", > > 2. ERROR true HttpSolrCall > null:org.apache.solr.update.processor.DistributedUpdateProcessor$ > DistributedUpdatesAsyncException: > Async exception during distributed update: Error from server at > 172.10.2.18:8983/solr/profileviews: Server Error request: > http://172.10.2.18:8983/solr/profileviews/update?update. > distrib=TOLEADER&distrib.from=http%3A%2F%2F172.10.2.19% > 3A8983%2Fsolr%2Fprofileviews%2F&wt=javabin&version=2 > Remote error message: empty String > > 3. So Many Threads like: > "name":"qtp959447386-21", > "state":"TIMED_WAITING", > > "lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject@6a1a2bf4 > ", > "cpuTime":"4522.0837ms", > "userTime":"3770.0000ms", > "stackTrace":["sun.misc.Unsafe.park(Native Method)", > > "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", > > "java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)", > > "org.eclipse.jetty.util.BlockingArrayQueue.poll( > BlockingArrayQueue.java:392)", > > "org.eclipse.jetty.util.thread.QueuedThreadPool. > idleJobPoll(QueuedThreadPool.java:563)", > > "org.eclipse.jetty.util.thread.QueuedThreadPool. > access$800(QueuedThreadPool.java:48)", > > "org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > QueuedThreadPool.java:626)", > "java.lang.Thread.run(Thread.java:748)" > -- ============================== 联创科技 知行如一 ==============================
