Are you sending updates in batches? Are you doing a commit after every update?
You should use batches and you should not commit after every update. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Apr 4, 2018, at 8:01 PM, 苗海泉 <mseaspr...@gmail.com> wrote: > > A lot of collection time, we also found that there are a lot of time_wait > thread, mainly committed submit thread and search thread, which led to the > rapid decline in the speed of solr, the number of these threads up to more > than 2,000. > I didn't have a solution to this problem, but I found that reloading would > reduce the number of commit threads. > > > > > <https://mailtrack.io/> Sent with Mailtrack > <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality&;> > > 2018-04-04 13:46 GMT+08:00 Doss <itsmed...@gmail.com>: > >> We have SOLR(7.0.1) cloud 3 VM Linux instances wit 4 CPU, 90 GB RAM with >> zookeeper (3.4.11) ensemble running on the same machines. We have 130 cores >> of overall size of 45GB. No Sharding, almost all VMs has the same copy of >> data. These nodes are under LB. >> >> Index Config: >> ============= >> >> <ramBufferSizeMB>300</ramBufferSizeMB> >> <mergePolicyFactory class="org.apache.solr.index. >> TieredMergePolicyFactory"> >> <int name="maxMergeAtOnce">30</int> >> <int name="maxMergeAtOnceExplicit">100</int> >> <double name="segmentsPerTier">30.0</double> >> </mergePolicyFactory> >> >> <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> >> <int name="maxMergeCount">18</int> >> <int name="maxThreadCount">6</int> >> </mergeScheduler> >> >> Commit Configs: >> =============== >> <autoCommit> >> <maxTime>${solr.autoCommit.maxTime:600000}</maxTime> >> <openSearcher>false</openSearcher> >> </autoCommit> >> >> <autoSoftCommit> >> <maxTime>${solr.autoSoftCommit.maxTime:60000}</maxTime> >> </autoSoftCommit> >> >> >> We do 3500 Insert / Updates per second spread across all 130 cores, We yet >> to start using selects effectively. >> >> The problem what we are facing is at times suddenly the thread count >> increase heavily which results SOLR non responsive or throwing 503 response >> for client (PHP HTTP CURL) requests. >> >> Today 04-04-2018 the thread dump shows that the peak went upto 13000+ >> >> Please hlep me in fixing this issue. Thanks! >> >> >> Sample Threads: >> =============== >> >> 1.updateExecutor-2-thread-25746-processing-http://// >> 172.10.2.19:8983//solr//profileviews x:profileviews r:core_node2 >> n:172.10.2.18:8983_solr s:shard1 c:profileviews", "state":"TIMED_WAITING", >> "lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ >> ConditionObject@297be1d5", >> "cpuTime":"162.4371ms", "userTime":"120.0000ms", >> "stackTrace":["sun.misc.Unsafe.park(Native Method)", >> "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", >> "java.util.concurrent.locks.AbstractQueuedSynchronizer$ >> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)", >> >> 2. ERROR true HttpSolrCall >> null:org.apache.solr.update.processor.DistributedUpdateProcessor$ >> DistributedUpdatesAsyncException: >> Async exception during distributed update: Error from server at >> 172.10.2.18:8983/solr/profileviews: Server Error request: >> http://172.10.2.18:8983/solr/profileviews/update?update. >> distrib=TOLEADER&distrib.from=http%3A%2F%2F172.10.2.19% >> 3A8983%2Fsolr%2Fprofileviews%2F&wt=javabin&version=2 >> Remote error message: empty String >> >> 3. So Many Threads like: >> "name":"qtp959447386-21", >> "state":"TIMED_WAITING", >> >> "lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ >> ConditionObject@6a1a2bf4 >> ", >> "cpuTime":"4522.0837ms", >> "userTime":"3770.0000ms", >> "stackTrace":["sun.misc.Unsafe.park(Native Method)", >> >> "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", >> >> "java.util.concurrent.locks.AbstractQueuedSynchronizer$ >> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)", >> >> "org.eclipse.jetty.util.BlockingArrayQueue.poll( >> BlockingArrayQueue.java:392)", >> >> "org.eclipse.jetty.util.thread.QueuedThreadPool. >> idleJobPoll(QueuedThreadPool.java:563)", >> >> "org.eclipse.jetty.util.thread.QueuedThreadPool. >> access$800(QueuedThreadPool.java:48)", >> >> "org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( >> QueuedThreadPool.java:626)", >> "java.lang.Thread.run(Thread.java:748)" >> > > > > -- > ============================== > 联创科技 > 知行如一 > ==============================
