I'd second the use of jstack to check your threads. Each request (be it a search or update) will generate a request handler thread on the Solr side (unless you've set the limits in the HttpShardHandlerFactory (solr.xml for solr-wide faults and/or under the requestHandler in SolrConfig.xml), we set maxConnectionsPerHost, corePoolSize and maximumPoolSize, since we ran into a similar issue.
Our system ironically didn't crash, we just had a JVM with.about 256000 threads, which was rather SSLLOOWW :) On the softCommit front, we have had some success with small softCommit times, but then we use SSDs (and have lots of memory and lots of shards). Once we get concrete figures, we'll publish them, but we are a fair way below 1s now with no major impact on indexing throughput (yet). But I would agree that unless you are really really sure you need it (and most people don't), keep to the "known limits". On 4 December 2013 16:09, Tim Potter <tim.pot...@lucidworks.com> wrote: > Hi Metin, > > I think removing the softCommit=true parameter on the client side will > definitely help as NRT wasn't designed to re-open searchers after every > document. Try every 1 second (or even every few seconds), I doubt your > users will notice. To get an idea of what threads are running in your JVM > process, you can use jstack. > > Cheers, > > Timothy Potter > Sr. Software Engineer, LucidWorks > www.lucidworks.com > > ________________________________________ > From: OSMAN Metin <metin.os...@canal-plus.com> > Sent: Wednesday, December 04, 2013 7:36 AM > To: solr-user@lucene.apache.org > Subject: Questions about commits and OOE > > Hi all, > > let me first explain our situation : > > We have > > > - two virtual servers with each : > > 4x SolR 4.4.0 on Tomcat 6 (+ with mod_cluster 1.2.0), each JVM has > -Xms2048m -Xmx2048m -XX:MaxPermSize=384m > 1x Zookeeper 3.4.5 (Only one of the two Zookeeper is active.) > CentOS 6.4 > Sun JDK 1.6.0-31 > 16 GB of RAM > 4 vCPU > > > - only one core and one shard > > - ~250000 docs and 50-100 MB of index size > > - two load balancers (apache + mod_cluster) who are both connected > to the 8 SolR nodes > > - 1 VIP pointing to these two LB > > The commit configuration is > > - every update request do a soft commit (i.e. param softCommit=true > in the http request) > > - autosoftcommit disabled > > - autocommit enabled every 15 seconds > > The client application is a java app with SolRj client using the previous > VIP as an endpoint. > We need NearRealTime modifications visible by the end users. > During the day, the client uses SolR with about 80% of select requests and > 20% of update requests. > Every morning, the client is sending a massive bunch of updates (about > 10000 in a few minutes). > > During this massive update, we have sometimes a peak of active threads > exceeding the limit of 8192 process authorized for the user running the > tomcat and zookeeper process. > When this happens, every hardCommit is failing with an "OutOfMemory : > unable to create native thread" message. > > > Now, I have some questions : > > - Why are there some many threads created ? Is the softCommit on > every update that opens a new thread ? > > - Once an OOE occurs, every hardcommit will be broken, even if the > number of threads opened on the system is low. Is there any way to "free" > the JVM ? The only solution we have found is to restart all the JVM. > > - When the OOE occurs, the SolR cloud console shows the leader node > as active and the others as recovering > > o is the replication working at that moment ? > > o as all the hardcommits are failing but the softcommits not, am I very > sure that I will not lose some updates when restarting all the nodes ? > > By the way, we are planning to > > - disable the softCommit parameter on the client side and to enable > the autosoftcommit instead. > > - create another server and make 3 zookeeper chorum instead of a > unique zookeeper master. > > - skip the use of load balancers and let zookeeper decide which node > will respond to the requests > > Any help would be appreciated ! > > Metin OSMAN >
