On 9/3/2014 3:31 PM, Ethan wrote: > We have SolrCloud instance with 2 solr nodes and 3 zk ensemble. One of the > solr node goes down as soon as we send search traffic to it, but update > works fine. > > When I analyzed thread dump I saw lot of blocked threads with following > error message. This explains why it couldn't create any native threads and > ran out of memory. The thread count went from 48 to 900 within minutes and > server came down. The other node with same configuration is taking all the > search and update traffic, and it running fine. > > Any pointers would be appreciated. > > http-bio-52158-exec-59" - Thread t@589 > java.lang.Thread.State: BLOCKED on > org.apache.lucene.search.FieldCache$CreationPlaceholder@29e0400b owned by: > http-bio-52158-exec-61
This may be completely useless information, but just in case, I thought I'd throw it out there as something you should be able to quickly and easily try. I would also be interested in seeing what comes of Erick's suggestions and questions. The default maxThreads value on tomcat (which from your stacktrace is very likely the container you're using) is 200. That setting has been increased to 10000 in the example jetty container that comes with Solr. Try setting maxThreads to 10000 in the configuration on all your tomcat instances. I *think* it would go in the HTTP Connector, but Jetty seems to have it in the Server. http://tomcat.apache.org/tomcat-5.5-doc/config/http.html Thanks, Shawn
