what's happening on the system when you see this? If you're heavily indexing and NOT using SolrJ.cloudSolrSever/Client, then a lot of threads can be occupied forwarding documents to the other shards.
Best, Erick On Wed, Aug 19, 2015 at 6:55 AM, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> wrote: > Jetty includes a QoSFilter, > https://wiki.eclipse.org/Jetty/Reference/QoSFilter, with some changes I think > it might be able to throttle the requests coming into Solr from truly > outside, e.g. not SolrCloud replication, ZooKeeper etc., so as to make sure > that Solr's own work could still get done. This is just an idea, not a > suggestion. > > -----Original Message----- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Wednesday, August 19, 2015 9:15 AM > To: solr-user@lucene.apache.org > Subject: Re: jetty.xml > > On 8/18/2015 11:50 PM, William Bell wrote: >> We sometimes get a spike in Solr, and we get like 3K of threads and >> then timeouts... >> >> In Solr 5.2.1 the defult jetty settings is kinda crazy for threads - >> since the value is HIGH! >> >> What do others recommend? > > The setting of 10000 is so that there is effectively no limit. Solr will > stop working right if it is not allowed to start threads whenever it wishes. > Solr is not a typical web application. > > As far as I know (any my knowledge could be wrong), a typical web application > that serves a website to users will handle all back-end details with the same > thread(s) that were created when the connection was opened. Putting a > relatively low limit on the number of threads in that situation is sensible. > > A very small Solr install with a low query volume will work in 200 threads > (the default limit in most containers), but it doesn't take very much to > exceed that. > > I have a Solr 4.9.1 dev install with 44 cores, running with the Jetty 8 > example included in the 4.x download. 19 of those cores are build cores, > with 19 cores for live indexes. The other six cores are always empty, with a > shards parameter in the search handler definition for distributed searching. > This install does NOT run in SolrCloud mode. > > This dev server sees very little traffic besides a few indexing requests > every minute and load balancer health checks. JConsole shows the number of > threads hovering between 230 and 235. If I scroll through the thread list, > most of them show a state of WAITING on various locking mechanisms, which > explains why my CPUs (8 CPU cores total) are not being overwhelmed with work > from all those threads. > > Solr and Lucene don't really have a runaway thread problem as far as I can > tell, but the system does use a fair number of them for basic operation, with > more cores/collections adding more threads. SolrCloud mode will also use > more threads. > > If you send requests to Solr at a vary fast rate, the servlet container may > also use a lot of threads. > > Thanks, > Shawn >
