[ I am sorry for breaking the thread, but my inbox has neither received my original post to the mailing list, nor Otis's response (so I can't reply to his response) ]
Thanks a bunch for your response Otis. Let me more thoroughly explain my experimental workload and why I am surprised Solr works so well. The most important characteristic of my workload is that many of the requests (60 per second) cause infinite loops within Solr. That is, each of those requests causes a separate infinite loop within it's request context. This workload is similar to an algorithmic-complexity attack --- a type of DoS. In every web-app stack I've tested (except Solr/Jetty and Solr/Tomcat) such workloads cause an immediate and complete denial of service. What happens for these vulnerable applications, is that the thread pool fills up with infinite loops, and incoming requests become rejected. But Solr manages to survive such an attack. My best guess is that Solr has an especially good overload strategy that quickly kicks out the infinite loop requests -- which lowers CPU contention, and allows other requests to be admitted. My first guess would be that Tomcat or Jetty is responsible for the good response to overload. However, there was a good discussion in 2008 on this mailing list about Solr Security: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200811.mbox/browser In this discuss Walter Underwood commented: "We have protected against several different DoS problems in our front-end code." Perhaps it is these front-end defenses that help Solr survive my workloads? Thanks! Mike Gagnon > Hm, I'm not sure how to approach this. Solr is not alone here - there's > container like jetty, solr inside it and lucene inside solr. > Next, that index is reeeeally small, so there is no disk IO. The request > rate is also not super high and if you did this over a fast connection then > there are also no issues with slow response writing or with having lots of > concurrent connections or running out of threads ... > > ...so it's not really that surprising solr keeps working :) > > But...tell us more. > > Otis > -- > Performance Monitoring - http://sematext.com/spm > > > > On Sep 12, 2012 8:51 PM, "Mike Gagnon" <mikegag...@gmail.com> wrote: > > Hi, > > I have been studying how server software responds to requests that cause > CPU overloads (such as infinite loops). > > In my experiments I have observed that Solr performs unusually well when > subjected to such loads. Every other piece of web software I've > experimented with drops to zero service under such loads. Do you know how > Solr achieves such good performance? I am guessing that when Solr is > overload sheds load to make room for incoming requests, but I could not > find any documentation that describes Solr's overload strategy. > > Experimental setup: I ran Solr 3.1 on a 12-core machine with 12 GB ram, > using it index and search about 10,000 pages on MediaWiki. I test both > Solr+Jetty and Solr+Tomcat. I submitted a variety of Solr queries at a rate > of 300 requests per second. At the same time, I submitted "overload > requests" at a rate of 60 requests per second. Each overload request caused > an infinite loop in Solr via > https://issues.apache.org/jira/browse/SOLR-2631. > > With Jetty about 70% of non-overload requests completed --- 95% of requests > completing within 0.6 seconds. > With Tomcat about 34% of non-overload requests completed --- 95% of > requests completing within 0.6 seconds. > > I also ran Solr+Jetty with non-overload requests coming in 65 requests per > second (overload requests remain at 60 requests per second). In this > workload, the completion rate drops to 15% and the 95th percentile latency > increases to 25. > > Cheers, > Mike Gagnon >
