Via this bug: https://issues.apache.org/jira/browse/SOLR-2631
> ... Solr can infinite loop, use 100% CPU and stack overflow, if you > execute the following HTTP request: > - http://localhost:8983/solr/select?qt=/admin/ping > - http://localhost:8983/solr/admin/ping?qt=/admin/ping I am running Solr 3.1, which has that bug. Thanks, Mike On Wed, Sep 19, 2012 at 8:20 AM, Erik Hatcher <erik.hatc...@gmail.com>wrote: > How are you triggering an infinite loop in your requests to Solr? > > Erik > > On Sep 19, 2012, at 11:12 , Mike Gagnon wrote: > > > [ I am sorry for breaking the thread, but my inbox has neither received > my > > original post to the mailing list, nor Otis's response (so I can't reply > to > > his response) ] > > > > Thanks a bunch for your response Otis. Let me more thoroughly explain my > > experimental workload and why I am surprised Solr works so well. > > > > The most important characteristic of my workload is that many of the > > requests (60 per second) cause infinite loops within Solr. That is, each > of > > those requests causes a separate infinite loop within it's request > context. > > > > This workload is similar to an algorithmic-complexity attack --- a type > of > > DoS. In every web-app stack I've tested (except Solr/Jetty and > > Solr/Tomcat) such workloads cause an immediate and complete denial of > > service. What happens for these vulnerable applications, is that the > thread > > pool fills up with infinite loops, and incoming requests become rejected. > > > > But Solr manages to survive such an attack. My best guess is that Solr > has > > an especially good overload strategy that quickly kicks out the infinite > > loop requests -- which lowers CPU contention, and allows other requests > to > > be admitted. > > > > My first guess would be that Tomcat or Jetty is responsible for the good > > response to overload. However, > > there was a good discussion in 2008 on this mailing list about Solr > > Security: > > > http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200811.mbox/browser > > > > In this discuss Walter Underwood commented: "We have protected against > > several different DoS problems in our front-end code." > > > > Perhaps it is these front-end defenses that help Solr survive my > workloads? > > > > Thanks! > > Mike Gagnon > > > > > >> Hm, I'm not sure how to approach this. Solr is not alone here - there's > >> container like jetty, solr inside it and lucene inside solr. > >> Next, that index is reeeeally small, so there is no disk IO. The request > >> rate is also not super high and if you did this over a fast connection > > then > >> there are also no issues with slow response writing or with having lots > of > >> concurrent connections or running out of threads ... > >> > >> ...so it's not really that surprising solr keeps working :) > >> > >> But...tell us more. > >> > >> Otis > >> -- > >> Performance Monitoring - http://sematext.com/spm > >> > >> > >> > >> On Sep 12, 2012 8:51 PM, "Mike Gagnon" <mikegag...@gmail.com> wrote: > >> > >> Hi, > >> > >> I have been studying how server software responds to requests that cause > >> CPU overloads (such as infinite loops). > >> > >> In my experiments I have observed that Solr performs unusually well when > >> subjected to such loads. Every other piece of web software I've > >> experimented with drops to zero service under such loads. Do you know > how > >> Solr achieves such good performance? I am guessing that when Solr is > >> overload sheds load to make room for incoming requests, but I could not > >> find any documentation that describes Solr's overload strategy. > >> > >> Experimental setup: I ran Solr 3.1 on a 12-core machine with 12 GB ram, > >> using it index and search about 10,000 pages on MediaWiki. I test both > >> Solr+Jetty and Solr+Tomcat. I submitted a variety of Solr queries at a > > rate > >> of 300 requests per second. At the same time, I submitted "overload > >> requests" at a rate of 60 requests per second. Each overload request > > caused > >> an infinite loop in Solr via > >> https://issues.apache.org/jira/browse/SOLR-2631. > >> > >> With Jetty about 70% of non-overload requests completed --- 95% of > > requests > >> completing within 0.6 seconds. > >> With Tomcat about 34% of non-overload requests completed --- 95% of > >> requests completing within 0.6 seconds. > >> > >> I also ran Solr+Jetty with non-overload requests coming in 65 requests > per > >> second (overload requests remain at 60 requests per second). In this > >> workload, the completion rate drops to 15% and the 95th percentile > latency > >> increases to 25. > >> > >> Cheers, > >> Mike Gagnon > >> > >
