Thanks. If Solr doesn't have any special logic for dealing with algorithmic-complexity attack-like overloads, then it sounds like Jetty and Tomcat are responsible for Solr's unusually good performance in my experiments (unusual compared to other non-Java web applications).
Cheers, Mike On Wed, Sep 19, 2012 at 8:30 AM, Walter Underwood <wun...@wunderwood.org>wrote: > The front-end code protection that I mentioned was outside of Solr. At > that time, requests with very large start values were slow, so we put code > in the front end to never request those. Even if the user wanted page 5000 > of the results, they would get page 100. > > Now, those requests are fast, so that external protection is not needed. > > I was running overload tests this summer and could not get Solr to behave > badly. The throughput would drop off with overload, but not too bad. This > was all with simple queries on a 1.2M doc index. > > wunder > Walter Underwood > Search Guy, Chegg > > On Sep 19, 2012, at 8:20 AM, Erik Hatcher wrote: > > > How are you triggering an infinite loop in your requests to Solr? > > > > Erik > > > > On Sep 19, 2012, at 11:12 , Mike Gagnon wrote: > > > >> [ I am sorry for breaking the thread, but my inbox has neither received > my > >> original post to the mailing list, nor Otis's response (so I can't > reply to > >> his response) ] > >> > >> Thanks a bunch for your response Otis. Let me more thoroughly explain > my > >> experimental workload and why I am surprised Solr works so well. > >> > >> The most important characteristic of my workload is that many of the > >> requests (60 per second) cause infinite loops within Solr. That is, > each of > >> those requests causes a separate infinite loop within it's request > context. > >> > >> This workload is similar to an algorithmic-complexity attack --- a type > of > >> DoS. In every web-app stack I've tested (except Solr/Jetty and > >> Solr/Tomcat) such workloads cause an immediate and complete denial of > >> service. What happens for these vulnerable applications, is that the > thread > >> pool fills up with infinite loops, and incoming requests become > rejected. > >> > >> But Solr manages to survive such an attack. My best guess is that Solr > has > >> an especially good overload strategy that quickly kicks out the infinite > >> loop requests -- which lowers CPU contention, and allows other requests > to > >> be admitted. > >> > >> My first guess would be that Tomcat or Jetty is responsible for the good > >> response to overload. However, > >> there was a good discussion in 2008 on this mailing list about Solr > >> Security: > >> > http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200811.mbox/browser > >> > >> In this discuss Walter Underwood commented: "We have protected against > >> several different DoS problems in our front-end code." > >> > >> Perhaps it is these front-end defenses that help Solr survive my > workloads? > >> > >> Thanks! > >> Mike Gagnon > >> > >> > >>> Hm, I'm not sure how to approach this. Solr is not alone here - there's > >>> container like jetty, solr inside it and lucene inside solr. > >>> Next, that index is reeeeally small, so there is no disk IO. The > request > >>> rate is also not super high and if you did this over a fast connection > >> then > >>> there are also no issues with slow response writing or with having > lots of > >>> concurrent connections or running out of threads ... > >>> > >>> ...so it's not really that surprising solr keeps working :) > >>> > >>> But...tell us more. > >>> > >>> Otis > >>> -- > >>> Performance Monitoring - http://sematext.com/spm > >>> > >>> > >>> > >>> On Sep 12, 2012 8:51 PM, "Mike Gagnon" <mikegag...@gmail.com> wrote: > >>> > >>> Hi, > >>> > >>> I have been studying how server software responds to requests that > cause > >>> CPU overloads (such as infinite loops). > >>> > >>> In my experiments I have observed that Solr performs unusually well > when > >>> subjected to such loads. Every other piece of web software I've > >>> experimented with drops to zero service under such loads. Do you know > how > >>> Solr achieves such good performance? I am guessing that when Solr is > >>> overload sheds load to make room for incoming requests, but I could not > >>> find any documentation that describes Solr's overload strategy. > >>> > >>> Experimental setup: I ran Solr 3.1 on a 12-core machine with 12 GB ram, > >>> using it index and search about 10,000 pages on MediaWiki. I test both > >>> Solr+Jetty and Solr+Tomcat. I submitted a variety of Solr queries at a > >> rate > >>> of 300 requests per second. At the same time, I submitted "overload > >>> requests" at a rate of 60 requests per second. Each overload request > >> caused > >>> an infinite loop in Solr via > >>> https://issues.apache.org/jira/browse/SOLR-2631. > >>> > >>> With Jetty about 70% of non-overload requests completed --- 95% of > >> requests > >>> completing within 0.6 seconds. > >>> With Tomcat about 34% of non-overload requests completed --- 95% of > >>> requests completing within 0.6 seconds. > >>> > >>> I also ran Solr+Jetty with non-overload requests coming in 65 requests > per > >>> second (overload requests remain at 60 requests per second). In this > >>> workload, the completion rate drops to 15% and the 95th percentile > latency > >>> increases to 25. > >>> > >>> Cheers, > >>> Mike Gagnon > >>> > > > > > > > >
