This is embarrassing. I just realized that in the experiments where I saw
Solr providing good service in the face of the overload requests, I was
actually sending requests at a rate of 30 requests per second, not 300
requests per second. Once I ratcheted up the rate a little bit, Solr
started to overload like the other applications I've tested.
Thanks for your time, and sorry for the mistake!
Mike
On Fri, Sep 21, 2012 at 7:19 AM, Mike Gagnon wrote:
> Thanks. If Solr doesn't have any special logic for dealing with
> algorithmic-complexity attack-like overloads, then it sounds like Jetty and
> Tomcat are responsible for Solr's unusually good performance in my
> experiments (unusual compared to other non-Java web applications).
>
> Cheers,
> Mike
>
> On Wed, Sep 19, 2012 at 8:30 AM, Walter Underwood
> wrote:
>
>> The front-end code protection that I mentioned was outside of Solr. At
>> that time, requests with very large start values were slow, so we put code
>> in the front end to never request those. Even if the user wanted page 5000
>> of the results, they would get page 100.
>>
>> Now, those requests are fast, so that external protection is not needed.
>>
>> I was running overload tests this summer and could not get Solr to behave
>> badly. The throughput would drop off with overload, but not too bad. This
>> was all with simple queries on a 1.2M doc index.
>>
>> wunder
>> Walter Underwood
>> Search Guy, Chegg
>>
>> On Sep 19, 2012, at 8:20 AM, Erik Hatcher wrote:
>>
>> > How are you triggering an infinite loop in your requests to Solr?
>> >
>> > Erik
>> >
>> > On Sep 19, 2012, at 11:12 , Mike Gagnon wrote:
>> >
>> >> [ I am sorry for breaking the thread, but my inbox has neither
>> received my
>> >> original post to the mailing list, nor Otis's response (so I can't
>> reply to
>> >> his response) ]
>> >>
>> >> Thanks a bunch for your response Otis. Let me more thoroughly explain
>> my
>> >> experimental workload and why I am surprised Solr works so well.
>> >>
>> >> The most important characteristic of my workload is that many of the
>> >> requests (60 per second) cause infinite loops within Solr. That is,
>> each of
>> >> those requests causes a separate infinite loop within it's request
>> context.
>> >>
>> >> This workload is similar to an algorithmic-complexity attack --- a
>> type of
>> >> DoS. In every web-app stack I've tested (except Solr/Jetty and
>> >> Solr/Tomcat) such workloads cause an immediate and complete denial of
>> >> service. What happens for these vulnerable applications, is that the
>> thread
>> >> pool fills up with infinite loops, and incoming requests become
>> rejected.
>> >>
>> >> But Solr manages to survive such an attack. My best guess is that Solr
>> has
>> >> an especially good overload strategy that quickly kicks out the
>> infinite
>> >> loop requests -- which lowers CPU contention, and allows other
>> requests to
>> >> be admitted.
>> >>
>> >> My first guess would be that Tomcat or Jetty is responsible for the
>> good
>> >> response to overload. However,
>> >> there was a good discussion in 2008 on this mailing list about Solr
>> >> Security:
>> >>
>> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200811.mbox/browser
>> >>
>> >> In this discuss Walter Underwood commented: "We have protected against
>> >> several different DoS problems in our front-end code."
>> >>
>> >> Perhaps it is these front-end defenses that help Solr survive my
>> workloads?
>> >>
>> >> Thanks!
>> >> Mike Gagnon
>> >>
>> >>
>> >>> Hm, I'm not sure how to approach this. Solr is not alone here -
>> there's
>> >>> container like jetty, solr inside it and lucene inside solr.
>> >>> Next, that index is rally small, so there is no disk IO. The
>> request
>> >>> rate is also not super high and if you did this over a fast connection
>> >> then
>> >>> there are also no issues with slow response writing or with having
>> lots of
>> >>> concurrent connections or running out of threads ...
>> >>>
>> >>> ...so it's not really that surprising solr keeps working :)
>> >>>
>> >>> But...tell us more.
>> >>>
>> >>> Otis
>> >>> --
>> >>> Performance Monitoring - http://sematext.com/spm
>> >>>
>> >>>
>> >>>
>> >>> On Sep 12, 2012 8:51 PM, "Mike Gagnon" wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I have been studying how server software responds to requests that
>> cause
>> >>> CPU overloads (such as infinite loops).
>> >>>
>> >>> In my experiments I have observed that Solr performs unusually well
>> when
>> >>> subjected to such loads. Every other piece of web software I've
>> >>> experimented with drops to zero service under such loads. Do you know
>> how
>> >>> Solr achieves such good performance? I am guessing that when Solr is
>> >>> overload sheds load to make room for incoming requests, but I could
>> not
>> >>> find any documentation that describes Solr's overload strategy.
>> >>>
>> >>> Experimental setup: I ran Solr 3.1 on a 12-core machine with 12 GB
>> ram,
>> >>> using it index and
