How are you triggering an infinite loop in your requests to Solr?
Erik
On Sep 19, 2012, at 11:12 , Mike Gagnon wrote:
> [ I am sorry for breaking the thread, but my inbox has neither received my
> original post to the mailing list, nor Otis's response (so I can't reply to
> his response) ]
>
> Thanks a bunch for your response Otis. Let me more thoroughly explain my
> experimental workload and why I am surprised Solr works so well.
>
> The most important characteristic of my workload is that many of the
> requests (60 per second) cause infinite loops within Solr. That is, each of
> those requests causes a separate infinite loop within it's request context.
>
> This workload is similar to an algorithmic-complexity attack --- a type of
> DoS. In every web-app stack I've tested (except Solr/Jetty and
> Solr/Tomcat) such workloads cause an immediate and complete denial of
> service. What happens for these vulnerable applications, is that the thread
> pool fills up with infinite loops, and incoming requests become rejected.
>
> But Solr manages to survive such an attack. My best guess is that Solr has
> an especially good overload strategy that quickly kicks out the infinite
> loop requests -- which lowers CPU contention, and allows other requests to
> be admitted.
>
> My first guess would be that Tomcat or Jetty is responsible for the good
> response to overload. However,
> there was a good discussion in 2008 on this mailing list about Solr
> Security:
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200811.mbox/browser
>
> In this discuss Walter Underwood commented: "We have protected against
> several different DoS problems in our front-end code."
>
> Perhaps it is these front-end defenses that help Solr survive my workloads?
>
> Thanks!
> Mike Gagnon
>
>
>> Hm, I'm not sure how to approach this. Solr is not alone here - there's
>> container like jetty, solr inside it and lucene inside solr.
>> Next, that index is reeeeally small, so there is no disk IO. The request
>> rate is also not super high and if you did this over a fast connection
> then
>> there are also no issues with slow response writing or with having lots of
>> concurrent connections or running out of threads ...
>>
>> ...so it's not really that surprising solr keeps working :)
>>
>> But...tell us more.
>>
>> Otis
>> --
>> Performance Monitoring - http://sematext.com/spm
>>
>>
>>
>> On Sep 12, 2012 8:51 PM, "Mike Gagnon" <mikegag...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have been studying how server software responds to requests that cause
>> CPU overloads (such as infinite loops).
>>
>> In my experiments I have observed that Solr performs unusually well when
>> subjected to such loads. Every other piece of web software I've
>> experimented with drops to zero service under such loads. Do you know how
>> Solr achieves such good performance? I am guessing that when Solr is
>> overload sheds load to make room for incoming requests, but I could not
>> find any documentation that describes Solr's overload strategy.
>>
>> Experimental setup: I ran Solr 3.1 on a 12-core machine with 12 GB ram,
>> using it index and search about 10,000 pages on MediaWiki. I test both
>> Solr+Jetty and Solr+Tomcat. I submitted a variety of Solr queries at a
> rate
>> of 300 requests per second. At the same time, I submitted "overload
>> requests" at a rate of 60 requests per second. Each overload request
> caused
>> an infinite loop in Solr via
>> https://issues.apache.org/jira/browse/SOLR-2631.
>>
>> With Jetty about 70% of non-overload requests completed --- 95% of
> requests
>> completing within 0.6 seconds.
>> With Tomcat about 34% of non-overload requests completed --- 95% of
>> requests completing within 0.6 seconds.
>>
>> I also ran Solr+Jetty with non-overload requests coming in 65 requests per
>> second (overload requests remain at 60 requests per second). In this
>> workload, the completion rate drops to 15% and the 95th percentile latency
>> increases to 25.
>>
>> Cheers,
>> Mike Gagnon
>>
