Ahhh, FieldCache loading... what version of Solr are you using? It's interesting it would take that long to load too (and maxing out one CPU - doesn't look particularly IO bound). How many documents are in this index?
-Yonik On Tue, Mar 9, 2010 at 12:33 PM, John Williams <j...@37signals.com> wrote: > Yonik, > > I have provided an image below gives details on what is causing the blocked > http thread. Is there any way to resolve this issue. > > Thanks, > John > > -- > John Williams > System Administrator > 37signals > > > > On Mar 9, 2010, at 10:41 AM, John Williams wrote: > >> Yonik, >> >> I got yourkit setup to profile the Tomcat instance and as you will see in >> the graph below all of the http threads are blocked (red) until around >> 4:40. This is the point where the instance becomes responsive and CPU usage >> drops. I have also ruled out GC being the issue by using the GC monitoring >> in yourkit. Let me know your thoughts and if you have any questions. >> >> Thanks for your assistance. >> >> Thanks, >> John >> >> -- >> John Williams >> System Administrator >> 37signals >> >> <Screen shot 2010-03-09 at 10.35.15 AM.png> >> On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: >> >>> On Mon, Mar 8, 2010 at 6:07 PM, John Williams <j...@37signals.com> wrote: >>>> Yonik, >>>> >>>> In all cases our "autowarmCount" is set to 0. Also, here is a link to our >>>> config. http://pastebin.com/iUgruqPd >>> >>> Weird... on a quick glance, I don't see anything in your config that >>> would cause work to be done on a commit. >>> I expected something like autowarming, or rebuilding a spellcheck >>> index, etc. I assume this is happening even w/o any requests hitting >>> the server? >>> >>> Could it be GC? You could use -verbose:gc or jconsole to check if >>> this corresponds to a big GC (which could naturally hit on an index >>> change). 5 minutes is really excessive though, and I wouldn't expect >>> it on startup. >>> >>> If it's not GC, perhaps the next step is to get some stack traces >>> during the spike (or use a profiler) to figure out where the time is >>> being spent. And verify that the solrconfig.xml shown actually still >>> matches the one you provided. >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> >>> >>>> Thanks, >>>> John >>>> >>>> -- >>>> John Williams >>>> System Administrator >>>> 37signals >>>> >>>> On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: >>>> >>>>> Is this just autowarming? >>>>> Check your autowarmCount parameters in solrconfig.xml >>>>> >>>>> -Yonik >>>>> http://www.lucidimagination.com >>>>> >>>>> On Mon, Mar 8, 2010 at 5:37 PM, John Williams <j...@37signals.com> wrote: >>>>>> Good afternoon. >>>>>> >>>>>> We have been experiencing an odd issue with one of our Solr nodes. Upon >>>>>> startup or when bringing in a new index we get a CPU spike for 5 minutes >>>>>> or so. I have attached a graph of this spike. During this time simple >>>>>> queries return without a problem but more complex queries do not return. >>>>>> Here are some more details about the instance: >>>>>> >>>>>> Index Size: ~16G >>>>>> Max Heap: 6144M >>>>>> GC Option: -XX:+UseConcMarkSweepGC >>>>>> System Memory: 16G >>>>>> >>>>>> We have a very similar instance to this one but with a much larger index >>>>>> that we are not seeing this sort of issue. >>>>>> >>>>>> Your help is greatly appreciated. Let me know if you need any additional >>>>>> information. >>>>>> >>>>>> Thanks, >>>>>> John >>>>>> >>>>>> -- >>>>>> John Williams >>>>>> System Administrator >>>>>> 37signals >>>>>> >>>>>> >>>>>> >>>> >>>> >> > > >
