That depends on your GC settings and generation sizes. And, instead of UseParallelGC you'd better use UseParNewGC in combination with CMS.
See 22: http://java.sun.com/docs/hotspot/gc1.4.2/faq.html > It's actually, as I understand it, expected JVM behavior to see the heap > rise to close to it's limit before it gets GC'd, that's how Java GC > works. Whether that should happen every 20 seconds or what, I don't nkow. > > Another option is setting better JVM garbage collection arguments, so GC > doesn't "stop the world" so often. I have had good luck with my Solr > using this: -XX:+UseParallelGC > > On 3/14/2011 4:15 PM, Doğacan Güney wrote: > > Hello again, > > > > 2011/3/14 Markus Jelsma<markus.jel...@openindex.io> > > > >>> Hello, > >>> > >>> 2011/3/14 Markus Jelsma<markus.jel...@openindex.io> > >>> > >>>> Hi Doğacan, > >>>> > >>>> Are you, at some point, running out of heap space? In my experience, > >>>> that's the common cause of increased load and excessivly high response > >>>> times (or time > >>>> outs). > >>> > >>> How much of a heap size would be enough? Our index size is growing > >>> slowly but we did not have this problem > >>> a couple weeks ago where index size was maybe 100mb smaller. > >> > >> Telling how much heap space is needed isn't easy to say. It usually > >> needs to > >> be increased when you run out of memory and get those nasty OOM errors, > >> are you getting them? > >> Replication eventes will increase heap usage due to cache warming > >> queries and > >> autowarming. > > > > Nope, no OOM errors. > > > >>> We left most of the caches in solrconfig as default and only increased > >>> filterCache to 1024. We only ask for "id"s (which > >>> are unique) and no other fields during queries (though we do faceting). > >>> Btw, 1.6gb of our index is stored fields (we store > >>> everything for now, even though we do not get them during queries), and > >>> about 1gb of index. > >> > >> Hmm, it seems 4000 would be enough indeed. What about the fieldCache, > >> are there > >> a lot of entries? Is there an insanity count? Do you use boost > >> functions? > > > > Insanity count is 0 and fieldCAche has 12 entries. We do use some > > boosting functions. > > > > Btw, I am monitoring output via jconsole with 8gb of ram and it still > > goes to 8gb every 20 seconds or so, > > gc runs, falls down to 1gb. > > > > Btw, our current revision was just a random choice but up until two weeks > > ago it has been rock-solid so we have been > > reluctant to update to another version. Would you recommend upgrading to > > latest trunk? > > > >> It might not have anything to do with memory at all but i'm just asking. > >> There > >> may be a bug in your revision causing this. > >> > >>> Anyway, Xmx was 4000m, we tried increasing it to 8000m but did not get > >> > >> any > >> > >>> improvement in load. I can try monitoring with Jconsole > >>> with 8gigs of heap to see if it helps. > >>> > >>>> Cheers, > >>>> > >>>>> Hello everyone, > >>>>> > >>>>> First of all here is our Solr setup: > >>>>> > >>>>> - Solr nightly build 986158 > >>>>> - Running solr inside the default jetty comes with solr build > >>>>> - 1 write only Master , 4 read only Slaves (quad core 5640 with 24gb > >> > >> of > >> > >>>>> RAM) - Index replicated (on optimize) to slaves via Solr Replication > >>>>> - Size of index is around 2.5gb > >>>>> - No incremental writes, index is created from scratch(delete old > >>>> > >>>> documents > >>>> > >>>>> -> commit new documents -> optimize) every 6 hours > >>>>> - Avg # of request per second is around 60 (for a single slave) > >>>>> - Avg time per request is around 25ms (before having problems) > >>>>> - Load on each is slave is around 2 > >>>>> > >>>>> We are using this set-up for months without any problem. However last > >>>> > >>>> week > >>>> > >>>>> we started to experience very weird performance problems like : > >>>>> > >>>>> - Avg time per request increased from 25ms to 200-300ms (even higher > >> > >> if > >> > >>>> we > >>>> > >>>>> don't restart the slaves) > >>>>> - Load on each slave increased from 2 to 15-20 (solr uses %400-%600 > >>>>> cpu) > >>>>> > >>>>> When we profile solr we see two very strange things : > >>>>> > >>>>> 1 - This is the jconsole output: > >>>>> > >>>>> https://skitch.com/meralan/rwwcf/mail-886x691 > >>>>> > >>>>> As you see gc runs for every 10-15 seconds and collects more than 1 > >> > >> gb > >> > >>>>> of memory. (Actually if you wait more than 10 minutes you see spikes > >>>>> up to > >>>> > >>>> 4gb > >>>> > >>>>> consistently) > >>>>> > >>>>> 2 - This is the newrelic output : > >>>>> > >>>>> https://skitch.com/meralan/rwwci/solr-requests-solr-new-relic-rpm > >>>>> > >>>>> As you see solr spent ridiculously long time in > >>>>> SolrDispatchFilter.doFilter() method. > >>>>> > >>>>> > >>>>> Apart form these, when we clean the index directory, re-replicate and > >>>>> restart each slave one by one we see a relief in the system but > >> > >> after > >> > >>>> some > >>>> > >>>>> time servers start to melt down again. Although deleting index and > >>>>> replicating doesn't solve the problem, we think that these problems > >> > >> are > >> > >>>>> somehow related to replication. Because symptoms started after > >>>> > >>>> replication > >>>> > >>>>> and once it heals itself after replication. I also see > >>>>> lucene-write.lock files in slaves (we don't have write.lock files in > >>>>> the master) which I think we shouldn't see. > >>>>> > >>>>> > >>>>> If anyone can give any sort of ideas, we will appreciate it. > >>>>> > >>>>> Regards, > >>>>> Dogacan Guney
