I've got the start of a Garbage Collection article here: http://www.lucidimagination.com/blog/2009/09/19/java-garbage-collection-boot-camp-draft/
I plan to tie it more into Lucene/Solr and add some more about the theory/methods in the final version. With so much RAM, I take it you prob have a handful of processors as well? You might start by trying the Concurrent Low Pause Collector if you have not. You might also pair it with the parallel new generation collector. If you still get long pauses, you might try lowering -XX:CMSInitiatingOccupancyFraction, to kick off major collections earlier. It can still be difficult with really large fieldcaches, because all of sudden, everything is released at once when the Reader goes away - but there should be some combo of settings that at least help alleviate the issue, especially by dedicating another processor to the task that can work somewhat in parallel without stopping your application threads for so long. If you have some success tuning, report back with your results if you could. -- - Mark http://www.lucidimagination.com Jonathan Ariel wrote: > Right, now I'm giving it 12GB of heap memory. > If I give it less (10GB) it throws the following exception: > > Sep 5, 2009 7:18:32 PM org.apache.solr.common.SolrException log > SEVERE: java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:361) > at > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72) > at > org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352) > at > org.apache.solr.request.SimpleFacets.getFieldCacheCounts(SimpleFacets.java:267) > at > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:185) > at > org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:207) > at > org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:104) > at > org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) > at > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) > at org.mortbay.jetty.Server.handle(Server.java:285) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502) > at > org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) > at > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226) > at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) > > On Fri, Sep 25, 2009 at 10:55 AM, Yonik Seeley > <yo...@lucidimagination.com>wrote: > > >> On Fri, Sep 25, 2009 at 9:30 AM, Jonathan Ariel <ionat...@gmail.com> >> wrote: >> >>> Hi to all! >>> Lately my solr servers seem to stop responding once in a while. I'm using >>> solr 1.3. >>> Of course I'm having more traffic on the servers. >>> So I logged the Garbage Collection activity to check if it's because of >>> that. It seems like 11% of the time the application runs, it is stopped >>> because of GC. And some times the GC takes up to 10 seconds! >>> Is is normal? My instances run on a 16GB RAM, Dual Quad Core Intel Xeon >>> servers. My index is around 10GB and I'm giving to the instances 10GB of >>> RAM. >>> >> Bigger heaps lead to bigger GC pauses in general. >> Do you mean that you are giving the JVM a 10GB heap? Were you getting >> OOM exceptions with a smaller heap? >> >> -Yonik >> http://www.lucidimagination.com >> >> > >
