I have been load testing solr 1.4.1 and have been running into OOM errors.
Not out of heap but with the GC overhead limit exceeded message meaning that
it didn't actually run out of heap space but just spent too much CPU time
trying to make room and gave up.
I got a heap dump and sent it through the Eclipse MAT and found that a
single WeakHashMap in FieldCacheImpl called readerCache is taking up 2.1GB
of my 2.6GB heap.
>From my understanding of WeakHashMaps the GC should be able to collect those
references if it needs to but for some reason it isn't here.
My questions are:
1) Any ideas why the GC is not collecting those weak references in that
single hashmap?
2) Is there a nob in the solr config that can limit the size of that cache?
Also, after the OOM is thrown solr doesn't respond much at all and throws
the exception below, however when I go to the code I see this
try {
processor.processAdd(addCmd);
addCmd.clear();
} catch (IOException e) {
throw new
SolrException(SolrException.ErrorCode.SERVER_ERROR, "ERROR adding document "
+ document);
}
}
So its swallowing the IOException and throwing a new one without setting
the cause so I can't see what the IOException is. Is this fixed in any
newer version? Should I open a bug?
Thanks a lot for your help
John
SEVERE: org.apache.solr.common.SolrException: ERROR adding document
SolrInputDocument[{de.id=de.id(1.0)={C2B3B03F1000012C549254560A568C18},
de.type=de.type(1.0)={Social
Contact}, sc.author=sc.author(1.0)={Author-3944},
sc.sourceType=sc.sourceType(1.0)={rss}, sc.link=sc.link(1.0)={
http://www.cisco.com/feed/date_12.07.10_16.18.03/idx/107
52}, sc.title=sc.title(1.0)={Title-erat metus eget vestibulum},
sc.publishedDate=sc.publishedDate(1.0)={Tue Dec 07 16:22:09 EST 2010},
sc.createdDate=sc.createdDate(1.0
)={Tue Dec 07 16:20:20 EST 2010},
sc.socialContactStatus=sc.socialContactStatus(1.0)={unread},
sc.socialContactStatusUserId=sc.socialContactStatusUserId(1.0)={}, sc.soc
ialContactStatusDate=sc.socialContactStatusDate(1.0)={Tue Dec 07 16:20:20
EST 2010}, sc.tags=sc.tags(1.0)={[]}, sc.authorId=sc.authorId(1.0)={},
sc.replyToId=sc.replyTo
Id(1.0)={}, sc.replyToAuthor=sc.replyToAuthor(1.0)={},
sc.replyToAuthorId=sc.replyToAuthorId(1.0)={},
sc.feedId=sc.feedId(1.0)={[124852]}, filterResult_124932_ti=filter
Result_124932_ti(1.0)={67},
filterStatus_124932_s=filterStatus_124932_s(1.0)={COMPLETED},
filterResult_124937_ti=filterResult_124937_ti(1.0)={67},
filterStatus_124937_s
=filterStatus_124937_s(1.0)={COMPLETED},
campaignDateAdded_124957_tdt=campaignDateAdded_124957_tdt(1.0)={Tue Dec 07
16:20:20 EST 2010}, campaignStatus_124957_s=campaign
Status_124957_s(1.0)={NEW},
campaignDateAdded_124947_tdt=campaignDateAdded_124947_tdt(1.0)={Tue Dec 07
16:20:20 EST 2010}, campaignStatus_124947_s=campaignStatus_124947
_s(1.0)={NEW},
sc.campaignResultsSummary=sc.campaignResultsSummary(1.0)={[NEW, NEW]}}]
at
org.apache.solr.handler.BinaryUpdateRequestHandler$2.document(BinaryUpdateRequestHandler.java:81)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$2.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:136)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$2.readIterator(JavaBinUpdateRequestCodec.java:126)
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:210)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$2.readNamedList(JavaBinUpdateRequestCodec.java:112)
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:175)
at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:101)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:141)
at
org.apache.solr.handler.BinaryUpdateRequestHandler.parseAndLoadDocs(BinaryUpdateRequestHandler.java:68)
at
org.apache.solr.handler.BinaryUpdateRequestHandler.access$000(BinaryUpdateRequestHandler.java:46)
at
org.apache.solr.handler.BinaryUpdateRequestHandler$1.load(BinaryUpdateRequestHandler.java:55)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:723)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
