We found a problem with the synonym list, and suspect there was some sort of recursion causing the memory to be gobbled up until the JVM crashed. Is this expected behavior from complex synonyms? Or could this be due to the combination of complex synonyms and a bad query format?
Jeremy D. Branham Tel: **DOTNET -----Original Message----- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 4:43 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -----Original Message----- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24:20,334 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,334 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://myserver.com:8080/svc/solr/wdsp 2014-05-30 10:24:20,333 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,336 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsp Jeremy D. Branham Tel: **DOTNET From: Worley, Chris [HR] Sent: Friday, May 30, 2014 11:38 AM To: Stoner, Susan [HR]; Branham, Jeremy [HR] Cc: Barrett, Kevin B [HR]; Aldrich, Daniel J [HR]; Duncan, Horace W [HR] Subject: suspect SOLR query from D029 (SOLR master) Looks like a suspect SOLR query was executed at approx. 8:54am this morning on the SOLR master server (D029) that caused the java GC to go through the roof. See below: 2014-05-30 08:54:17,092 ERROR - SolrDispatchFilter - null:java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2377) at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1698) at org.apache.lucene.index.TermContext.build(TermContext.java:95) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:166) at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82) at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:690) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1501) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1367) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) Unfortunately, I don't have the SOLR query... -CW ________________________________ This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. ________________________________ This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. ________________________________ This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
