Hi,
I'm having a problem with one of my shards. Since yesterday, SOLR keeps
repeating the same exception over and over for this shard.
The webinterface for this SOLR instance is also not working (it hangs on the
Loading indicator).
Nov 7, 2013 9:08:12 AM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: [website1_shard1_replica3] webapp=/solr path=/update
params={update.distrib=TOLEADER&wt=javabin&version=2} {} 0 0
Nov 7, 2013 9:08:12 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: SolrCoreState already closed
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:79)
at
org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:276)
at
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
at
org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
at
org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
I have about 3GB of logfiles for this single message. Reloading the collection
does not work. Reloading the specific shard core returns the same exception.
The only option seems to be to restart the server. But because it's the leader
for a lot of collections, I want to know why this is happening. I've seen this
problem before, and I haven't figured out what is causing it.
I've reported a different problem a few days ago with 'hanging' deleted
logfiles. Could this be related? Could the hanging logfiles prevent a new
Searcher from opening? I've updated two of my three hosts to 4.5.1 but after
only 2 days uptime, I'm still seeing about 11.000 deleted logfiles in the lsof
output.
Best regards,
Eric Bus
