On Apr 5, 2012, at 7:11 AM, Jam Luo wrote: > Hi > I deployed a solr cluster,the code version is > "NightlyBuilds apache-solr-4.0-2012-03-19_09-25-37". > Cluster has 4 nodes named "A", "B", "C", "D", "num_shards=2", A and > C in shard1 , B and D in shard2, A and B is the leader of their shard. It > has ran 2 days, added 20m docs, all of then are OK, but after this ,C > occured a Exception "org.apache.lucene.store.AlreadyClosedException: this > IndexWriter is closed", and jetty on C was not down, node C exist in > zookeeper in "/live_nodes". In this time, The A try to ask C to recover, > but C can't an response, so A get a Exception, the log is: > > <D0><C5>Ϣ: try and ask http://node23:8983/solr to recover > <CB><C4><D4><C2> 04, 2012 8:02:36 <CF><C2><CE><E7> > org.apache.solr.update.processor.DistributedUpdateProcessor doFinish > <D0><C5>Ϣ: try and ask http://node23:8983/solr to recover > <CB><C4><D4><C2> 04, 2012 8:02:36 <CF><C2><CE><E7> > org.apache.solr.update.processor.DistributedUpdateProcessor doFinish > <D0><C5>Ϣ: Could not tell a replica to recover > org.apache.solr.client.solrj.SolrServerException: http://node23:8983/solr > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:496) > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:347) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:816) > at > org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:176) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1549) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:441) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:262) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) > at org.eclipse.jetty.server.Server.handle(Server.java:351) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) > at > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) > at > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.net.ConnectException: <BE>ܾ<F8><C1><AC><BD><D3> > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) > at java.net.Socket.connect(Socket.java:579) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140) > at > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125) > at > org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at > org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:426) > ... 34 more > > There are 2400 lines log as above at 8:02:36 in 1 second, and it is not > over until I shut down the C's jetty server. The important is the client > can 't add any document to cluster in this time, it will recover after C > server shutdown also. > /clusterstate.json data in zookeeper is: > {"mblog":{ > "shard1":{ > "A:8983_solr_mblog":{ > "shard":"shard1", > "leader":"true", > "state":"active", > "core":"mblog", > "collection":"mblog", > "node_name":"A:8983_solr", > "base_url":"http://A:8983/solr"}, > "C:8983_solr_mblog":{ > "shard":"shard1", > "state":"recovering", > "core":"mblog", > "collection":"mblog", > "node_name":"C:8983_solr", > "base_url":"http://C:8983/solr"}}, > "shard2":{ > "B:8983_solr_mblog":{ > "shard":"shard2", > "state":"active", > "leader":"true", > "core":"mblog", > "collection":"mblog", > "node_name":"B:8983_solr", > "base_url":"http://B:8983/solr"}, > "D:8983_solr_mblog":{ > "shard":"shard2", > "state":"active", > "core":"mblog", > "collection":"mblog", > "node_name":"D:8983_solr", > "base_url":"http://D:8983/solr"}}}}
I'd try bouncing the node that had the error. Sounds like there is some bug involved, unless it's been fixed since that build - this should not occur (an IndexWriter being closed while something tries to use it). > > > The last question is how to config the solr_cluster.properties file and > excute the Cluster Admin Commands described in > http://wiki.apache.org/solr/NewSolrCloudDesign ? We didn't end up using that design for a lot of things - we should probably note that at the top of that doc. - Mark Miller lucidimagination.com
