Solr Cloud Shards and Replica not reviving after restarting

[Tim Burner]

Hi Everyone,

I have installed Solr Cloud 4.6.2 with external Zookeeper and Tomcat,
having 3 shards with 2 replica each. I tried indexing some documents which
went easy.

After which I restarted my Tomcat, and now the Shards are not getting up,
its coming up with bunch of Exceptions. First exception was "*no servers
hosting shard:"*

All the replica and leader are down and not responding, its even giving

RecoveryStrategy Error while trying to recover.
core=recollection_shard1_replica1:org.apache.solr.client.solrj.SolrServerException:
Server refused connection at: http://192.168.2.183:9090/solr

It would be great if you can help me out solving this issue. Expert advice
needed.

Thanks in Advance!

Vague Behavior while setting Solr Cloud

[Tim Burner]

Hi Everyone,

I am trying to setup Solr Cloud referring to the blog
http://myjeeva.com/solrcloud-cluster-single-collection-deployment.html

if I complete the set in one go, then it seems to be going fine.

when the setup is complete and I am trying to restart Solr by restarted
Tomcat instance, it does not deploy and moreover the shards and replicas
are not up.

Urgent call, let me know if you know anything!

Thanks in Advance!

Re: Solr Cloud Shards and Replica not reviving after restarting

[Tim Burner]

Thanks Erick,

I much appreciate your help I got it fixed, actually there were some
background process already running for tomcat which weren't stopped by the
time I faced these issues.

Thanks again!


On Wed, May 21, 2014 at 8:25 AM, Erick Erickson wrote:

> First thing I'd look at is the log on the server. It's possible that
> you've changed the configuration such that Solr can't start. Shot in
> the dark, but that's where I'd start looking.
>
> Best,
> Erick
>
> On Tue, May 20, 2014 at 4:45 AM, Tim Burner  wrote:
> > Hi Everyone,
> >
> > I have installed Solr Cloud 4.6.2 with external Zookeeper and Tomcat,
> > having 3 shards with 2 replica each. I tried indexing some documents
> which
> > went easy.
> >
> > After which I restarted my Tomcat, and now the Shards are not getting up,
> > its coming up with bunch of Exceptions. First exception was "*no servers
> > hosting shard:"*
> >
> > All the replica and leader are down and not responding, its even giving
> >
> > RecoveryStrategy Error while trying to recover.
> >
> core=recollection_shard1_replica1:org.apache.solr.client.solrj.SolrServerException:
> > Server refused connection at: http://192.168.2.183:9090/solr
> >
> > It would be great if you can help me out solving this issue. Expert
> advice
> > needed.
> >
> > Thanks in Advance!
>

Re: Vague Behavior while setting Solr Cloud

[Tim Burner]

Thanks Shawn,

I much appreciate your help I got it fixed, actually there were some
background process already running for tomcat which weren't stopped by the
time I faced these issues.

Thanks again!
Tim


On Tue, May 20, 2014 at 11:33 PM, Shawn Heisey  wrote:

> On 5/20/2014 7:10 AM, Tim Burner wrote:
> > I am trying to setup Solr Cloud referring to the blog
> > http://myjeeva.com/solrcloud-cluster-single-collection-deployment.html
> >
> > if I complete the set in one go, then it seems to be going fine.
> >
> > when the setup is complete and I am trying to restart Solr by restarted
> > Tomcat instance, it does not deploy and moreover the shards and replicas
> > are not up.
>
> You've given us nearly zero information about what the problem is.  All
> we know right now is that you restart tomcat and Solr doesn't deploy.
> See this wiki page:
>
> http://wiki.apache.org/solr/UsingMailingLists
>
> Getting specific, we'll need tomcat logs, Solr logs, versions of
> everything.  We might also need your config and schema, depending on
> what the other information reveals.
>
> Thanks,
> Shawn
>
>

Indexing getting failed after some millions of documents

[Tim Burner]

Hi Everyone,

I have installed Solr-4.6 Cloud with external Zookeeper-3.4.5 and Tomcat-7,
the configuration is as mentioned below.

Single Machine Cluster Setup with 3 shards and 2 Replica deployed on 3
Tomcats with 3 Zookeeper.

Everything is installed good and fine, I start with the index and till I
reach some millions of documents(~1.6M) the indexing stops saying "*#503
Service Unavailable" *and the Cloud Dashboard log says

*"ERROR DistributedUpdateProcessor ClusterState says we are the leader,​
but locally we don't think so"*


*"ERROR SolrCore org.apache.solr.common.SolrException: ClusterState says we
are the leader (http://host:port1/solr/recollection_shard1_replica1),​ but
locally we don't think so. Request came from
http://host:port2/solr/recollection_shard2_replica1/"*


*"ERROR ZkController Error registering
SolrCore:org.apache.solr.common.SolrException: Error getting leader from zk
for shard shard2"*

Any suggestions/advice would be appreciated!

Thanks!
Tim

Re: Indexing getting failed after some millions of documents

[Tim Burner]

Hi Eric,

I am running the code from Eclipse with default heap size of 384Mb and
indexing using Solr SimplePostTool, posting xml files through Http Request.
I feel its not concern with Heap, otherwise the program would have made my
process pretty slow rather this is observed every 2-3 hours of indexing
after which some of the node goes down.

I personally feel this may be due to the reason of Leader re election,
because again last exception traced on my Cloud UI Log(mentioned below),
couple of question striking me.

1) Is the leader election not getting over in the zookeeper alloted time.
2) Do I need to increase the zookeerTimeOut param with some greater value
from what its been set currently.
3) Can't we manually elect the Leader and let the election happens if it
goes down.

ERROR StreamingSolrServers error

org.apache.solr.common.SolrException: Service Unavailable



request: http://host:port2
/solr/recollection_shard3_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fhost%3Aport1%2Fsolr%2Frecollection_shard1_replica1%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

14:01:19 ERROR SolrCore org.apache.solr.common.SolrException: No registered
leader was found,​ collection:recollection slice:shard3

org.apache.solr.common.SolrException: No registered leader was found,
collection:recollection slice:shard3
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:467)
at
org.apache.solr.update.SolrCmdDistributor$RetryNode.checkRetry(SolrCmdDistributor.java:351)
at
org.apache.solr.update.SolrCmdDistributor.doRetriesIfNeeded(SolrCmdDistributor.java:78)
at
org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:61)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:499)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1288)
at
org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1002)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

Looking forward for your response.

Thanks,
Tim


On Wed, May 21, 2014 at 8:34 PM, Erick Erickson wrote:

> How much memory have you allocated the JVMs? Also, what's does the
> Solr log show on the machine that isn't coming up? Sounds like the
> node went down and perhaps went into recovery
>
> And how are you indexing?
>
> Best,
> Erick
>
> On Tue, May 20, 2014 at 11:54 PM, Tim Burner 
> wrote:
> > Hi Everyone,
> >
> > I have installed Solr-4.6 Cloud with external Zookeeper-3.4.5 and
> Tomcat-7,
> > the configuration is as mentioned below.
> >
> > Single Machine Cluster Setup with 3 shards and 2 Replica deployed on 3
> > Tomcats with 3 Zookeeper.
> >
> > Everything is installed good and fine, I start with the index