SolrCloud: Understanding Replication
Hi, I currently have a standalone SOLR 4.5.1deployment on an EC2 instance with a single collection and core containing an index that's roughly 10G. I've used this as a proof of concept, prototype and staging during development phases and I'm about to release to production. For this release, I've setup 4 EC2 instances with 3 servers in Zookeeper ensemble and the 4 servers running SolrCloud. My intention is to have my current collection on a single Shard replicated 4 times based on the high availability requirements. For that, I'm using an ELB as load balancer to spread the query load to all 4 instances. For this, I've rsync'ed my current 10G Collection to all 4 SOLR instances in my SolrCloud and started them all up. They all come up and do the elections and what nots and all are queryable, which is great. The idea being to load they current index as it is and then start updating it instead of reindexing it all from scratch. BUT... 1) Using zkCLI, I can see that clusterstate shows all instances as down and this is illustrated on the Solr Admin interface by showing all 4 instances using the down color. Is that normal? How can I change that? How come if all 4 instances answer queries just fine? 2) It doesn't seem like the instances are replicating... aka if I add a document to the collection it doesn't get replicated to the other instances. Why is that? What should I look for in SOLR logs that would tell me that replication is happening? I clearly see in there the "/admin/ping" requests made by the load balancer doing health checks and requests made to the admin interface but can never find requests made to "/replicate" that would trigger the replication handler. There's obviously something I've done wrong put I can't put my finger on it. I would appreciate your insight on my situation. Thanks, Marc Campeau
Re: SolrCloud: Understanding Replication
Hi, forgot to mention that I'm migrating the index from Solr 4.5.1 to 4.8.1. Thanks, Marc Campeau 2014-05-30 9:54 GMT-04:00 Marc Campeau : > Hi, > > I currently have a standalone SOLR 4.5.1deployment on an EC2 instance with > a single collection and core containing an index that's roughly 10G. I've > used this as a proof of concept, prototype and staging during development > phases and I'm about to release to production. > > For this release, I've setup 4 EC2 instances with 3 servers in Zookeeper > ensemble and the 4 servers running SolrCloud. My intention is to have my > current collection on a single Shard replicated 4 times based on the high > availability requirements. For that, I'm using an ELB as load balancer to > spread the query load to all 4 instances. For this, I've rsync'ed my > current 10G Collection to all 4 SOLR instances in my SolrCloud and started > them all up. They all come up and do the elections and what nots and all > are queryable, which is great. The idea being to load they current index as > it is and then start updating it instead of reindexing it all from scratch. > > BUT... > > 1) Using zkCLI, I can see that clusterstate shows all instances as down > and this is illustrated on the Solr Admin interface by showing all 4 > instances using the down color. Is that normal? How can I change that? How > come if all 4 instances answer queries just fine? > > 2) It doesn't seem like the instances are replicating... aka if I add a > document to the collection it doesn't get replicated to the other > instances. Why is that? What should I look for in SOLR logs that would tell > me that replication is happening? I clearly see in there the "/admin/ping" > requests made by the load balancer doing health checks and requests made to > the admin interface but can never find requests made to "/replicate" that > would trigger the replication handler. > > There's obviously something I've done wrong put I can't put my finger on > it. I would appreciate your insight on my situation. > > Thanks, > > Marc Campeau > >
Re: SolrCloud: Understanding Replication
2014-05-30 12:24 GMT-04:00 Erick Erickson :
> Let's back up a bit here. Why are you copying your indexes around?
> SolrCloud does all this for you. I suspect you've somehow made a mis-step.
>
I started by copying the index around because my 4.5.1 instance is not
setup as Cloud and I wanted to avoid reindexing all my data when migrating
to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
trying to get replication happening when I populate an empty collection.
> So here's what I'd do by preference; Just set up a new collection and
> re-index. Make sure all of the nodes are up and then just go ahead and
> index to any of them. If you're using SolrJ, CloudSolrServer will be a bit
> more efficient than sending the docs to random nodes, but that's not
> necessary.
>
I've been trying that this morning. Stop the instances, deleted the
contents of /data on all my 4.8.1 instances then started them again...
they all show up in a 1 shard cluster as 4 replicas and one is the
leader... they're still shown as down in clusterstate. Then I sent a
document to be added to one of the nodes specifically. Only that node now
contains the document. It hasn't been replicated to the other instances.
When I issue queries to the collection for that document through my load
balancer it works roughtly 1/4 times, in accordance with the fact that it's
only on the instance where it was added.
Must I use the CLI API for collections to create this new collection or can
I just do it old style by creating subfolder in /solr directory with my
confs?
Here's the log of these operations
LOG of Instance where document was added:
2758138 [qtp1781256139-14] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
{add=[Listing_3446279]} 0 271
2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
[mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
QTime=1
[... More Pings ... ]
2773138 [commitScheduler-7-thread-1] INFO
org.apache.solr.update.UpdateHandler – start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
2773377 [commitScheduler-7-thread-1] INFO
org.apache.solr.search.SolrIndexSearcher – Opening
Searcher@175816a5[mycollection]
main
2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
– QuerySenderListener sending requests to Searcher@175816a5[mycollection]
main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
– QuerySenderListener done.
2773390 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
– [mycollection] Registered new searcher Searcher@175816a5[mycollection]
main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
2773390 [commitScheduler-7-thread-1] INFO
org.apache.solr.update.UpdateHandler – end_commit_flush
[... More Pings ... ]
2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr-4.8.0/example/solr/mycollection/data/index
lockFactory=NativeFSLockFactory@/opt/solr-4.8.0/example/solr/mycollection/data/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr-4.8.0/example/solr/mycollection/data/index
lockFactory=NativeFSLockFactory@/opt/solr-4.8.0/example/solr/mycollection/data/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
2799884 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore – newest
commit generation = 2
2799887 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
SolrIndexSearcher has not changed - not re-opening:
org.apache.solr.search.SolrIndexSearcher
2799887 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
end_commit_flush
2799888 [qtp1781256139-18] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
http://192.168.150.90:8983/solr/mycollection/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
{commit=} 0 96
2800051 [qtp1781256139-14] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json} {commit=}
0 611
2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2802085 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler – No
uncommi
Re: SolrCloud: Understanding Replication
As of now I'm trying reindexing everything. Basically I have now an empty
collection and when I add a document it's not getting replicated. Not
trying anymore to load my old index anymore.
Marc
2014-05-30 16:44 GMT-04:00 Jason Hellman :
> Marc,
>
> Fundamentally it’s a good solution design to always be capable of
> reposting (reindexing) your data to Solr. You are demonstrating a classic
> use case of this, which is upgrade. Is there a critical reason why you are
> avoiding this step?
>
> Jason
>
> On May 30, 2014, at 10:38 AM, Marc Campeau wrote:
>
> > 2014-05-30 12:24 GMT-04:00 Erick Erickson :
> >
> >> Let's back up a bit here. Why are you copying your indexes around?
> >> SolrCloud does all this for you. I suspect you've somehow made a
> mis-step.
> >>
> >
> > I started by copying the index around because my 4.5.1 instance is not
> > setup as Cloud and I wanted to avoid reindexing all my data when
> migrating
> > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> > trying to get replication happening when I populate an empty collection.
> >
> >
> >> So here's what I'd do by preference; Just set up a new collection and
> >> re-index. Make sure all of the nodes are up and then just go ahead and
> >> index to any of them. If you're using SolrJ, CloudSolrServer will be a
> bit
> >> more efficient than sending the docs to random nodes, but that's not
> >> necessary.
> >>
> >
> > I've been trying that this morning. Stop the instances, deleted the
> > contents of /data on all my 4.8.1 instances then started them again...
> > they all show up in a 1 shard cluster as 4 replicas and one is the
> > leader... they're still shown as down in clusterstate. Then I sent a
> > document to be added to one of the nodes specifically. Only that node now
> > contains the document. It hasn't been replicated to the other instances.
> >
> > When I issue queries to the collection for that document through my load
> > balancer it works roughtly 1/4 times, in accordance with the fact that
> it's
> > only on the instance where it was added.
> >
> > Must I use the CLI API for collections to create this new collection or
> can
> > I just do it old style by creating subfolder in /solr directory with my
> > confs?
> >
> >
> > Here's the log of these operations
> >
> > LOG of Instance where document was added:
> >
> > 2758138 [qtp1781256139-14] INFO
> > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > {add=[Listing_3446279]} 0 271
> > 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > [mycollection] webapp=/solr path=/admin/ping params={} hits=0 status=0
> > QTime=1
> >
> > [... More Pings ... ]
> >
> > 2773138 [commitScheduler-7-thread-1] INFO
> > org.apache.solr.update.UpdateHandler – start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> > 2773377 [commitScheduler-7-thread-1] INFO
> > org.apache.solr.search.SolrIndexSearcher – Opening
> > Searcher@175816a5[mycollection]
> > main
> > 2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> > – QuerySenderListener sending requests to Searcher@175816a5
> [mycollection]
> > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > 2773389 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> > – QuerySenderListener done.
> > 2773390 [searcherExecutor-5-thread-1] INFO org.apache.solr.core.SolrCore
> > – [mycollection] Registered new searcher Searcher@175816a5[mycollection]
> > main{StandardDirectoryReader(segments_1:3:nrt _0(4.8):C1)}
> > 2773390 [commitScheduler-7-thread-1] INFO
> > org.apache.solr.update.UpdateHandler – end_commit_flush
> >
> > [... More Pings ... ]
> >
> > 2799792 [qtp1781256139-18] INFO org.apache.solr.update.UpdateHandler –
> > start
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> > 2799883 [qtp1781256139-18] INFO org.apache.solr.core.SolrCore –
> > SolrDeletionPolicy.onCommit: commits: num=2
> > commit{dir=NRTCachingDirectory(MMapDirectory@
> /opt/solr-4.8.0/example/solr/mycollection/data/index
> > lockFactory=NativeFSLockFactory@
> /opt/solr-4.8.0/example/solr/mycollection/data/ind
Re: SolrCloud: Understanding Replication
cuments that are being indexed.
>
> If none of that is a problem, I'd really consider just doing the example
> code first. Get _that_ set up without doing _any_ changes and get it
> running, _then_ try your stuff. It really sounds like you've got a
> fundamental mis-configuration and are beating your head against the wall to
> no good purpose.
>
> Best
> Erick
>
>
> On Fri, May 30, 2014 at 1:58 PM, Marc Campeau wrote:
>
> > As of now I'm trying reindexing everything. Basically I have now an empty
> > collection and when I add a document it's not getting replicated. Not
> > trying anymore to load my old index anymore.
> >
> > Marc
> >
> >
> > 2014-05-30 16:44 GMT-04:00 Jason Hellman >:
> >
> > > Marc,
> > >
> > > Fundamentally it’s a good solution design to always be capable of
> > > reposting (reindexing) your data to Solr. You are demonstrating a
> > classic
> > > use case of this, which is upgrade. Is there a critical reason why you
> > are
> > > avoiding this step?
> > >
> > > Jason
> > >
> > > On May 30, 2014, at 10:38 AM, Marc Campeau wrote:
> > >
> > > > 2014-05-30 12:24 GMT-04:00 Erick Erickson :
> > > >
> > > >> Let's back up a bit here. Why are you copying your indexes around?
> > > >> SolrCloud does all this for you. I suspect you've somehow made a
> > > mis-step.
> > > >>
> > > >
> > > > I started by copying the index around because my 4.5.1 instance is
> not
> > > > setup as Cloud and I wanted to avoid reindexing all my data when
> > > migrating
> > > > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just
> > > > trying to get replication happening when I populate an empty
> > collection.
> > > >
> > > >
> > > >> So here's what I'd do by preference; Just set up a new collection
> and
> > > >> re-index. Make sure all of the nodes are up and then just go ahead
> and
> > > >> index to any of them. If you're using SolrJ, CloudSolrServer will
> be a
> > > bit
> > > >> more efficient than sending the docs to random nodes, but that's not
> > > >> necessary.
> > > >>
> > > >
> > > > I've been trying that this morning. Stop the instances, deleted the
> > > > contents of /data on all my 4.8.1 instances then started them
> again...
> > > > they all show up in a 1 shard cluster as 4 replicas and one is the
> > > > leader... they're still shown as down in clusterstate. Then I sent a
> > > > document to be added to one of the nodes specifically. Only that node
> > now
> > > > contains the document. It hasn't been replicated to the other
> > instances.
> > > >
> > > > When I issue queries to the collection for that document through my
> > load
> > > > balancer it works roughtly 1/4 times, in accordance with the fact
> that
> > > it's
> > > > only on the instance where it was added.
> > > >
> > > > Must I use the CLI API for collections to create this new collection
> or
> > > can
> > > > I just do it old style by creating subfolder in /solr directory with
> my
> > > > confs?
> > > >
> > > >
> > > > Here's the log of these operations
> > > >
> > > > LOG of Instance where document was added:
> > > >
> > > > 2758138 [qtp1781256139-14] INFO
> > > > org.apache.solr.update.processor.LogUpdateProcessor – [mycollection]
> > > > webapp=/solr path=/update/ params={indent=on&version=2.2&wt=json}
> > > > {add=[Listing_3446279]} 0 271
> > > > 2769177 [qtp1781256139-12] INFO org.apache.solr.core.SolrCore –
> > > > [mycollection] webapp=/solr path=/admin/ping params={} hits=0
> status=0
> > > > QTime=1
> > > >
> > > > [... More Pings ... ]
> > > >
> > > > 2773138 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.update.UpdateHandler – start
> > > >
> > >
> >
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> > > > 2773377 [commitScheduler-7-thread-1] INFO
> > > > org.apache.solr.search.SolrIndexSearcher – Opening
> >
Re: SolrCloud: Understanding Replication
I notice I have this in the logs when I start SOLR for default example (I had the same with my own connection) 21242 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext – Enough replicas found to continue. 21242 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext – I may be the new leader - try and sync 21242 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.SyncStrategy – Sync replicas to http://192.168.150.90:8983/solr/collection1/ 21243 [coreZkRegister-1-thread-1] ERROR org.apache.solr.cloud.SyncStrategy – No UpdateLog found - cannot sync 21243 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext – We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 21243 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext – I am the new leader: http://192.168.150.90:8983/solr/collection1/ shard1 21243 [coreZkRegister-1-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /collections/collection1/leaders/shard1 2014-05-30 17:54 GMT-04:00 Erick Erickson : > bq: it's not getting replicated > > This is definitely not what's expected. Are you by chance _configuring_ > replication while at the same time using SolrCloud? Posting your > solrconfig.xml would help answer that. > > This should be all that's in your solrconfig.xml file when running under > SolrCloud: > > > No parameters. No polling intervals. No URLs for the master (there isn't > one anyway). Nothing but what's above. > > You also need to insure that your soft commit interval (or hard commit if > you have openSearcher set to true) is exceeded before you can search > documents that are being indexed. > > If none of that is a problem, I'd really consider just doing the example > code first. Get _that_ set up without doing _any_ changes and get it > running, _then_ try your stuff. It really sounds like you've got a > fundamental mis-configuration and are beating your head against the wall to > no good purpose. > > Best > Erick > > > On Fri, May 30, 2014 at 1:58 PM, Marc Campeau wrote: > > > As of now I'm trying reindexing everything. Basically I have now an empty > > collection and when I add a document it's not getting replicated. Not > > trying anymore to load my old index anymore. > > > > Marc > > > > > > 2014-05-30 16:44 GMT-04:00 Jason Hellman >: > > > > > Marc, > > > > > > Fundamentally it’s a good solution design to always be capable of > > > reposting (reindexing) your data to Solr. You are demonstrating a > > classic > > > use case of this, which is upgrade. Is there a critical reason why you > > are > > > avoiding this step? > > > > > > Jason > > > > > > On May 30, 2014, at 10:38 AM, Marc Campeau wrote: > > > > > > > 2014-05-30 12:24 GMT-04:00 Erick Erickson : > > > > > > > >> Let's back up a bit here. Why are you copying your indexes around? > > > >> SolrCloud does all this for you. I suspect you've somehow made a > > > mis-step. > > > >> > > > > > > > > I started by copying the index around because my 4.5.1 instance is > not > > > > setup as Cloud and I wanted to avoid reindexing all my data when > > > migrating > > > > to my new 4.8.1 SolrCloud setup. I've now put that aside and I'm just > > > > trying to get replication happening when I populate an empty > > collection. > > > > > > > > > > > >> So here's what I'd do by preference; Just set up a new collection > and > > > >> re-index. Make sure all of the nodes are up and then just go ahead > and > > > >> index to any of them. If you're using SolrJ, CloudSolrServer will > be a > > > bit > > > >> more efficient than sending the docs to random nodes, but that's not > > > >> necessary. > > > >> > > > > > > > > I've been trying that this morning. Stop the instances, deleted the > > > > contents of /data on all my 4.8.1 instances then started them > again... > > > > they all show up in a 1 shard cluster as 4 replicas and one is the > > > > leader... they're still shown as down in clusterstate. Then I sent a > > > > document to be added to one of the nodes specifically. Only that node > > now > > > > contains the document. It hasn't been replicated to the
Re: SolrCloud: Understanding Replication
Hi, first I'd like to thank those who've spent their time reading this, especially Erick, Jason and Shawn. Thank you! I've finally got it working. Shawn was right, I needed to enable updateLog in solrconfig.xml and create the tlog directory. After doing, so I could index documents sent on any node, they would go to the leader which indexed them but they still weren't being replicated to the replicas. After looking at the logs, I remembered some thing I had read somewhere (would like to include link here but can't find it) about when using a Zookeeper ensemble and wanting to reset your Solr Collection you should not only clear your data folder and tlog folder but you should also clear out your Zookeeper data folders on all members of ensemble. So I stopped all Solr Instances in cluster, and then all members of Zookeeper ensemble. Deleted the Zookeeper data/version_2 folder on all Zookeeper instances. Deleted the Solr index data on all Solr instances. Restarted the Zookeeper Ensemble, pushed the SOLR Configs to it and started the remainder of the Solr cluster. That's when it all started working. Documents I had started indexing that were solely on the leader got replicated to replicas and then any document sent to any member of cluster would be indexed and replicated. So lessons learned: 1) Don't forget to enable updateLog. Needs to be tested: an index that doesn't have updateLog associated to it cannot be made into replica, or at least cannot just be copied over to replica nodes. 2) When you want to reset a collection, it's a good idea to reset the zookeeper ensemble data, especially if your Solr Cluster wasn't replicating. Marc 2014-06-02 16:31 GMT-04:00 Shawn Heisey : > On 6/2/2014 2:21 PM, Marc Campeau wrote: > > I notice I have this in the logs when I start SOLR for default example (I > > had the same with my own connection) > > > > 21242 [coreZkRegister-1-thread-1] INFO > > org.apache.solr.cloud.ShardLeaderElectionContext – Enough replicas > found > > to continue. > > 21242 [coreZkRegister-1-thread-1] INFO > > org.apache.solr.cloud.ShardLeaderElectionContext – I may be the new > > leader - try and sync > > 21242 [coreZkRegister-1-thread-1] INFO > org.apache.solr.cloud.SyncStrategy > > – Sync replicas to http://192.168.150.90:8983/solr/collection1/ > > 21243 [coreZkRegister-1-thread-1] ERROR > org.apache.solr.cloud.SyncStrategy > > – No UpdateLog found - cannot sync > > 21243 [coreZkRegister-1-thread-1] INFO > > org.apache.solr.cloud.ShardLeaderElectionContext – We failed sync, but > we > > have no versions - we can't sync in that case - we were active before, so > > become leader anyway > > 21243 [coreZkRegister-1-thread-1] INFO > > org.apache.solr.cloud.ShardLeaderElectionContext – I am the new leader: > > http://192.168.150.90:8983/solr/collection1/ shard1 > > 21243 [coreZkRegister-1-thread-1] INFO > > org.apache.solr.common.cloud.SolrZkClient – makePath: > > /collections/collection1/leaders/shard1 > > The "No UpdateLog found - cannot sync" message suggests that either you > have disabled the updateLog feature in solrconfig.xml or that you have > deleted the tlog directory (or possibly the entire data directory). > Hopefully it's the latter, because SolrCloud cannot function properly if > the updateLog is disabled. For data integrity reasons, the updateLog is > always recommended, whether using SolrCloud or not. > > Thanks, > Shawn > >
Shard Replicas not getting replicated data from leader
Hi,
I have setup 4 Solr (4.9.0) Nodes into a single shard for a given
collection, meaning I should have 4 replicated nodes. I have 3 Zookeepers
in ensemble managing the configs for this collection. I have a load
balancer in front of the 4 nodes to split traffic between them.
I start this collection with an empty data/index directory.
When I send /update requests to the load balancers I see these going to all
4 nodes. Also, I can see that all FOLLOWERs distribute the requests they
receive to the LEADER as is expected. But for some reason the FOLLOWERS are
not getting /replication requests from the LEADER. So the collection for
the leader contains many thousand of documents and is on the 8th
generation. I see that it's replicable in the admin interface, yet all
FOLLOWER nodes have an empty index.
Hence, I need your insights please.
Thanks,
Marc
To Note:
When I startup my nodes I see the following error in solr.log:
1) When Zookeeper does a clusterstate update, all nodes have their starte
"DOWN", why? This I means that in the Solr Admin interface they show up has
down. This never updates to active.
2) I have a warning : org.apache.solr.rest.ManagedResource; No registered
observers for /rest/managed, which I need to update solrconfig.xml to fix
3) I have the following error:
ERROR - 2014-07-16 19:49:25.336; org.apache.solr.cloud.SyncStrategy; No
UpdateLog found - cannot sync
SOLR.LOG
-
[]
INFO - 2014-07-16 19:47:30.870;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
numShards=null message={
"operation":"state",
"state":"down",
"base_url":"http://192.168.150.90:8983/solr";,
"core":"collection_name",
"roles":null,
"node_name":"192.168.150.90:8983_solr",
"shard":null,
"collection":"collection_name",
"numShards":null,
"core_node_name":null}
INFO - 2014-07-16 19:47:30.871;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node1 is
already registered
[]
WARN - 2014-07-16 19:47:34.535; org.apache.solr.rest.ManagedResource; No
registered observers for /rest/managed
[]
INFO - 2014-07-16 19:48:25.135;
org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
INFO - 2014-07-16 19:48:25.287;
org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
INFO - 2014-07-16 19:48:25.291;
org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
ZooKeeper...
INFO - 2014-07-16 19:48:25.293;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
numShards=null message={
"operation":"state",
"state":"down",
"base_url":"http://192.168.200.90:8983/solr";,
"core":"collection_name",
"roles":null,
"node_name":"192.168.200.90:8983_solr",
"shard":null,
"collection":"collection_name",
"numShards":null,
"core_node_name":null}
INFO - 2014-07-16 19:48:25.293;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node2 is
already registered
INFO - 2014-07-16 19:48:25.293;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is already
registered
[]
INFO - 2014-07-16 19:49:00.188;
org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
INFO - 2014-07-16 19:49:00.322;
org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
INFO - 2014-07-16 19:49:00.335;
org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
ZooKeeper...
INFO - 2014-07-16 19:49:00.337;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
numShards=null message={
"operation":"state",
"state":"down",
"base_url":"http://192.168.200.91:8983/solr";,
"core":"collection_name",
"roles":null,
"node_name":"192.168.200.91:8983_solr",
"shard":null,
"collection":"collection_name",
"numShards":null,
"core_node_name":null}
INFO - 2014-07-16 19:49:00.337;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node3 is
already registered
INFO - 2014-07-16 19:49:00.337;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is already
registered
[]
INFO - 2014-07-16 19:49:21.220;
org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
INFO - 2014-07-16 19:49:21.350;
org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
INFO - 2014-07-16 19:49:21.357;
org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
ZooKeeper...
INFO - 2014-07-16 19:49:21.359;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
numShards=null message={
"operation":"state",
"state":"down",
"base_url":"http://192.168.150.91:8983/solr";,
"core":"collection_name",
"roles":null,
"node_name":"192.168.150.91:8983_solr",
"shard":null,
"collection":"collection_name",
"numShards":null,
"core_node_name":null}
Re: Shard Replicas not getting replicated data from leader
Turns out updateLog had been disabled in solrconfig.xml
Marc
2014-07-16 16:44 GMT-04:00 Marc Campeau :
> Hi,
>
> I have setup 4 Solr (4.9.0) Nodes into a single shard for a given
> collection, meaning I should have 4 replicated nodes. I have 3 Zookeepers
> in ensemble managing the configs for this collection. I have a load
> balancer in front of the 4 nodes to split traffic between them.
>
> I start this collection with an empty data/index directory.
>
> When I send /update requests to the load balancers I see these going to
> all 4 nodes. Also, I can see that all FOLLOWERs distribute the requests
> they receive to the LEADER as is expected. But for some reason the
> FOLLOWERS are not getting /replication requests from the LEADER. So the
> collection for the leader contains many thousand of documents and is on the
> 8th generation. I see that it's replicable in the admin interface, yet all
> FOLLOWER nodes have an empty index.
>
> Hence, I need your insights please.
>
> Thanks,
>
> Marc
>
> To Note:
>
> When I startup my nodes I see the following error in solr.log:
> 1) When Zookeeper does a clusterstate update, all nodes have their starte
> "DOWN", why? This I means that in the Solr Admin interface they show up has
> down. This never updates to active.
>
> 2) I have a warning : org.apache.solr.rest.ManagedResource; No registered
> observers for /rest/managed, which I need to update solrconfig.xml to fix
>
> 3) I have the following error:
> ERROR - 2014-07-16 19:49:25.336; org.apache.solr.cloud.SyncStrategy; No
> UpdateLog found - cannot sync
>
> SOLR.LOG
> -
> []
> INFO - 2014-07-16 19:47:30.870;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> numShards=null message={
> "operation":"state",
> "state":"down",
> "base_url":"http://192.168.150.90:8983/solr";,
> "core":"collection_name",
> "roles":null,
> "node_name":"192.168.150.90:8983_solr",
> "shard":null,
> "collection":"collection_name",
> "numShards":null,
> "core_node_name":null}
> INFO - 2014-07-16 19:47:30.871;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node1 is
> already registered
> []
> WARN - 2014-07-16 19:47:34.535; org.apache.solr.rest.ManagedResource; No
> registered observers for /rest/managed
> []
> INFO - 2014-07-16 19:48:25.135;
> org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
> INFO - 2014-07-16 19:48:25.287;
> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
> fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> INFO - 2014-07-16 19:48:25.291;
> org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> ZooKeeper...
> INFO - 2014-07-16 19:48:25.293;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> numShards=null message={
> "operation":"state",
> "state":"down",
> "base_url":"http://192.168.200.90:8983/solr";,
> "core":"collection_name",
> "roles":null,
> "node_name":"192.168.200.90:8983_solr",
> "shard":null,
> "collection":"collection_name",
> "numShards":null,
> "core_node_name":null}
> INFO - 2014-07-16 19:48:25.293;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node2 is
> already registered
> INFO - 2014-07-16 19:48:25.293;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is already
> registered
> []
> INFO - 2014-07-16 19:49:00.188;
> org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
> INFO - 2014-07-16 19:49:00.322;
> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
> fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> INFO - 2014-07-16 19:49:00.335;
> org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> ZooKeeper...
> INFO - 2014-07-16 19:49:00.337;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> numShards=null message={
> "operation":"state",
> "state":"down",
> "base_url":"http://192.168.200.91:8983/solr";,
> "core":"collection_name",
> "roles":null,
> "node_name":"192.168.200.91:8983_solr",
> "shard":null,
> "collection":&qu
Schema Updates and Core Reload with SolrCloud
Hi, I want to apply a change to a Schema of a Collection deployed on SolrCloud. This SolrCloud consists of a 3 Zookeepers ensemble overseeing 4 Solr Instances that each have a replicated version of a single Collection. As I understand it, I need to update my schema.xml and then upload it to the Zookeeper ensemble. Then I need to reload the Collection. My questions are: Do I only need to reload one Shard (Primary, anyone will do) and the schema update will be propagated to other shards or must I reload every single shard one after the other for the changes to the schema to be applied? Is there an order in which shards should be reloaded? Thanks for your time, Marc Campeau
Re: Schema Updates and Core Reload with SolrCloud
This past thread from June might hold the answer to my question. Posting this here for other folks that might stumble upon this question. -- Forwarded message -- From: Shawn Heisey Date: 2014-06-28 1:36 GMT-04:00 Subject: Re: Some questions about Solrcloud To: solr-user@lucene.apache.org On 6/27/2014 11:07 PM, spir...@gmail.com wrote: > Thanks for answers ! > > So all my changes i'll must provide via zookeeper ? I asking because this moment confused me when i bootstrap one node with full config, then added 2 nodes to cluster, but conf dirs on new appended nodes is empty (solr/collection1/conf i mean). > > And small question - if i bootstrap Solr with one configured core and second core without data and default config. SolrCloud show me both cores with replicas, how i must managed second core - add data , update configs, via zookeeper or using Coreadmin API ? In my opinion, the bootstrap options should never be used. You should upload configurations to zookeeper using the zkcli script included with the Solr download in cloud-scripts. Then you can create collections using the Collections API, and give the name of the config that you uploaded earlier as a parameter. If you do things this way, there will be no conf directories in your cores -- even if they exist, they are not used with SolrCloud. When you re-upload an existing config to zookeeper to make changes, getting the changes to take effect is as simple as using the Collections API to RELOAD the collection. Thanks, Shawn 2014-08-27 13:53 GMT-04:00 Marc Campeau : > Hi, > > I want to apply a change to a Schema of a Collection deployed on SolrCloud. > > This SolrCloud consists of a 3 Zookeepers ensemble overseeing 4 Solr > Instances that each have a replicated version of a single Collection. > > As I understand it, I need to update my schema.xml and then upload it to > the Zookeeper ensemble. Then I need to reload the Collection. > > My questions are: > > Do I only need to reload one Shard (Primary, anyone will do) and the > schema update will be propagated to other shards or must I reload every > single shard one after the other for the changes to the schema to be > applied? > > Is there an order in which shards should be reloaded? > > Thanks for your time, > > Marc Campeau >
Re: Schema Updates and Core Reload with SolrCloud
Well I got my answer by trying it. I issued the Collections API reload action on one of the shards (Shard4 to be exact, not the leader) and I got this back as output. So the action triggers a reload of all shards composing the Collection. 0 11957 0 4585 0 2839 0 2655 0 7705 2014-08-27 14:00 GMT-04:00 Marc Campeau : > This past thread from June might hold the answer to my question. Posting > this here for other folks that might stumble upon this question. > > -- Forwarded message -- > From: Shawn Heisey > Date: 2014-06-28 1:36 GMT-04:00 > Subject: Re: Some questions about Solrcloud > To: solr-user@lucene.apache.org > > > On 6/27/2014 11:07 PM, spir...@gmail.com wrote: > > Thanks for answers ! > > > > So all my changes i'll must provide via zookeeper ? I asking because > this moment confused me when i bootstrap one node with full config, then > added 2 nodes to cluster, but conf dirs on new appended nodes is empty > (solr/collection1/conf i mean). > > > > And small question - if i bootstrap Solr with one configured core and > second core without data and default config. SolrCloud show me both cores > with replicas, how i must managed second core - add data , update configs, > via zookeeper or using Coreadmin API ? > > In my opinion, the bootstrap options should never be used. You should > upload configurations to zookeeper using the zkcli script included with > the Solr download in cloud-scripts. Then you can create collections > using the Collections API, and give the name of the config that you > uploaded earlier as a parameter. > > If you do things this way, there will be no conf directories in your > cores -- even if they exist, they are not used with SolrCloud. When you > re-upload an existing config to zookeeper to make changes, getting the > changes to take effect is as simple as using the Collections API to > RELOAD the collection. > > Thanks, > Shawn > > > > 2014-08-27 13:53 GMT-04:00 Marc Campeau : > > Hi, >> >> I want to apply a change to a Schema of a Collection deployed on >> SolrCloud. >> >> This SolrCloud consists of a 3 Zookeepers ensemble overseeing 4 Solr >> Instances that each have a replicated version of a single Collection. >> >> As I understand it, I need to update my schema.xml and then upload it to >> the Zookeeper ensemble. Then I need to reload the Collection. >> >> My questions are: >> >> Do I only need to reload one Shard (Primary, anyone will do) and the >> schema update will be propagated to other shards or must I reload every >> single shard one after the other for the changes to the schema to be >> applied? >> >> Is there an order in which shards should be reloaded? >> >> Thanks for your time, >> >> Marc Campeau >> > >
Re: Schema Updates and Core Reload with SolrCloud
Thanks for your validation Shawn. Marc 2014-08-27 15:18 GMT-04:00 Shawn Heisey : > On 8/27/2014 1:03 PM, Marc Campeau wrote: > > Well I got my answer by trying it. I issued the Collections API reload > > action on one of the shards (Shard4 to be exact, not the leader) and I > got > > this back as output. So the action triggers a reload of all shards > > composing the Collection. > > The Collections API is not tied to a specific core. The URL path is > /solr/admin/collections ... not /solr/corename/admin/collections. A > collections API RELOAD, unlike a CoreAdmin RELOAD, does indeed apply to > the entire collection. You can send the request to any Solr instance in > the cloud. > > Thanks, > Shawn > >
