I.e. tell the customer that in order to have automatic failover and recovery in a 2-location setup we require at least one ZK instance in a separate third location. Kind of a tough requirement but necessary to safe-guard against split brain during network partition.
If a third location is not an option, how would you setup ZK for manual reconfiguration? Two ZK in DC1 and one in DC2 would give you automatic recovery in case DC2 falls out, but if DC1 falls out, WRITE would be disabled and to resume write in DC2 only, one would need to stop Solr + ZK, reconfigure ZK in DC2 as standalone (or setup two more) and then start Solr again with only one ZK. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 23. mai 2017 kl. 11.14 skrev Markus Jelsma <markus.jel...@openindex.io>: > > I would probably start by renting a VM at a third location to run Zookeeper. > > Markus > > -----Original message----- >> From:Jan Høydahl <jan....@cominvent.com> >> Sent: Tuesday 23rd May 2017 11:09 >> To: solr-user <solr-user@lucene.apache.org> >> Subject: Spread SolrCloud across two locations >> >> Hi, >> >> A customer has two locations (a few km apart) with super-fast networking >> in-between, so for day-to-day operation they view all VMs in both locations >> as a pool of servers. They typically spin up redundant servers for various >> services in each zone and if a zone should fail (they are a few km apart), >> the other will just continue working. >> >> How can we best support such a setup with Cloud and Zookeeper? >> They do not need (or want) CDCR since latency and bandwidth is no problem, >> and CDCR is active-passive only so it anyway requires manual intervention to >> catch up if indexing is switched to the passive DC temporarily. >> If it was not for ZK I would setup one Cloud cluster and make sure each >> shard was replicated cross zones and all would be fine. >> But ZK really requires a third location in order to tolerate loss of an >> entire location/zone. >> All solutions I can think of involves manual intervention, re-configuring of >> ZK followed by a restart of the surviving Solr nodes in order to point to >> the “new” ZK. >> >> How have you guys solved such setups? >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> >>
