Here's what we've decided to do. All updates and deletes from our collections will no longer be applied directly to SolrCloud via Solrj. Instead, they will become messages of a certain topic that go through a RabbitMQ exchange, where an agent in each data center subscribes to the topic with a queue specific to its data center. We will run each agent as a separate webapp inside the same Tomcat instance that hosts Solr itself, on each of our Solr servers. As messages come in, the agent receives them, and then uses Solrj to update them directly into SolrCloud.
The key is RabbitMQ's ability to send the same message to multiple queues that subscribe to the same topic. If each data center sets up a single queue that subscribes to the correct topic, both data centers will receive all the update and delete messages, and will update their indexes accordingly. The net is we have two completely separate SolrCloud clusters, with 2 Solr servers and 3 Zookeepers each, which are all kept up to date in almost lock step. We're planning on using this capability both to provide for a hot disaster recovery backup in a remote data center, as well as to provide distributed active/active search indexes across many data centers. As long as every update/delete message goes into the same federated RabbitMQ exchange and queue, all data centers will receive the update/delete messages and keep their indexes up to date independently. We're also talking to the folks at DataStax about their commercial product, which seems to layer Solr atop the Cassandra distributed data store. This might provide and even more elegant solution that what we're doing. But that is a bit further down the road. Thanks for the help, Darrell Burgan -----Original Message----- From: Darrell Burgan Sent: Wednesday, February 05, 2014 6:48 PM To: solr-user@lucene.apache.org Subject: RE: SolrCloud multiple data center support Let's say I was primarily interested in ensuring there is a DR copy of the search index that is replicated to the remote data center, but I do not want the Solr instances in the remote data center to be part of the SolrCloud cluster, and that I am willing to accept some downtime in bringing up a Solr cluster in the remote data center if we have to use it. Can I use old http-based replication from a remote slave against one of the SolrCloud servers to accomplish that? Primary Data Center 3 x Zookeeper 2 x Solr (clustered via SolrCloud) 1 x collection 1 x shard Remote Data Center 1 x Solr (configured as standalone replication slave against one of the primary data center Solr servers) Would this work to at least get the data to the remote data center in a reliable way? Thanks, Darrell