Hello Greg, Consul and Zookeeper are quite similar in their offering with respect to what SolrCloud needs. Service discovery, watches on distributed cluster state, updates of configuration could all be handled through Consul. Plus, Consul does offer built-in capabilities for multi-datacenter scenarios and encryption. Also, the capability to inquire Consul via DNS, i.e., without any client-side library requirements, is quite compelling. One could integrate Java, C/C++, C#/.NET, Python, Ruby and other types of clients without much effort.
The largest benefit, however, I would see for the zoo of services around Solr. At least in my experience, SolrCloud for serious applications is never deployed by itself. There will be numerous services for data collection, semantic processing, log management, monitoring, administration, reporting and user front-ends around the core SolrCloud. This zoo is hard to manage and especially the coordination of configuration and cluster consistency is hard to manage. Consul could help here as it comes from the more operations-type level of managing an elastic set of services in data centers. So, after singing the praises, why have I not started using Consul then? :-) First and foremost: Zookeeper from the Hadoop/Apache ecosystem is already integrated with SolrCloud. Ripping it out and replacing it with something similar but not quite the same would require significant effort, esp. for testing this thoroughly. My clients are not willing to pay for basic groundworks. Second: Consul looks nice but documentation leaves many questions open. Once you start setting it up, there will be questions where you have to dive into the code for answers. Consul does not give me the same "mature" impression as Zookeeper. So, I am still using our own service management framework for the zoo of services in typical search clouds. Consul is young, however, and may evolve. The version is 0.4.1 and I don't use anything with a zero in front to manage a serious customer infrastructure. Would you trust the a customer's 50-100 TB of source data to a set of SolrClouds based on a 0.x Consul? ;-) Third: Consul lacks a decent integration with log management. In any distributed environment, you don't just want to keep a snapshot of the moment, but rather a possibly long history of state changes and statistics, so there is a chance to not just monitor, but also to act. In that respect, we would need more of cloud management recipes integrated, without having to pull out the entire Puppet or Chef stack that will come with its own view of the world. That again is a topic of maturity and being fit for real-life requirements. I would love to see Consul evolve into that type of lightweight cloud management with basic services integrated. But: some way to go still. There are other issues, but these are the major ones from my perspective. So, the concept is nice, Hashimoto et al. are known to be creative heads, and therefore I will keep watching what's happing there, but I won't use Consul for any real customer projects yet - not even that part that is not SolrCloud-dependent. Best regards, --Jürgen On 01.11.2014 00:08, Greg Solovyev wrote: > I am investigating a project to make SolrCloud run on Consul instead of > ZooKeeper. So far, my research revealed no such efforts, but I wanted to > check with this list to make sure I am not going to be reinventing the wheel. > Have anyone attempted using Consul instead of ZK to coordinate SolrCloud > nodes? > > Thanks, > Greg > -- Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С уважением *i.A. Jürgen Wagner* Head of Competence Center "Intelligence" & Senior Cloud Consultant Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543 E-Mail: juergen.wag...@devoteam.com <mailto:juergen.wag...@devoteam.com>, URL: www.devoteam.de <http://www.devoteam.de/> ------------------------------------------------------------------------ Managing Board: Jürgen Hatzipantelis (CEO) Address of Record: 64331 Weiterstadt, Germany; Commercial Register: Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
