On 3/8/2015 2:05 PM, Saumitra Srivastav wrote: > I want to start working on adding a TCP layer for client to node and > inter-node communication. > > I am not up to date on recent changes happening to Solr. So before I start > looking into code, I would like to know if there is already some work done > in this direction, which I can reuse. Are there any know > challenges/complexities? > > I would appreciate any help to kick start this effort. Also, what would be > the best way to discuss and get feedback on design from contributors? Open a > JIRA??
What follows is my mostly my opinion, interspersed with a few things that might be loosely called facts: I personally do not mind at all that Solr uses HTTP. Some people view it as an inefficient protocol, but the overhead is low on all but the slowest connections. On a LAN, it is probably not even worth discussing. Most of the time, there will be a LAN connection between the Solr client and the Solr server. I think there are two primary advantages to HTTP: *) Testing can be carried out in a browser with hand-typed URLs. *) We don't have to worry about writing the code to handle the underlying protocol. The first point makes testing throughout the Solr deployment process VERY easy. Expanding on the second point: There are well-tested and mostly bug-free HTTP client and server libraries available for us to use. We don't have to figure out how to write them, troubleshoot them, or optimize them. There are hundreds or thousands of other people using those libraries that can find and report bugs and inefficiencies, often including the fix in the bug report. We might be able to make Solr a little more efficient if we use our own TCP protocol instead of HTTP, but there are drawbacks. It takes a lot of experience to write a network protocol from scratch, and we are bound to make mistakes, mistakes that will cause users a LOT of problems. The authors of those libraries that I mentioned have already gone through that pain, often many times. Solr must be a standalone application to realistically use a custom TCP protocol directly. Although there are already loose plans in place to pull the HTTP and network layers into Solr and make it a standalone application, we are not there yet. If you want to work on a new protocol, feel free. If there is anything I can do to help in between my other duties, I will. I do not know of any existing work that you can re-use, although it's possible there might be something. You might look at the Zookeeper project for an existing implementation of a custom network protocol in Java. I suspect that it will take a very long time before any such work is as stable as HttpClient and the Servlet API (implemented by Jetty). Unless you can demonstrate that stability, it will not become the default protocol. Thanks, Shawn
